I know this doesn't answer the actual question at the bottom of your post, but I'd add another key testing mode that is important to evaluate [and may change your pre-touch approach somewhat]:
- Run a burst of messages, then go to absolute zero rate for 10-20 seconds, then burst again, and watch the latency of the first message in the burst. - Do the above, but with some other continuous write-heavy process running on the same system, and maybe some background reading. The purpose of this background activity is to cause enough file-system buffering pressure to ensure that you actually observe whatever periodic kernel flushing of dirty buffers looks like. I suspect that while a 1MB-lookahed pre-touch triggered when 1MB-away !MB boundaries are crossed)(will work well to reduce latency impacts in general, it will perform much better in continuous workloads and in an otherwise "quiet" system. When the system has it's own separate reasons to occasionally flush dirty buffers, and the workload being measures goes quiet in between latency-sensitive bursts, a 1MB-head thing may be too far. E.g. some sort of option for a "log banging" pre-touch mechanism (where you write/pre-touch at some fixed offsets ahead of where you are for each message) may be a more successful at keeping the log-edge dirty in memory. On Monday, July 10, 2017 at 9:56:30 AM UTC-7, Roman Leventov wrote: > > I'm trying to improve the latency of synchronous logging for Log4j: > https://github.com/apache/logging-log4j2/pull/87 > > One of the means is "write pretouch" of the mapped file contents: > https://github.com/leventov/logging-log4j2/blob/6d1c0e6b9a73b46ba7dd28ff853694f2012873c0/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/UnsafeMappedByteBufferWriteTouch.java > > . Currently it is done as the following: when logger hits the 1st MB of the > file, a task is scheduled to write-pretouch file contents between 1MB and > 2MB marks (i. e. the 2nd MB) in a background thread. When logger starts to > write in the 2nd MB, a task for write-pretouch of the 3rd MB is scheduled, > and so on, i. e. at 1 MB blocks. > > Some testing is done on 4-core MBP, in 3 logging threads: > - new version looks great at 2k messages/sec/thread: > https://cdn.rawgit.com/leventov/eee84006a53dfb8d2acaebfe126bb499/raw/805c548d1f41e49e67b6815dde84d92be0568012/load2k.html > > (and pretouch actually contributes a lot to this result, without pretouch > the new version is just a little better than the old) > - decent at 20k: > https://cdn.rawgit.com/leventov/eee84006a53dfb8d2acaebfe126bb499/raw/805c548d1f41e49e67b6815dde84d92be0568012/load20k.html > - but suddenly much worse at 100k messages/sec/thread, than the old > version: > https://cdn.rawgit.com/leventov/eee84006a53dfb8d2acaebfe126bb499/raw/805c548d1f41e49e67b6815dde84d92be0568012/load100k.html > > I tried different approaches to pretouch, but it consistency incurs > extremely long pauses, when running at "pretty extreme" message rates. Only > not doing pretouch at all allows to avoid those pauses. > > Does anybody have a theory what's going on here? Is it more likely the > feature of consumer-grade hardware, or the pretouch approach falls in bad > inference with the MM implementation in OS X? > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
