Thanks Jay. This is a very helpful investigation! On 05/24/2012 01:40 PM, Jay Kreps wrote: > > Unfortunately *any* call to fsync will block appends even in a background > thread so how can we give control over physical disk persistence without > introducing high latency for the producer? The answer is that the linux > pdflush daemon actually does a very similar thing to our flush parameters. > pdflush is a daemon running on every linux machine that controls the > writing of buffered/cached data back to disk. It allows you to control the > percentage of memory filled with dirty pages by giving it either a > percentage of memory, a time out for any dirty page to be written, or a > fixed number of dirty bytes.
This would however by necessity by a global setting right? (Assuming there is no /proc trickery to change per-pid pdflush behaviour)
