On 07/02/2019 19:47, Marc Roos wrote:
Is this difference not related to chaching? And you filling up some
cache/queue at some point? If you do a sync after each write, do you
have still the same results?

No, the slow operations are slow from the very beginning. It's not about filling a buffer/cache somewhere. I'm guessing the slow operations trigger several synchronous writes to the underlying OSDs, while the fast ones don't. But I'd like to know more about why exactly there is this significant performance hit to truncation operations vs. normal writes.

To give some more numbers:

echo test | dd of=b conv=notrunc

This completes extremely quickly (microseconds). The data obviously remains in the client cache at this point. This is what I want.

echo test | dd of=b conv=notrunc,fdatasync

This runs quickly until the fdatasync(), then that takes ~12ms, which is about what I'd expect for a synchronous write to the underlying HDDs. Or maybe that's two writes?

echo test | dd of=b

This takes ~10ms in the best case for the open() call (sometimes 30-40 or even more), and 6-8ms for the write() call.

echo test | dd of=b conv=fdatasync

This takes ~10ms for the open() call, ~8ms for the write() call, and ~18ms for the fdatasync() call.

So it seems like truncating/recreating an existing file introduces several disk I/Os worth of latency and forces synchronous behavior somewhere down the stack, while merely creating a new file or writing to an existing one without truncation does not.

Hector Martin (hec...@marcansoft.com)
Public Key: https://mrcn.st/pub
ceph-users mailing list

Reply via email to