Just to clarify a little further.  Your comment about the inode lock is correct 
(keeping in mind the comment from Andreas about more recent changes), but 
Lustre (in buffered, rather than O_DIRECT mode)returns to user space before the 
data is down on the server.  It maintains the pages in the page cache on the 
client until it knows for sure it will not have to resend them.  This means 
that - in the inode lock case - while only one process can be actively writing 
data to the page cache at a given moment, Lustre returns to userspace before 
the data is written out.

This means that while (in the inode lock case) the 'writing pages in to the 
page cache and setting them up to go out' (which is nearly all of most write 
syscalls) can't overlap, the network and disk portions or Lustre I/Os can 
overlap.  That's how multiple writes can be in flight at once even with single 
entry for writing processes.

This also applies to one process - it can get many I/Os in flight at once, and 
achieve much higher bandwidth by overlapping that time they spend in flight.  
So, buffered writing is generally >> direct writing.  Extremely large write 
sizes can mitigate this, as Andreas suggested.

- Patrick
From: Dilger, Andreas <>
Sent: Friday, October 14, 2016 4:38:19 PM
To: John Bauer; Riccardo Veraldi
Cc:; Patrick Farrell
Subject: Re: [lustre-discuss] Lustre on ZFS pooer direct I/O performance

John, with newer Lustre clients it is possible for multiple threads to submit 
non-overlapping writes concurrently (also not conflicting within a single 
page), see LU-1669 for details.

Even so, O_DIRECT writes need to be synchronous to disk on the OSS, as Patrick 
reports, because if the OSS fails before the write is on disk there is no 
cached copy of the data on the client that can be used to resend the RPC.

The problem is that the ZFS OSD has very long transaction commit times for 
synchronous writes because it does not yet have support for the ZIL.  Using 
buffered writes, or having very large O_DIRECT writes (e.g. 40MB or larger) and 
large RPCs (4MB, or up to 16MB in 2.9.0) to amortize the sync overhead may be 
beneficial if you really want to use O_DIRECT.

The other potential issue is that you have 20 OSTs on a single OSS, which isn't 
going to have very good performance.  Spreading the OSTs across multiple OSS 
nodes is going to improve your performance significantly when there are 
multiple clients writing, as there will be N times the OSS network bandwidth, N 
times the CPU, N times the RAM.  It only makes sense to have 20 OSTs/OSS if 
your workload is only a single client and you want the maximum possible 
capacity for a given cost.

Is each OST a separate VDEV and separate zpool, or are they a single zpool?  
Separate zpools have less overhead for maximum performance, but only one VDEV 
per zpool means that metadata ditto blocks are written twice per RAID-Z2 VDEV, 
which isn't very efficient.  Having at least 3 VDEVs per zpool is better in 
this regard.

Cheers, Andreas
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division

On 2016/10/14, 15:22, "John Bauer" 
<<>> wrote:

I thought at one time there was an inode lock held for the duration of the 
direct I/O read or write. So that even if one had multiple application threads 
writing direct, only one was "in flight" at a time. Has that changed?

Sent from my iPhone

On Oct 14, 2016, at 3:16 PM, Patrick Farrell 
<<>> wrote:

Sorry, I phrased one thing wrong:
I said "transferring to the network", but it's actually until it's received 
confirmation the data has been received successfully, I believe.

In any case, only one I/O (per thread) can be outstanding at a time with direct 

From: lustre-discuss 
 on behalf of Patrick Farrell <<>>
Sent: Friday, October 14, 2016 3:12:22 PM
To: Riccardo Veraldi;<>
Subject: Re: [lustre-discuss] Lustre on ZFS pooer direct I/O performance


While the difference is extreme, direct I/O write performance will always be 
poor.  Direct I/O writes cannot be asynchronous, since they don't use the page 
cache.  This means Lustre cannot return from one write (and start the next) 
until it has finished transferring the data to the network.

This means you can only have one I/O in flight at a time.  Good write 
performance from Lustre (or any network filesystem) depends on keeping a lot of 
data in flight at once.

What sort of direct write performance were you hoping for?  It will never match 
that 800 MB/s from one thread you see with buffered I/O.

- Patrick

From: lustre-discuss 
 on behalf of Riccardo Veraldi 
Sent: Friday, October 14, 2016 2:22:32 PM
Subject: [lustre-discuss] Lustre on ZFS pooer direct I/O performance


I would like how may I improve the situation of my lustre cluster.

I have 1 MDS and 1 OSS with 20 OST defined.

Each OST is a 8x Disks RAIDZ2.

A single process write performance is around 800MB/sec

anyway if I force direct I/O, for example using oflag=direct in dd, the
write performance drop as low as 8MB/sec

with 1MB block size. And each write it's about 120ms latency.

I used these ZFS settings

options zfs zfs_prefetch_disable=1
options zfs zfs_txg_history=120
options zfs metaslab_debug_unload=1

i am quite worried for the low performance.

Any hints or suggestions that may help me to improve the situation ?

thank you


lustre-discuss mailing list<>
lustre-discuss mailing list<>
lustre-discuss mailing list

Reply via email to