Re: NFS - slow
same. am i doing something wrong? I found NFSv4 to be much *slower* than NFSv3 on FreeBSD, when I benchmarked it a year or so ago. both are just right in you read (NFSv4 taking a bit more CPU), and both are awful at writes. for me now the only way to get NFS working well is to use unfsd with fsync commented out in sources. get very fast and not compliant with NFS protocol, which i don't care. i don't have database logs on NFS, but sometimes need to run non freebsd device without HDD and compile something using NFS disk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
David Brodbeck wrote: On Mon, Apr 30, 2012 at 10:00 PM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: i tried nfsv4, tested under FreeBSD over localhost and it is roughly the same. am i doing something wrong? I found NFSv4 to be much *slower* than NFSv3 on FreeBSD, when I benchmarked it a year or so ago. If delegations are not enabled, there is additional overhead doing the Open operations against the server. Delegations are not enabled by default in the server, because there isn't code to handle conflicts with opens done locally on the server. (ie. Delegations work iff the volumes exported over NFSv4 are not accessed locally in the server.) I think there are also some issues w.r.t. name caching in the client that still need to be resolved. NFSv4 should provide better byte range locking, plus NFSv4 ACLs and a few other things. However, it is more complex and will not perform better than NFSv3, at least until delegations are used (or pNFS, which is a part of NFSv4.1). rick -- David Brodbeck System Administrator, Linguistics University of Washington ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
On Mon, Apr 30, 2012 at 10:00 PM, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: i tried nfsv4, tested under FreeBSD over localhost and it is roughly the same. am i doing something wrong? I found NFSv4 to be much *slower* than NFSv3 on FreeBSD, when I benchmarked it a year or so ago. -- David Brodbeck System Administrator, Linguistics University of Washington ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
Probably not. NFSv4 writes are done exactly the same as NFSv3. (It changes other stuff, like locking, adding support for ACLs, etc.) I do have a patch is there any chance to improving it? i mean to cluster writes up to MAXBSIZE on nfs server, just like local UFS do? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
new (default on 9) sets it to MAXBSIZE, which is currently 64K, but I would like to get that increased. (A quick test suggested that the kernel works when MAXBSIZE is set to 128K, but I haven't done much testing yet.) From a long time my kernels have MAXBSIZE of 2 megabytes which is right value for modern hard drives and gives FAR better performance. I don't know why it isn't default. I will test NFSv4 again today. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
Wojciech Puchar wrote: i tried nfsv4, tested under FreeBSD over localhost and it is roughly the same. am i doing something wrong? Probably not. NFSv4 writes are done exactly the same as NFSv3. (It changes other stuff, like locking, adding support for ACLs, etc.) I do have a patch that allows the client to do more extension caching to local disk in the client (called Packrats), but that isn't ready for prime time yet. NFSv4.1 optionally supports pNFS, where reading and writing can be done to Data Servers (DS) separate from the NFS (called Metadata Server or MDS). I`m working on the client side of this, but it is also a work-in-progress and no work on a NFSv4.1 server for FreeBSD has been done yet, as far as I know. If you have increased MAXBSIZE in both the client and server machines and use the new (experimental in 8.x) client and server, they will use a larger rsize, wsize for NFSv3 as well as NFSv4. (Capturing packets and looking at them in wireshark will tell you what the actual rsize, wsize is. A patch to nfsstat to get the actual mount options in use is another of my `to do`items. If anyone else wants to work on this, I`d be happy to help them. On Mon, 30 Apr 2012, Peter Jeremy wrote: On 2012-Apr-27 22:05:42 +0200, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: is there any way to speed up NFS server? ... - write works terribly. it performs sync on every write IMHO, You don't mention which NFS server or NFS version you are using but for traditional NFS, this is by design. The NFS server is stateless and NFS server failures are transparent (other than time-wise) to the client. This means that once the server acknowledges a write, it guarantees the client will be able to later retrieve that data, even if the server crashes. This implies that the server needs to do a synchronous write to disk before it can return the acknowledgement back to the client. -- Peter Jeremy Btw, For NFSv3 and 4, the story is slightly different than the above. A client can do writes with a flag that is either FILESYNC or UNSTABLE. For FILESYNC, the server must do exactly what the above says. That is, the data and any required metadata changes, must be on stable storage before the server replies to the RPC. For UNSTABLE, the server can simply save the data in memory and reply OK to the RPC. For this case, the client needs to do a separate Commit RPC later and the server must store the data on stable storage at that time. (For this case, the client needs to keep the data written UNSTABLE in its cache and be prepared to re-write it, if the server reboots before the Commit RPC is done.) - When any app. does a fsync(2), the client needs to do a Commit RPC if it has been doing UNSTABLE writes. Most clients, including FreeBSD, do writes with UNSTABLE. However, one limitation on the FreeBSD client is that it currently only keeps track of one contiguous modified byte range in a buffer cache block. When an app. in the client does non-contiguous writes to the same buffer cache block, it must write the old modified byte range to the server with FILESYNC before it copies the newly written data into the buffer cache block. This happens frequently for builds during the loader phase. (jhb and I have looked at this. I have an experimental patch that makes the modified byte range a list, but it requires changes to struct buf. I think it is worth persuing. It is a client side patch, since that is where things can be improved, if clients avoid doing FILESYNC or frequent Commit RPCs.) rick ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
On 2012-Apr-27 22:05:42 +0200, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: is there any way to speed up NFS server? ... - write works terribly. it performs sync on every write IMHO, You don't mention which NFS server or NFS version you are using but for traditional NFS, this is by design. The NFS server is stateless and NFS server failures are transparent (other than time-wise) to the client. This means that once the server acknowledges a write, it guarantees the client will be able to later retrieve that data, even if the server crashes. This implies that the server needs to do a synchronous write to disk before it can return the acknowledgement back to the client. -- Peter Jeremy pgpDukb68dqQA.pgp Description: PGP signature
Re: NFS - slow
Wojciech Puchar wrote: the server is required to do that. (ie. Make sure the data is stored on stable storage, so it can't be lost if the server crashes/reboots.) Expensive NFS servers can use non-volatile RAM to speed this up, but a generic FreeBSD box can't do that. Some clients (I believe ESXi is one of these) requests FILE_SYNC all the time, but all clients will do so sooner or later. If you are exporting ZFS volumes and don't mind violating the NFS RFCs and risking data loss, there is a ZFS option that helps. I don't use ZFS, but I think the option is (sync=disabled) or something like that. (ZFS folks can help out, if you want that.) Even using vfs.nfsrv.async=1 breaks the above. thank you for answering. i don't use or plan to use ZFS. and i am aware of this NFS feature but i don't understand - even with syncs disabled, why writes are not clustered. i always see 32kB writes in systat The old (default on NFSv3) server sets the maximum wsize to 32K. The new (default on 9) sets it to MAXBSIZE, which is currently 64K, but I would like to get that increased. (A quick test suggested that the kernel works when MAXBSIZE is set to 128K, but I haven't done much testing yet.) when running unfsd from ports it doesn't have that problem and works FASTER than kernel nfs. But you had taken out fsync() calls, which breaks the protocol, as above. rick ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
i tried nfsv4, tested under FreeBSD over localhost and it is roughly the same. am i doing something wrong? On Mon, 30 Apr 2012, Peter Jeremy wrote: On 2012-Apr-27 22:05:42 +0200, Wojciech Puchar woj...@wojtek.tensor.gdynia.pl wrote: is there any way to speed up NFS server? ... - write works terribly. it performs sync on every write IMHO, You don't mention which NFS server or NFS version you are using but for traditional NFS, this is by design. The NFS server is stateless and NFS server failures are transparent (other than time-wise) to the client. This means that once the server acknowledges a write, it guarantees the client will be able to later retrieve that data, even if the server crashes. This implies that the server needs to do a synchronous write to disk before it can return the acknowledgement back to the client. -- Peter Jeremy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
Wojciech Puchar wrote: is there any way to speed up NFS server? from what i noticed: - reads works fast and good, like accessed locally, readahead up to maxbsize works fine on large files etc. - write works terribly. it performs sync on every write IMHO, setting vfs.nfsrv.async=1 improves things SLIGHTLY, but still - writes are sent to hard disk every single block - no clustering. am i doing something wrong or is it that broken? Since I haven't seen anyone else answer this, I'll throw out my $0.00 worth one more time. (This topic comes up regularily on the mailing lists.) Not broken, it's just a feature of NFS. When the client says FILE_SYNC, the server is required to do that. (ie. Make sure the data is stored on stable storage, so it can't be lost if the server crashes/reboots.) Expensive NFS servers can use non-volatile RAM to speed this up, but a generic FreeBSD box can't do that. Some clients (I believe ESXi is one of these) requests FILE_SYNC all the time, but all clients will do so sooner or later. If you are exporting ZFS volumes and don't mind violating the NFS RFCs and risking data loss, there is a ZFS option that helps. I don't use ZFS, but I think the option is (sync=disabled) or something like that. (ZFS folks can help out, if you want that.) Even using vfs.nfsrv.async=1 breaks the above. Once you do this, when an application in a client does a successful fsync() and assumes the data is safely stored and then the server crashes, the data can still be lost. rick i tried user space nfs from ports, it's funny but it's performance is actually better after i removed fsync from code. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: NFS - slow
the server is required to do that. (ie. Make sure the data is stored on stable storage, so it can't be lost if the server crashes/reboots.) Expensive NFS servers can use non-volatile RAM to speed this up, but a generic FreeBSD box can't do that. Some clients (I believe ESXi is one of these) requests FILE_SYNC all the time, but all clients will do so sooner or later. If you are exporting ZFS volumes and don't mind violating the NFS RFCs and risking data loss, there is a ZFS option that helps. I don't use ZFS, but I think the option is (sync=disabled) or something like that. (ZFS folks can help out, if you want that.) Even using vfs.nfsrv.async=1 breaks the above. thank you for answering. i don't use or plan to use ZFS. and i am aware of this NFS feature but i don't understand - even with syncs disabled, why writes are not clustered. i always see 32kB writes in systat when running unfsd from ports it doesn't have that problem and works FASTER than kernel nfs. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
NFS - slow
is there any way to speed up NFS server? from what i noticed: - reads works fast and good, like accessed locally, readahead up to maxbsize works fine on large files etc. - write works terribly. it performs sync on every write IMHO, setting vfs.nfsrv.async=1 improves things SLIGHTLY, but still - writes are sent to hard disk every single block - no clustering. am i doing something wrong or is it that broken? i tried user space nfs from ports, it's funny but it's performance is actually better after i removed fsync from code. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
