Hello, Eric. Thanks for your very comprehensive answer. I believe it's great it'll be in archives for future users. I haven't decided how to go completely, but have several questions here.
You mentioned: > 10.100.255.3:/storage/primary3 on /mnt/0ab13de9-2310-334c-b438-94dfb0b8ec84 type nfs4 (rw,relatime,sync,vers=4.0,rsize=1048576,wsize=1048576, namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,proto= tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.100. 255.2,local_lock=none,addr=10.100.255.3) But I see at my Ubuntu 14.04 client and ACS 4.9: 192.168.3.218:/export/primary on /mnt/4e5a8e18-a425-32cc-a78b-30535229efef type nfs (rw,vers=4,addr=192.168.3.218,clientaddr=192.168.3.220) Any Idea what's wrong? It's just does as I copied above... The other thing is it's not a question works it or not out of the box, but how to tweak rsize/wsize anyway to make it work better, also, you have "sync", but our other ACS (4.3 with Ubuntu 14.04) uses "async". Where it comes from? 2017-10-20 14:12 GMT+07:00 Eric Green <eric.lee.gr...@gmail.com>: > Okay. So: > > 1) Don't use EXT4 with LVM/RAID, it performs terribly with QCOW2. Use XFS. > 2) I didn't do anything to my NFS mount options and they came out fine: > > 10.100.255.3:/storage/primary3 on /mnt/0ab13de9-2310-334c-b438-94dfb0b8ec84 > type nfs4 (rw,relatime,sync,vers=4.0,rsize=1048576,wsize=1048576, > namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,proto= > tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.100. > 255.2,local_lock=none,addr=10.100.255.3) > > 3) ZFS works relatively well, but if you are using it with SSD's you > *must* set the alignment to match the SSD alignment, and *must* set the ZFS > block size to match the SSD internal block size, else performance is > terrible. For performance, use LVM/XFS/RAID. *EXCEPT* it is really hard to > make that perform on SSD's too, if you have a block size and alignment > mismatch performance will be *terrible* with LVM/XFS/RAID. > > 4) Hardware RAID with a battery backup tends to result in much faster > writes *if writing to rotating storage* and only if using Linux LVM/XFS. > ZFS does its own redundancy better, so don't use BBU with ZFS. If writing > to SSD's, you will get better performance with the Linux MD RAID or ZFS, > but note you must be *very* careful about block sizes and alignment. Don't > use a BBU hardware raid with SSD's, performance will be terrible compared > to Linux MD RAID or ZFS for a number of reasons. > > 5) Needless to say the ZFS NFS shares work fine, *if* you've done your > homework and set them up well with proper alignment and block size for your > hardware. However, for rotational storage the LVM/XFS/RAID will be faster, > especially with the hardware raid and BBU. > > My own CloudStack implementation has two storage servers that use > LVM/XFS/RAID for storage and one storage server that uses ZFS for storage. > The ZFS server has two 12-disk RAID groups, one made up of SSD's for a > database, the other made up of large rotational storage drives. It also has > a NVMe card that is used for log and cache for the rotational storage. I > spent a *lot* of time trying to get the SSD's to perform under RAID/LVM/XFS > and just couldn't get everything to agree on alignment. That was when I > said foo on that and put ZFS on there, and since I was using ZFS for one > RAID group, it made sense to use it on the other too. I'm using RAID10 on > the SSD RAID group, and RAIDZ2 on the rotational storage (which is there > for bulk storage where performance isn't a priority). > > Storage is not a limitation on my cloud, especially since I have four > other storage servers that I can throw at it if necessary. RAM and CPU are, > so that's my next task -- get more compute servers into the Cloudstack > cluster. > > > > On Oct 19, 2017, at 22:16, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> > wrote: > > > > Hello, friends. > > > > I'm planning to deploy new cluster with KVM and shared NFS storage soon. > > Right now I already have such deploys that operate fine. Currently, I'm > > trying to compare different storage options for new cluster between > > - LVM+EXT4+HWRAID+BBU, > > - LVM+EXT4+MDADM and > > - ZFS. > > > > HW setup is: > > - 5 x Micron 5100PRO 1.9TB > > - LSI9300-8i HBA or LSI9266-8i+BBU > > - Cisco UCS server w/ 24x2.5" / 2xE5-2650 / 64GB RAM > > > > I have got competitive performance results between them locally already, > > but now I need to test over NFS. I'm pretty sure that first two options > > will operate nice with ACS default NFS mount args (because I already have > > such cases in prod), but ZFS is quite smart thing, so I started to > > investigate how to change NFS client mount options and unfortunately > > haven't succeed defining the proper place where cloudstack agent > determines > > how to mount share and what args to use. I read a lot of ZFS-related > > articles and people write rsize/wsize affect quite much, so I wonder how > to > > instruct cloudstack agent to use specific rsize/wsize args to mount > primary > > storage. > > > > Also, I haven't found in ACS archives mentions about ZFS NFS share, so > > might be it's a bad case for ACS because of Qcow image format?, but I > think > > it could be a good one so want to test personally. > > > > Any suggestions are welcome. > > > > -- > > With best regards, Ivan Kudryavtsev > > Bitworks Software, Ltd. > > Cell: +7-923-414-1515 > > WWW: http://bitworks.software/ <http://bw-sw.com/> > > -- With best regards, Ivan Kudryavtsev Bitworks Software, Ltd. Cell: +7-923-414-1515 WWW: http://bitworks.software/ <http://bw-sw.com/>
