[zfs-discuss] Send Receive (and why does 'ls' modify a snapshot?)
I am trying to keep a file system (actually quite a few) in sync across two systems for DR purposes, but I am encountering something that I find strange. Maybe its not strange, and I just don't understand - but I will pose to you fine people to help answer my question. This is all scripted, but I have pulled out the relevant commands for your reference: I have a file system on localnode and I create a snapshot call NMINUS1 which I then send to my remotenode. zfs snapshot GT/t...@nminus1 zfs send GT/t...@nminus1 | ssh remotenode zfs receive GT/t...@nminus1 This works properly, I then get the mountpoint property of the filesystem on the localnode and using that, set it to the same on the remote node: $mp = zfs get -Ho value mountpoint GT/test ssh remotenode zfs set mountpoint=$mp GT/test Again, works fine and this completes my inital setup. From that point onwards, I want to send an incremental snapshot on a, say, nightly basis. So I create a new snapshot (NSNAP), send that across, and then remove the old snap and rename the new to NMINUS1 ... so zfs snapshot GT/t...@nsnap zfs send -i NMINUS1 GT/t...@nsnap | ssh remotenode zfs receive GT/test -- On both nodes zfs destroy GT/t...@nminus1 zfs rename GT/t...@nsnap GT/t...@nminus1 Now everything works fine unless I perform a simple 'ls' on the filesystem on the remote node. On the local node I can modify the contents of GT/test at any time, add or remove files, etc. and when I send the incremental snapshot to the remote node, it completes properly, and I can do this as many times as I want, but as soon as I issue that # ls /GT/test on the remote node, the next time I try to send an incremental snapshot I get the following error: # zfs send -i NMINUS1 GT/t...@nsnap | ssh remotenode zfs receive GT/test cannot receive incremental stream: destination GT/ahg has been modified since most recent snapshot Other than modifying possibly access time - what has been change in the snapshot that causes this problem?? One item of note (or not) one system is SPARC one is AMD based. Thanks for any ideas. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Send Receive (and why does 'ls' modify a snapshot?)
Thanks ... the -F works perfectly, and provides a further benefit in that the client can mess with the file system as much as they want for testing purposes, but when it comes time to ensure it is synchronized each night, it will revert back to the previous state. Thanks -Tony -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Zpool, RaidZ how it spreads its disk load?
Greetings learned ZFS geeks guru’s, Yet another question comes from my continued ZFS performance testing. This has to do with zpool iostat, and the strangeness that I do see. I’ve created an eight (8) disk raidz pool from a Sun 3510 fibre array giving me a 465G volume. # zpool create tp raidz c4t600 ... 8 disks worth of zpool # zfs create tp/pool # zfs set recordsize=8k tp/pool # zfs set mountpoint=/pool tp/pool I then create a 100G data file that is created by sequentially writing 64k blocks to the test data file. When I then issue a # zpool iostat -v tp 10 I see the following strange behaviour. I see anywhere from up to 16 iterations (ie 160 seconds) of the following, where there are only writes to 2 of the 8 disks: capacity operationsbandwidth pool used avail read write read write -- - - - - - - testpool29.7G 514G 0 2.76K 0 22.1M raidz129.7G 514G 0 2.76K 0 22.1M c4t600C0FF00A74531B659C5C00d0s6 - - 0 0 0 0 c4t600C0FF00A74533F3CF1AD00d0s6 - - 0 0 0 0 c4t600C0FF00A74534C5560FB00d0s6 - - 0 0 0 0 c4t600C0FF00A74535E50E5A400d0s6 - - 0 1.38K 0 2.76M c4t600C0FF00A74537C1C061500d0s6 - - 0 0 0 0 c4t600C0FF00A745343B08C4B00d0s6 - - 0 0 0 0 c4t600C0FF00A745379CB90B600d0s6 - - 0 0 0 0 c4t600C0FF00A74530237AA9300d0s6 - - 0 1.38K 0 2.76M -- - - - - - - During these periods, my data file does not grow in size, but then I see writes to all of the disks like the following: capacity operationsbandwidth pool used avail read write read write -- - - - - - - testpool64.0G 480G 0 1.45K 0 11.6M raidz164.0G 480G 0 1.45K 0 11.6M c4t600C0FF00A74531B659C5C00d0s6 - - 0246 0 8.22M c4t600C0FF00A74533F3CF1AD00d0s6 - - 0220 0 8.23M c4t600C0FF00A74534C5560FB00d0s6 - - 0254 0 8.20M c4t600C0FF00A74535E50E5A400d0s6 - - 0740 0 1.45M c4t600C0FF00A74537C1C061500d0s6 - - 0299 0 8.21M c4t600C0FF00A745343B08C4B00d0s6 - - 0284 0 8.21M c4t600C0FF00A745379CB90B600d0s6 - - 0266 0 8.22M c4t600C0FF00A74530237AA9300d0s6 - - 0740 0 1.45M -- - - - - - - And my data file will increase in size, but also notice notice, in the above, those disks that were being written to before, have a load that is consistent with the previous example. For background, the server, and the storage are dedicated solely to this testing, and there are no other applications being run at this time. I thought that RaidZ would spread its load across all disks somewhat evenly. Can someone explain this result? I can consistently reproduce it as well. Thanks -Tony This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
RE: [zfs-discuss] Testing of UFS, VxFS and ZFS
The volume is 7+1. I have created the volume using both the default (DRL) as well as 'nolog' to turn it off, both with similar performance. On the advice of Henk, after he had looked over my data, he is notice that the veritas test seems to be almost entirely using file system cache. I will retest with a much larger file to defeat this cache (I do not want to modify my mount options). If this then shows similar performance (I will also retest with ZFS with the same file size) then the question will probably have more to do with how ZFS handles file system caching. -Tony -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: April 16, 2007 2:16 PM To: [EMAIL PROTECTED] Subject: Re: [zfs-discuss] Testing of UFS, VxFS and ZFS Is the VxVM volume 8-wide? It is not clear from your creation commands. -- richard Tony Galway wrote: I had previously undertaken a benchmark that pits out of box performance of UFS via SVM, VxFS and ZFS but was waylaid due to some outstanding availability issues in ZFS. These have been taken care of, and I am once again undertaking this challenge on behalf of my customer. The idea behind this benchmark is to show a. How ZFS might displace the current commercial volume and file system management applications being used. b. The learning curve of moving from current volume management products to ZFS. c. Performance differences across the different volume management products. VDBench is the test bed of choice as this has been accepted by the customer as a telling and accurate indicator of performance. The last time I attempted this test it had been suggested that VDBench is not appropriate to testing ZFS, I cannot see that being a problem, VDBench is a tool - if it highlights performance problems, then I would think it is a very effective tool so that we might better be able to fix those deficiencies. Now, to the heart of my problem! The test hardware is a T2000 connected to a 12 disk SE3510 (presenting as JBOD) through a brocade switch, and I am using Solaris 10 11/06. For Veritas, I am using Storage Foundation Suite 5.0. The systems were jumpstarted to the same configuration before testing a different volume management software to ensure there were no artifacts remaining from any previous test. I present my vdbench definition below for your information: sd=FS,lun=/pool/TESTFILE,size=10g,threads=8 wd=DWR,sd=FS,rdpct=100,seekpct=80 wd=ETL,sd=FS,rdpct=0, seekpct=80 wd=OLT,sd=FS,rdpct=70, seekpct=80 rd=R1-DWR,wd=DWR,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R1-ETL,wd=ETL,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R1-OLT,wd=OLT,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R2-DWR,wd=DWR,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R2-ETL,wd=ETL,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R2-OLT,wd=OLT,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R3-DWR,wd=DWR,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R3-ETL,wd=ETL,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) rd=R3-OLT,wd=OLT,iorate=max,elapsed=1800,interval=30,forxfersize=(1k,2k,4k,8 k,16k,32k,64k,128k) As you can see, it is fairly straight forward and I take the average of the three runs in each of ETL, OLT and DWR workloads. As an aside, I am also performing this test for various file system block sizes as applicable as well. I then ran this workload against a Raid-5 LUN created and mounted in each of the different file system types. Please note that one of the test criteria is that the associated volume management software create the Raid-5 LUN, not the disk subsystem. 1. UFS via SVM # metainit d20 -r d1 . d8 # newfs /dev/md/dsk/d20 # mount /dev/md/dsk/d20 /pool 2. ZFS # zfs create pool raidz d1 . d8 3. VxFS - Veritas SF5.0 # vxdisk init SUN35100_0 .. SUN35100_7 # vxdg init testdg SUN35100_0 . # vxassist -g testdg make pool 418283m layout=raid5 Now to my problem - Performance! Given the test as defined above, VxFS absolutely blows the doors off of both UFS and ZFS during write operations. For example, during a single test on an 8k file system block, I have the following average IO Rates: *ETL * *OLTP * *DWR * *UFS * 390.00 1298.44 23173.60 *VxFS * 15323.10 27329.04 22889.91 *ZFS * 2122.23 7299.36 22940.63 If you look
[zfs-discuss] ZFS Performance compared to UFS VxFS
A question (well lets make it 3 really) – Is vdbench a useful tool when testing file system performance of a ZFS file system? Secondly - is ZFS write performance really much worse than UFS or VxFS? and Third - what is a good benchmarking tool to test ZFS vs UFS vs VxFS? The reason I ask is this – I am using the following – very simple write only test scenario using vdbench - sd=ZFS,lun=/pool/TESTFILE,size=10g,threads=8 wd=ETL,sd=ZFS,rdpct=0, seekpct=80 rd=ETL,wd=ETL,iorate=max,elapsed=1800,interval=5,forxfersize=(1k,4k,8k,32k) For those not familiar with vdbench - this scenario reads as follows: SD = Storage Definition = A 10g file in on my ZFS directory - and use 8 concurrent threads for the test which indicate the maximum concurrent IO's for this Storage definition WD = Workload Definition - Use the ZFS storage pool, with rdpct = 0 - means 100% write, and seekpct = 80 means that 80% of the IO ops will start a new random seek address. RD = Run Definition = This is the actual run - we want to run 4 tests with 1, 4, 8 32k blocks - report back to me every 5 seconds what's happening, and iorate = max says 'have at er' - run this thing to the ground if possible - as I am not simulating a specific workload per se but trying to see what the maximum performance I might see would be. So - I am seeing disheartening results. My test configuration is as follows: T2000 / 32Gb connected via 2 x 2GB fibre connections to a 3510 array which has 5 disks assigned to the host acting only as a JBOD. This configuration (including 3510) is dedicated to testing performance of ZFS vs SVM with UFS vs VxVM with VxFS – but note that at this point the veritas packages have not been added yet – we do not want any perception that another package may impact performance of any of the tests. This is a Solaris 10 06/06 install – with all the latest patches and no out of box optimizations performed. The first suite of tests is to only see differences in out of box performance and manageability. The results I am seeing are this – read performance is acceptable and on par with UFS, however when it comes to write – based on the performance I am seeing - the system seems to not be able to keep up with the writes. I am seeing large periods of time where is no reported activity, and if I am looking at zfs iostat I do see consistent writing however - also if I terminate the test - zfs will continue to write for about 2 - 3 minutes after all other activity has stopped. The following is an example of what I see - Example 1 is from the initial file create, example 2 is from the actual test run. What is most important to note are the lines with big gapping 0's running through them -and the associated large response times surrounding them. If anyone has any information on ZFS performance - it would be apprecated. 12:40:59.815 Starting RD=New_file_format_for_sd=ZFS; I/O rate: 5000; Elapsed: 36 seconds. For loops: xfers ize=65536 threads=8 intervali/o MB/sec bytes read resp resp respcpu% cpu% rate 1024**2 i/opct time max stddev sys+usr sys 12:41:03.362 14858.99 303.69 65536 0.001.411 59.843 2.46821.4 16.7 12:41:04.101 2 878.8554.93 65536 0.002.219 98.016 8.79513.1 7.5 12:41:05.069 3 147.65 9.23 65536 0.00 74.181 1022.374 232.319 9.7 4.0 12:41:06.056 4 160.9710.06 65536 0.00 52.750 425.264 105.263 8.0 3.8 12:41:07.085 51025.1064.07 65536 0.008.832 492.696 50.99510.2 5.6 12:41:08.077 6 664.6741.54 65536 0.001.5502.963 0.511 6.3 5.6 12:41:09.113 71740.72 108.79 65536 0.006.627 1139.357 75.57310.9 9.9 12:41:10.160 8 111.72 6.98 65536 0.00 98.972 1426.363 361.047 6.0 3.5 12:41:11.029 93888.81 243.05 65536 0.001.6537.255 0.68720.3 18.0 12:41:12.07110 0.00 0.00 0 0.000.0000.000 0.000 3.5 3.2 12:41:13.04511 0.00 0.00 0 0.000.0000.000 0.000 3.4 3.2 12:41:14.07512 0.00 0.00 0 0.000.0000.000 0.000 3.3 2.8 12:41:15.034134183.62 261.48 65536 0.008.080 3230.801 143.61119.6 19.1 12:41:16.071141238.0277.38 65536 0.006.348 136.478 10.50612.1 9.9 12:41:17.057151136.1371.01 65536 0.006.214 102.408 11.09410.1 9.5 12:41:18.05316 805.2050.32 65536 0.008.028 134.882 16.480 7.9 7.6 12:41:19.04017 0.00 0.00 0 0.000.0000.000 0.000 3.3 3.2 12:41:20.03918 0.00 0.00 0 0.000.0000.000
