Re: [zfs-discuss] Can't offline a RAID-Z2 device: no valid replica
You're right, from the documentation it definitely should work. Still, it doesn't. At least not in Solaris 10. But i am not a zfs-developer, so this should probably answered by them. I will give it a try with a recent OpneSolaris-VM and check, wether this works in newer implementations of zfs. The pool is not using the disk anymore anyway, so (from the zfs point of view) there is no need to offline the disk. If you want to stop the io-system from trying to access the disk, pull it out or wait until it gives up... Yes, there is. I don't want the disk to become online if the system reboots, because what actually happens is that it *never* gives up (well, at least not in more than 24 hours), and all I/O to the zpool stop as long as there are those errors. Yes, I know it should continue working. In practice, it does not (though it used to be much worse in previous versions of S10, with all I/O stopping on all disks and volumes, both ZFS and UFS, and usually ending in a panic). And the zpool command hangs, and never finished. The only way to get out of it is to use cfgadm to send multiple hardware resets to the SATA device, then disconnect it. At this point, zpool completes and shows the disk as having faulted. Again you are right, that this is a very annoying behaviour. the same thing happens with DiskSuite pools and ufs when a disk is failing as well, though. For me it is not a zfs problem, but a Solaris problem. The kernel should stop trying to access failing disks a LOT earlier instead of blocking the complete I/O for the whole system. I always understood zfs as a concept for hot pluggable disks. This is the way i use it and that is why i never really had this problem. Whenever i run into this behaviour, i simply pull the disk in question and replace it. The time those hickups affect the performance of our production eviroment have never been longer than a couple of minutes. Tom -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Can't offline a RAID-Z2 device: no valid replica
FYI: In b117 it works as expected and stated in the documentation. Tom -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Can't offline a RAID-Z2 device: no valid replica
You can't replace it because this disk is still a valid member of the pool, although it is marked faulty. Put in a replacement disk, add this to the pool and replace the faulty one with the new disk. Regards, Tom -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Can't offline a RAID-Z2 device: no valid replica
You could offline the disk if [b]this[/b] disk (not the pool) had a replica. Nothing wrong with the documentation. Hmm, maybe it is little misleading here. I walked into the same trap. The pool is not using the disk anymore anyway, so (from the zfs point of view) there is no need to offline the disk. If you want to stop the io-system from trying to access the disk, pull it out or wait until it gives up... -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Avoiding performance decrease when pool usage is over 80%
Ralf Ramge schrieb: Thomas Liesner wrote: Does this mean, that if i have a pool of 7TB with one filesystem for all users with a quota of 6TB i'd be alright? Yep. Although I *really* recommend creating individual file systems, e.g. if you have 1,000 users on your server, I'd create 1,000 file systems with a quota of 6 GB each. Easier to handle, more flexible to use, easier to backup, it allows better use of snapshots and it's easier to migrate single users to other servers. Thanks for your recommendation, still this would not meet our needs. All the data in the production pool must be accessible to all users on this system and will be worked on by all users on this system. Hence, one shared fs for all users is perfectly fine. Thanks for all your input, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Avoiding performance decrease when pool usage is over 80%
bda wrote: I haven't noticed this behavior when ZFS has (as recommended) the full disk. Good to know, as i intended to use the whole disks anyway. Thanks, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Avoiding performance decrease when pool usage is over 80%
Ralf Ramge wrote: Quotas are applied to file systems, not pools, and a such are pretty independent from the pool size. I found it best to give every user his/her own filesystem and applying individual quotas afterwards. Does this mean, that if i have a pool of 7TB with one filesystem for all users with a quota of 6TB i'd be alright? The usage of that fs would never be over 80%, right? Like in the following example for the pool shares with a poolsize of 228G an one fs with a quota of 100G: shares 228G28K 220G 1%/shares shares/production 100G 8,4G92G 9%/shares/production This would suite me perfectly, as this would be exactly what i wanted to do ;) Thanks, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Avoiding performance decrease when pool usage is over 80%
Nobody out there who ever had problems with low diskspace? Regrads, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] We can't import pool zfs faulted
If you can't use zpool status, you probably should check wether your system is right and not all devices needed for this pool are currently available... i.e. format... Regards, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Avoiding perfromance decrease when pool over 80% usage
Hi all, i am planning a zfs-fileserver for a larger prepress-company in Germany. Knowing that users tend to use all the space they can get, i am looking for a solution to avoid a rapid performance loss when the production-pool is more than 80% used. Would it be a practical solution to just set the quota of the pool to something below 80% of the available space? Regards, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Sun's storage product roadmap?
Hi, from sun germany i got the info hat the 2u JBODs wille be officially announced in q1 2008 and the 4u JBODs in q2 2008. Both will have SAS connectors and support either SAS and SATA drives. Ragards, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS is very slow in our test, when the capacity is high
Hi, did you read the following? http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide Currently, pool performance can degrade when a pool is very full and filesystems are updated frequently, such as on a busy mail server. Under these circumstances, keep pool space under 80% utilization to maintain pool performance. I wonder if defining a zfs quota of roughly 80% of the whole pool capacity would help to keep performance up. Users always use all the space available. Regards, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Fileserver performance tests
Hi, compression is off. I've checked rw-perfomance with 20 simultaneous cp and with the following... #!/usr/bin/bash for ((i=1; i=20; i++)) do cp lala$i lulu$i done (lala1-20 are 2gb files) ...and ended up with 546mb/s. Not too bad at all. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Which SAS JBOD-enclosure
Hi all, i am currently using two XStore XJ 1100 SAS JBOD enclosures(http://www.xtore.com/product_detail.asp?id_cat=11) attached to a x4200 for testing. So far it works rather nicly, but i am still looking for alternatives. The Infortrend JBOD-expansions are not deliverable at the moment. What else is out there on the market? Regards, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Fileserver performance tests
Hi Eric, Are you talking about the documentation at: http://sourceforge.net/projects/filebench or: http://www.opensolaris.org/os/community/performance/filebench/ and: http://www.solarisinternals.com/wiki/index.php/FileBench ? i was talking about the solarisinternals wiki. I can't find any documentation at the sourceforge site and the opensolaris site refers to solarisinternals for a more detailed documentation. The INSTALL document within the distribution refers to solarisinternals and pkgadd which of course isn't working without providing a package ;) This is the output of make within filebench/filebench: [EMAIL PROTECTED] # make make: Warning: Can't find `../Makefile.cmd': Datei oder Verzeichnis nicht gefunden make: Fatal error in reader: Makefile, line 27: Read of include file `../Makefile.cmd' failed Before looking at the results, decide if that really *is* your expected workload Sure enough i have to dig deeper into the filebench workloads and create my own workload to represent my expected workload even better, but the tasks within the fileserver workload are already quite representative (i could skip the append test though...) Regards, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Fileserver performance tests
Hi again, i did not want to compare the filebench test with the single mkfile command. Still, i was hoping to see similar numbers in the filbench stats. Any hints what i could do to further improve the performance? Would a raid1 over two stripes be faster? TIA, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Fileserver performance tests
Hi, i checked with $nthreads=20 which will roughly represent the expected load and these are the results: IO Summary: 7989 ops 7914.2 ops/s, (996/979 r/w) 142.7mb/s,255us cpu/op, 0.2ms latency BTW, smpatch is still running and further tests will get done when the system is rebooted. The figures published at... http://blogs.sun.com/timthomas/feed/entries/atom?cat=%2FSun+Fire+X4500 ...made me expect to see higher rates with my setup. I have seen the new filebench at sourceforge, but did not manage to install. It's a source ditrsibution now and the wiki and readmes are not updated yet. A simple make didn't do the trick though ;) Thanks again, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Fileserver performance tests
Hi, i checked with $nthreads=20 which will roughly represent the expected load and these are the results: IO Summary: 7989 ops 7914.2 ops/s, (996/979 r/w) 142.7mb/s, 255us cpu/op, 0.2ms latency BTW, smpatch is still running and further tests will get done when the system is rebooted. The figures published at... http://blogs.sun.com/timthomas/feed/entries/atom?cat=%2FSun+Fire+X4500 ...made me expect to see higher rates with my setup. I have seen the new filebench at sourceforge, but did not manage to install. It's a source ditrsibution now and the wiki and readmes are not updated yet. A simple make didn't do the trick though ;) Thanks again, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Fileserver performance tests
i wanted to test some simultanious sequential writes and wrote this little snippet: #!/bin/bash for ((i=1; i=20; i++)) do dd if=/dev/zero of=lala$i bs=128k count=32768 done While the script was running i watched zpool iostat and measured the time between starting and stopping of the writes (usually i saw bandwth figures around 500...) The result was 409 mb/s in writes. Not too bad at all :) Now the same with sequential reads: #!/bin/bash for ((i=1; i=20; i++)) do dd if=lala$i of=/dev/zero bs=128k done again checked with zpool iostat seeing even higher numbers around 850 and the result was 910mb/s... wow that all looks quite promising :) Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Fileserver performance tests
Hi all, i want to replace a bunch of Apple Xserves with Xraids and HFS+ (brr) by Sun x4200 with SAS-Jbods and ZFS. The application will be the Helios UB+ fileserver suite. I installed the latest Solaris 10 on a x4200 with 8gig of ram and two Sun SAS controllers, attached two sas-jbods with 8 SATA-HDDs each und created a zfs pool as a raid 10 by doing something like the following: [i]zpool create zfs_raid10_16_disks mirror c3t0d0 c4t0d0 mirror c3t1d0 c4t1d0 mirror c3t2d0 c4t2d0 mirror c3t3d0 c4t3d0 mirror c3t4d0 c4t4d0 mirror c3t5d0 c4t5d0 mirror c3t6d0 c4t6d0 mirror c3t7d0 c4t7d0[/i] the i set noatime and ran the following filebench tests: [i] [EMAIL PROTECTED] # ./filebench filebench load fileserver 12746: 7.445: FileServer Version 1.14 2005/06/21 21:18:52 personality successfully loaded 12746: 7.445: Usage: set $dir=dir 12746: 7.445:set $filesize=sizedefaults to 131072 12746: 7.445:set $nfiles=value defaults to 1000 12746: 7.445:set $nthreads=value defaults to 100 12746: 7.445:set $meaniosize=value defaults to 16384 12746: 7.445:set $meandirwidth=size defaults to 20 12746: 7.445: (sets mean dir width and dir depth is calculated as log (width, nfiles) 12746: 7.445: 12746: 7.445:run runtime (e.g. run 60) 12746: 7.445: syntax error, token expected on line 43 filebench set $dir=/zfs_raid10_16_disks/test filebench run 60 12746: 47.198: Fileset bigfileset: 1000 files, avg dir = 20.0, avg depth = 2.3, mbytes=122 12746: 47.218: Removed any existing fileset bigfileset in 1 seconds 12746: 47.218: Creating fileset bigfileset... 12746: 60.222: Preallocated 1000 of 1000 of fileset bigfileset in 14 seconds 12746: 60.222: Creating/pre-allocating files 12746: 60.222: Starting 1 filereader instances 12751: 61.228: Starting 100 filereaderthread threads 12746: 64.228: Running... 12746: 65.238: Run took 1 seconds... 12746: 65.266: Per-Operation Breakdown statfile1 988ops/s 0.0mb/s 0.0ms/op 22us/op-cpu deletefile1 991ops/s 0.0mb/s 0.0ms/op 48us/op-cpu closefile2997ops/s 0.0mb/s 0.0ms/op4us/op-cpu readfile1 997ops/s 139.8mb/s 0.2ms/op 175us/op-cpu openfile2 997ops/s 0.0mb/s 0.0ms/op 28us/op-cpu closefile1 1081ops/s 0.0mb/s 0.0ms/op6us/op-cpu appendfilerand1 982ops/s 14.9mb/s 0.1ms/op 91us/op-cpu openfile1 982ops/s 0.0mb/s 0.0ms/op 27us/op-cpu 12746: 65.266: IO Summary: 8088 ops 8017.4 ops/s, (997/982 r/w) 155.6mb/s,508us cpu/op, 0.2ms 12746: 65.266: Shutting down processes filebench[/i] I expected to see some higher numbers really... a simple time mkfile 16g lala gave me something like 280Mb/s. Would anyone comment on this? TIA, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] SAS-controller recommodations
Hi all, i am about to put together a one month test configuration for a graphics-production server (prepress-filer that is). I would like to test zfs on a x4200 with two sas2sata-jbods attached. Initially i wanted to use an infortrend fc2sata-jbod-enclosure but these are at out of production and 2gbit only. The alternative would be tthe menitoned sas2sata-jbod (S12S-J1002-R) but i would need two pci-e sas-controller - without raid functionality - known to run extremly well under solaris x86. Any hints? TIA, Tom This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
