[zfs-discuss] Re: While we're sharing server info...
So, since you are using the ahci driver, does your cfgadm output show sata devices too? =) This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Rsync update to ZFS server over SSH faster than over NFS?
On Mon, May 21, 2007 at 06:21:40PM -0500, Albert Chin wrote: On Mon, May 21, 2007 at 06:11:36PM -0500, Nicolas Williams wrote: On Mon, May 21, 2007 at 06:09:46PM -0500, Albert Chin wrote: But still, how is tar/SSH any more multi-threaded than tar/NFS? It's not that it is, but that NFS sync semantics and ZFS sync semantics conspire against single-threaded performance. What's why we have set zfs:zfs_nocacheflush = 1 in /etc/system. But, that's only helps ZFS. Is there something similar for NFS? NFS's semantics for open() and friends is that they are synchronous, whereas POSIX's semantics are that they are not. You're paying for a sync() after every open. I'm not sure the semantics of NFS are at all relevant for the complete performance picture. NFS writes are(/used to be) synchronous, but the client hides that from processes; similarly, the client could hide the fact that creates are synchronous, but that's a bit trickier because creates can fail. Casper ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Rsync update to ZFS server over SSH faster than over NFS?
Try mounting the other way, so you read form NFS and write to ZFS (~DAS). That should perform significantly better. NFS write is slow (compared to read) because of syncronous ack. If you for some reason cant mount the other way, then you may want to play with NFS mount-options for write-buffer sizes (wsize=) and async (risky but faster!) I guess rsync has a wider ack-window that makes it faster than NFS write (i.e. buffers more without wait-state). Filesizes may have influence on the picture. Pål This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Rsync update to ZFS server over SSH faster than over NFS?
Pål Baltzersen wrote: Try mounting the other way, so you read form NFS and write to ZFS (~DAS). That should perform significantly better. NFS write is slow (compared to read) because of syncronous ack. If you for some reason cant mount the other way, then you may want to play with NFS mount-options for write-buffer sizes (wsize=) and async (risky but faster!) I guess rsync has a wider ack-window that makes it faster than NFS write (i.e. buffers more without wait-state). Filesizes may have influence on the picture. Could also try NFS over SSH :-) -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: ZFS - Use h/w raid or not? Thoughts. Considerations.
What if your HW-RAID-controller dies? in say 2 years or more.. What will read your disks as a configured RAID? Do you know how to (re)configure the controller or restore the config without destroying your data? Do you know for sure that a spare-part and firmware will be identical, or at least compatible? How good is your service subscription? Maybe only scrapyards and museums will have what you had. =o With ZFS/JBOD you will be safe; just get a new controller (or server) -- any kind that is protocol-compatible (and OS-complatible) you may have floating around (SATA2 | SCSI | FC..) -- and zpool import :) And you can safely buy the latestgratest and come out with something better than you had. With ZFS I prefer JBOD. For performance you may want external HW-RAIDs (=2) and let ZFS mirror them as virtual JBOD. -- depends where the bottleneck is; I/O or spindle. I disable any RAID-features on internal RAID-chipsets (nForce etc.). This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: ZFS - Use h/w raid or not? Thoughts. Considerations.
On 5/22/07, Pål Baltzersen [EMAIL PROTECTED] wrote: What if your HW-RAID-controller dies? in say 2 years or more.. What will read your disks as a configured RAID? Do you know how to (re)configure the controller or restore the config without destroying your data? Do you know for sure that a spare-part and firmware will be identical, or at least compatible? How good is your service subscription? Maybe only scrapyards and museums will have what you had. =o Be careful when talking about RAID controllers in general. They are not created equal! I can remove all of my disks from my RAID controller, reshuffle them and put them back in a new random order.. my controller will continue functioning correctly. I can remove my RAID controller and replace with a similar firmware (or higher), and my volumes will continue to live with correct initiators blacklisted/not. Hardware raid controllers have done the job for many years ... I'm a little bit concerned about the new message (from some) out there that they are no good anymore. Given the code on those controllers are probably not as elegant as zfs ... and given my personal preference of being in control, I cannot dismiss the fact that some of these storage units are fast as hell, especially when you start piling on the pressure! I'm also interested to see how Sun handles this phenomenon, and how they position zfs so that it doesn't eat into their high-margin (be it low turnover) Storagetek block storage. I'm also interested to see whether they will release a product dedicated for a solaris/zfs environment. Interesting times... PS: I've also noticed some persperation on the heads of some Symantec account managers. :) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Rsync update to ZFS server over SSH faster than over NFS?
allyourbase Take off every ZIL! http://number9.hellooperator.net/articles/2007/02/12/zil-communication /allyourbase On 22/05/07, Albert Chin [EMAIL PROTECTED] wrote: On Mon, May 21, 2007 at 06:11:36PM -0500, Nicolas Williams wrote: On Mon, May 21, 2007 at 06:09:46PM -0500, Albert Chin wrote: But still, how is tar/SSH any more multi-threaded than tar/NFS? It's not that it is, but that NFS sync semantics and ZFS sync semantics conspire against single-threaded performance. What's why we have set zfs:zfs_nocacheflush = 1 in /etc/system. But, that's only helps ZFS. Is there something similar for NFS? -- albert chin ([EMAIL PROTECTED]) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: ZFS - Use h/w raid or not? Thoughts. Considerations.
On 22-May-07, at 11:01 AM, Louwtjie Burger wrote: On 5/22/07, Pål Baltzersen [EMAIL PROTECTED] wrote: What if your HW-RAID-controller dies? in say 2 years or more.. What will read your disks as a configured RAID? Do you know how to (re)configure the controller or restore the config without destroying your data? Do you know for sure that a spare-part and firmware will be identical, or at least compatible? How good is your service subscription? Maybe only scrapyards and museums will have what you had. =o Be careful when talking about RAID controllers in general. They are not created equal! ... Hardware raid controllers have done the job for many years ... Not quite the same job as ZFS, which offers integrity guarantees that RAID subsystems cannot. I'm a little bit concerned about the new message (from some) out there that they are no good anymore. Given the code on those controllers are probably not as elegant as zfs ... and given my personal preference of being in control, I cannot dismiss the fact that some of these storage units are fast as hell, ... Being in control may mean *avoiding* black box RAID hardware in favour of inspectable maintainable open source software, which was the earlier poster's point. --Toby___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Rsync update to ZFS server over SSH faster than over NFS?
On Tue, May 22, 2007 at 10:04:34AM +0200, [EMAIL PROTECTED] wrote: On Mon, May 21, 2007 at 06:21:40PM -0500, Albert Chin wrote: On Mon, May 21, 2007 at 06:11:36PM -0500, Nicolas Williams wrote: On Mon, May 21, 2007 at 06:09:46PM -0500, Albert Chin wrote: But still, how is tar/SSH any more multi-threaded than tar/NFS? It's not that it is, but that NFS sync semantics and ZFS sync semantics conspire against single-threaded performance. What's why we have set zfs:zfs_nocacheflush = 1 in /etc/system. But, that's only helps ZFS. Is there something similar for NFS? NFS's semantics for open() and friends is that they are synchronous, whereas POSIX's semantics are that they are not. You're paying for a sync() after every open. I'm not sure the semantics of NFS are at all relevant for the complete performance picture. NFS writes are(/used to be) synchronous, but the client hides that from processes; similarly, the client could hide the fact that creates are synchronous, but that's a bit trickier because creates can fail. But it sounds tricky enough that it can't be pulled off. It'd be nice to have async versions of all fs-related syscalls... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Rsync update to ZFS server over SSH faster than over NFS?
On Mon, May 21, 2007 at 13:23:48 -0800, Marion Hakanson wrote: Albert Chin wrote: Why can't the NFS performance match that of SSH? My first guess is the NFS vs array cache-flush issue. Have you configured the 6140 to ignore SYNCHRONIZE_CACHE requests? That'll make a huge difference for NFS clients of ZFS file servers. Doesn't setting zfs:zfs_nocacheflush=1 achieve the same result: http://blogs.digitar.com/jjww/?itemid=44 The 6140 has a non-volatile cache. Dunno if it's order-preserving though. -- albert chin ([EMAIL PROTECTED]) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] I don't understand these benchmarks
MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU /fastsan/zfs3 (with 3 snapshots and a clone) 100 5414 98.8 31819 58.9 46620 95.4 2895 98.8 100516 99.8 7490.8 172.3 /fastsan/zfs3 (with 1 snapshot and a clone) 100 5316 97.5 65448 99.7 50097 99.0 2871 98.5 99271 99.5 8379.9 192.8 /fastsan/zfs3 (no snapshots) 100 5394 98.3 63724 99.7 48285 98.1 2884 98.5 98892 99.5 8396.3 192.8 I'm using bonnie on a test server with no load other than what I'm doing. As far as I can tell, I get better performance the more snapshots I have. Huh? This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] AVS replication vs ZFS send recieve for odd sized volume pairs
For the moment, SolarisCluster 3.2 does not support using AVS replication within a cluster for failover of storage. We do support using storage based replication for failover data with high end Hitachi based storage. Also at this point SolarisCluster does not ship with support for zfs send. You could probably write your own agent for zfs send using our agent builder tool. However, integrating this with the HANFS agent that ships with SolarisCluster will require that you are familiar with all of the failures that you may hit and what recovery action you want to take. -Charles a habman wrote: Hello all, I am interested in setting up an HA NFS server with zfs as the storage filesystem on Solaris 10 + Sun Cluster 3.2. This is an HPC environment with a 70 node cluster attached. File sizes are 1-200meg or so, with an average around 10meg. I have two servers, and due to changing specs through time I have ended up with heterogeneus storage. They are physically close to each other, so no offsite replication needs. Server A has an areca 12 port raid card attached to 12x400 gig drives. Server B has an onboard raid with 6 available slots which I plan on populating with either 750 gig or 1tb drives. With AVS 4.0 (which I have running on a test volume pair) I am able to mirror the zpools at the block level, but I am forced to have an equal number of LUNs for it to work on( AVS mirrors block devices that zfs works on top of). If I carve up each raid set into 4 volumes, AVS those(plus bitmap volumes) and then ZFS stripe over that, theoretically I am in business, although this has a couple of downsides. If I want to maximize my performance first, while keeping a margin of safety in this replicated environment, how can I best use my storage? Option one: AVS + Hardware raid 5 on each side. Make 4 LUNs and zfs stripe on top. Hardware raid takes care of drive failure. AVS ensures that the whole storage pool is replicated at all times to Server B. This method does not take advantage of disk caching zfs can do, nor additional performance scheduling zfs would like to manage at the drive level. Also unknown is how the SC3.2 HA ZFS module will work on an AVS zfs filesystem as I believe it was designed for a fiberchannel shared set of disks. On the plus side with this method we have block level replication, so close to instantaneous sync between filesystems. Option two: Full zfs pools on both side using zfs send+zfs recieve for the replication. This has benifits because my pools can be different sized and grow and thats ok. Could also be mounted on server B as well (most of the time). Downside is I have to hack a zfs send + recieve script+cron job, which is not likely as bombproof as the tried and tested AVS? So... basically, how are you all doing replication between two different disk topologies using zfs? I am a solaris newbie, attracted by the smell of the zfs, and so pardon my lack of in depth knowledge into these issues. Thank you in advance. Ahab ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
