[zfs-discuss] SSD strange performance problem, resilvering helps during operation
Hi, We have a X4150 with a J4400 attached. Configured with 2x32GB SSD's, in mirror configuration (ZIL) and 12x 500GB SATA disks. We are running this setup for over a half year now in production for NFS and iSCSI for a bunch of virtual machines (currently about 100 VM's, Mostly Linux, some Windows) Since last week we have performance problems, cause IO Wait in the VM's. Of course we did a big search in networking issue's, hanging machines, filewall traffic tests, but were unable to find any problems. So we had a look into the zpool and dropped one of the mirrored SSD's from the pool (we had some indication the ZIL was not working ok). No success. After adding the disk, we discovered the IO wait during the resilvering process was OK, or at least much better, again. So last night we did the same handling, dropped added the same disk, and yes, again, the IO wait looked better. This morning the same story. Because this machine is a production machine, we cannot tolerate to much experiments. We now know this operation saves us for about 4 to 6 hours (time to resilvering), but we didn't had the courage to detach/attach the other SSD yet. We will try only a resilver, without detach/attach, this night, to see what happens. Can anybody explain how the detach/attach and resilver process works, and especially if there is something different during the resilvering and the handling of the SSD's/slog disks? Regards, Mart -- Greenhost - Duurzame Hosting Derde Kostverlorenkade 35 1054 TS Amsterdam T: 020 489 4349 F: 020 489 2306 KvK: 34187349 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] SSD strange performance problem, resilvering helps during operation
Mart van Santen wrote: Hi, We have a X4150 with a J4400 attached. Configured with 2x32GB SSD's, in mirror configuration (ZIL) and 12x 500GB SATA disks. We are running this setup for over a half year now in production for NFS and iSCSI for a bunch of virtual machines (currently about 100 VM's, Mostly Linux, some Windows) Since last week we have performance problems, cause IO Wait in the VM's. Of course we did a big search in networking issue's, hanging machines, filewall traffic tests, but were unable to find any problems. So we had a look into the zpool and dropped one of the mirrored SSD's from the pool (we had some indication the ZIL was not working ok). No success. After adding the disk, we discovered the IO wait during the resilvering process was OK, or at least much better, again. So last night we did the same handling, dropped added the same disk, and yes, again, the IO wait looked better. This morning the same story. Because this machine is a production machine, we cannot tolerate to much experiments. We now know this operation saves us for about 4 to 6 hours (time to resilvering), but we didn't had the courage to detach/attach the other SSD yet. We will try only a resilver, without detach/attach, this night, to see what happens. Can anybody explain how the detach/attach and resilver process works, and especially if there is something different during the resilvering and the handling of the SSD's/slog disks? Regards, Mart Do the I/O problems go away when only one of the SSDs is attached? Frankly, I'm betting that your SSDs are wearing out. Resilvering will essentially be one big streaming write, which is optimal for SSDs (even an SLC-based SSD, as you likely have, performs far better when writing large amounts of data at once). NFS (and to a lesser extent iSCSI) is generally a whole lot of random small writes, which are hard on an SSD (especially MLC-based ones, but even SLC ones). The resilvering process is likely turning many of the random writes coming in to the system into a large streaming write to the /resilvering/ drive. My guess is that the SSD you are having problems with has reached the end of it's useful lifespan, and the I/O problems you are seeing during normal operation are the result of that SSD's problems with committing data. There's no cure for this, other than replacing the SSD with a new one. SSDs are not hard drives. Even high-quality modern ones have /significantly/ lower USE lifespans than an HD - that is, a heavily-used SSD will die well before a HD, but a very-lightly used SSD will likely outlast a HD. And, in the case of SSDs, writes are far harder on the SSD than reads are. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] SSD strange performance problem, resilvering helps during operation
It might be helpful to contact SSD vendor, report the issue and inquire if half a year wearing out is expected behavior for this model. Further, if you have an option to replace one (or both) SSDs with fresh ones, this could tell for sure if they are the root cause. Regards, Andrey On Mon, Dec 21, 2009 at 1:18 PM, Erik Trimble erik.trim...@sun.com wrote: Mart van Santen wrote: Hi, We have a X4150 with a J4400 attached. Configured with 2x32GB SSD's, in mirror configuration (ZIL) and 12x 500GB SATA disks. We are running this setup for over a half year now in production for NFS and iSCSI for a bunch of virtual machines (currently about 100 VM's, Mostly Linux, some Windows) Since last week we have performance problems, cause IO Wait in the VM's. Of course we did a big search in networking issue's, hanging machines, filewall traffic tests, but were unable to find any problems. So we had a look into the zpool and dropped one of the mirrored SSD's from the pool (we had some indication the ZIL was not working ok). No success. After adding the disk, we discovered the IO wait during the resilvering process was OK, or at least much better, again. So last night we did the same handling, dropped added the same disk, and yes, again, the IO wait looked better. This morning the same story. Because this machine is a production machine, we cannot tolerate to much experiments. We now know this operation saves us for about 4 to 6 hours (time to resilvering), but we didn't had the courage to detach/attach the other SSD yet. We will try only a resilver, without detach/attach, this night, to see what happens. Can anybody explain how the detach/attach and resilver process works, and especially if there is something different during the resilvering and the handling of the SSD's/slog disks? Regards, Mart Do the I/O problems go away when only one of the SSDs is attached? Frankly, I'm betting that your SSDs are wearing out. Resilvering will essentially be one big streaming write, which is optimal for SSDs (even an SLC-based SSD, as you likely have, performs far better when writing large amounts of data at once). NFS (and to a lesser extent iSCSI) is generally a whole lot of random small writes, which are hard on an SSD (especially MLC-based ones, but even SLC ones). The resilvering process is likely turning many of the random writes coming in to the system into a large streaming write to the /resilvering/ drive. My guess is that the SSD you are having problems with has reached the end of it's useful lifespan, and the I/O problems you are seeing during normal operation are the result of that SSD's problems with committing data. There's no cure for this, other than replacing the SSD with a new one. SSDs are not hard drives. Even high-quality modern ones have /significantly/ lower USE lifespans than an HD - that is, a heavily-used SSD will die well before a HD, but a very-lightly used SSD will likely outlast a HD. And, in the case of SSDs, writes are far harder on the SSD than reads are. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] SSD strange performance problem, resilvering helps during operation
Hi, Do the I/O problems go away when only one of the SSDs is attached? No, the problem stays with only one SSD. The problem is only less when resilvering, but not totally disappeared (maybe because of the resilver overhead). Frankly, I'm betting that your SSDs are wearing out. Resilvering will essentially be one big streaming write, which is optimal for SSDs (even an SLC-based SSD, as you likely have, performs far better when writing large amounts of data at once). NFS (and to a lesser extent iSCSI) is generally a whole lot of random small writes, which are hard on an SSD (especially MLC-based ones, but even SLC ones). The resilvering process is likely turning many of the random writes coming in to the system into a large streaming write to the /resilvering/ drive. Hmm, interesting theory. Next I well execute only a resilver to see if the same happens. I assume when adding a new disk, even though it's only a slog disk, the whole tank will resilver? If I look to the zpool iostat currently I see a lot of reads on the separate SATA disks (not on the tank/or raidz2 pools), assuming resilvering takes place there and the SSD's are already synced. My guess is that the SSD you are having problems with has reached the end of it's useful lifespan, and the I/O problems you are seeing during normal operation are the result of that SSD's problems with committing data. There's no cure for this, other than replacing the SSD with a new one. SSDs are not hard drives. Even high-quality modern ones have /significantly/ lower USE lifespans than an HD - that is, a heavily-used SSD will die well before a HD, but a very-lightly used SSD will likely outlast a HD. And, in the case of SSDs, writes are far harder on the SSD than reads are. Is about half a year for these disk not really short? Sure, we have some I/O, but not that many write operations, about ~80-140 iops, anyway, I will try to get new disks from SUN (we have SLC disks from Sun). Is there any knowledge about the life time of SSD's? Maybe in terms of amount of I/O Operations? Regards, Mart van Santen -- Greenhost - Duurzame Hosting Derde Kostverlorenkade 35 1054 TS Amsterdam T: 020 489 4349 F: 020 489 2306 KvK: 34187349 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] SSD strange performance problem, resilvering helps during operation
Mart van Santen wrote: Hi, Do the I/O problems go away when only one of the SSDs is attached? No, the problem stays with only one SSD. The problem is only less when resilvering, but not totally disappeared (maybe because of the resilver overhead). The resilver is likely masking some underlying problem. :-( Frankly, I'm betting that your SSDs are wearing out. Resilvering will essentially be one big streaming write, which is optimal for SSDs (even an SLC-based SSD, as you likely have, performs far better when writing large amounts of data at once). NFS (and to a lesser extent iSCSI) is generally a whole lot of random small writes, which are hard on an SSD (especially MLC-based ones, but even SLC ones). The resilvering process is likely turning many of the random writes coming in to the system into a large streaming write to the /resilvering/ drive. Hmm, interesting theory. Next I well execute only a resilver to see if the same happens. I assume when adding a new disk, even though it's only a slog disk, the whole tank will resilver? If I look to the zpool iostat currently I see a lot of reads on the separate SATA disks (not on the tank/or raidz2 pools), assuming resilvering takes place there and the SSD's are already synced. I'm not 100% sure, but replacing a device in a mirrored ZIL should only generate I/O on the other ZIL device, not on the main pool devices. SSDs are not hard drives. Even high-quality modern ones have /significantly/ lower USE lifespans than an HD - that is, a heavily-used SSD will die well before a HD, but a very-lightly used SSD will likely outlast a HD. And, in the case of SSDs, writes are far harder on the SSD than reads are. Is about half a year for these disk not really short? Sure, we have some I/O, but not that many write operations, about ~80-140 iops, anyway, I will try to get new disks from SUN (we have SLC disks from Sun). Is there any knowledge about the life time of SSD's? Maybe in terms of amount of I/O Operations? Regards, Mart van Santen That's not enough time for that level of IOPS to wear out the SSDs (which, are likely OEM Intel X25-E). Something else is wrong. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
