John, > Hello All- > > The box hosting my iSCSI targets is a Sun Ultra 40 M2,running > Solaris 10 update 4. It has eight 750GB SATA disks. Seven of those > are in a RAID-Z pool, against which I have created about half a > dozen iSCSI targets. > > These targets work beautifully and correctly from four machines that > I'm mounting them. The problem comes after about two weeks of > uptime. Around that point, the speed of IO requests to the iSCSI > targets becomes painfully slow. A 290mb file takes forty five > minutes to copy. A 13 GB (single file) takes 36 hours. The > operations will eventually complete without error, but they are just > extremely slow. > > The first time it happened, I restarted the Solaris box, and speed > came back up to normal. This time when it happened, I simply issued > an 'svcadm restart iscsitgt' and thereafter, everything started to > work perfectly again. That same 36 hour transfer only took fifteen > minutes, etc. > > While using the ZFS pool was very slow over iSCSI, operations on the > Ultra 40 itself functioned as expected, and IO was snappy and quick > (what took 36 hours over iSCSI took only four minutes on that same > RAID-Z pool). > > I'm guessing this is something gone awry in the iscsitgtd? I'm not > sure where to go from here, but would appreciate any insight anybody > can offer.
For a quite a while the iSCSI Team has been focusing their efforts on T10 State Machine issues, but only recently becoming aware of the consequences of errant state machine processing, namely memory leaks. What this can mean is that over a period of time, days, weeks, months, ..., the virtual process size keeps getting larger and larger. This is likely the reason that overall performance of the iSCSI Target degrades over time, and a restart of the daemon restores normal behavior, until such a time as this defect is resolved and the resolution made available. For those interested, Solaris provides an excellent tool for identifying memory management problems, as described here: http://developers.sun.com/solaris/articles/libumem_library.html To use the mechanism with an SMF controlled process, one can set (or unset) the variables as follows: # svccfg -s iscsitgt setenv LD_PRELOAD libumem.so # svccfg -s iscsitgt setenv UMEM_LOGGING transaction,contents # svccfg -s iscsitgt setenv UMEM_DEBUG default # svcadm refresh iscsitgt # svcadm restart iscsitgt To check if various environmentals are enabled # pargs -e `pgrep -f iscsitgt` | grep -i umem Conversely, if set and you want to clear these settings invoke the following: # svccfg -s iscsitgt unsetenv LD_PRELOAD # svccfg -s iscsitgt unsetenv UMEM_DEBUG # svccfg -s iscsitgt unsetenv UMEM_LOGGING # svccfg -s iscsitgt UNsetenv UMEM_OPTIONS # svcadm refresh iscsitgt # svcadm restart iscsitgt > For what it's worth, the four client machines using these iSCSI > targets are all Windows based clients, running the 2.06 version of > the Microsoft iSCSI initiator. > > Thanks to all- > John Tracy > > > This message posted from opensolaris.org > _______________________________________________ > storage-discuss mailing list > [email protected] > http://mail.opensolaris.org/mailman/listinfo/storage-discuss Jim Dunham Storage Platform Software Group Sun Microsystems, Inc. wk: 781.442.4042 http://blogs.sun.com/avs http://www.opensolaris.org/os/project/avs/ http://www.opensolaris.org/os/project/iscsitgt/ http://www.opensolaris.org/os/community/storage/ _______________________________________________ storage-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/storage-discuss
