> I'm more interested in block devices that are mounted > using iSCSI or > similar. Didn't swap via > NFS have to use a remote file mounted as a block > device?
No; on the server, a command such as mkfile is ultimately used to create a regular file, with the sticky bit set, which nowadays has the following meaning (from the sticky(5) man page): > If the sticky bit is set on a regular file and no execute bits are set, the > system's page cache will not be used to hold the file's data. This bit is > normally set on swap files of diskless clients so that accesses to these files > do not flush more valuable data from the system's cache. Moreover, by > default such files are treated as swap files, whose inode modification times > may not necessarily be correctly recorded on permanent storage. In practice, one probably doesn't run mkfile by hand, but follows procedures such as are described in http://docs.sun.com/app/docs/doc/819-2379/clientsetup-2?l=en&a=view&q=diskless to set up the client. To enable swapping over NFS, Solaris (if not SunOS 4.x before!) was modified to allow swapping not only to block devices, but to regular files. It doesn't really matter whether those are on local UFS (done sometimes to extend swap space when it would be inconvenient to repartition) or are accessed via NFS. I did see some notes in the code that it took more kernel stack space to swap over NFS, but I gather that any related bugs were fixed a long time ago. Swapping over NFS is really no big deal at all on Solaris; I've run a (SPARC) system very happily making zero use of local disk; indeed, the only time it was accessed was when I ran a a command (that I wrote myself) in an rc file to spin it down! Since the system (Voyager) had no fan, it was silent with the disk spun down (except that I could hear the hum of the power supply or something, esp. when doing a window resize or the like). And just incidentally, device files and FIFOs seen over NFS all refer to devices (or FIFOs) on the client; they have nothing to do with any devices on the server. So there's really no such thing as accessing a block device over NFS. (the old SVR3/SVR4.0 RFS remote file protocol could access remote devices that way; and I think even allow processes on different systems to communicate via an RFS-shared FIFO; but RFS is long dead; gone since something like Solaris 2.3, and not used by anything else current that I'm aware of) > Even then the situation is not what you seem to be > describing with a > tmpfs file. I'm well aware that has nothing to do with your situation, I was just saying that there are some cases where one would probably not want to swap on a particular block device. In particular, one certainly does not need to use lofi to make a regular file look like a block device to swap on it; in Solaris, one can use a regular file as-is just fine for swap. (with the exception that the regular file may not be located on a local zfs filesystem; but one can swap to a zfs zvol, and as far as I know, could also swap to a regular file mounted from a server on which it ultimately resided on a zfs filesystem; so in your situation, that limitation should be irrelevant) > For this analysis though, I'm trying to determine > whether *any* current > *nix can reliably swap > over a network to any kind of block device using its > internal NIC rather > than an iSCSI HBA > which looks like a disk controller to the device. Solaris swapping to a regular file over NFS seems to me to have been reasonably reliable - it's done it for years. Nor was it necessarily the first to use that sort of approach; long before iSCSI was dreamed up, Apollos had diskless nodes that swapped over the network (using their own proprietary remote filesystem, not NFS). As such, I wouldn't see the need to have an over-the-network block device like iSCSI for diskless swapping on Solaris. > It seems to me that emulation of such a device (ie > with its own IP stack > that hooks in at a low > level to IP) would fit the bill - essentially by > moving the memory > reservation and allocation > problem - but no system currently has such a > facility. I'm hoping > someone will say I'm > wrong in this - and I'm wondering if it might be > possible to achieve a > similar effect using Xen > or xVM or whatever with a swapless dom0 that can > provide a swap facility > to a guest. But I > don't know whether that would work either. Sufficiently recent Solaris (not sure if it's in 8/07, or only in Solaris Express) has a kernel-based iSCSI initator (as well as the longstanding ability to swap to a regular file over NFS). So I suppose it _could_ swap to a block device via iSCSI. But just as there were notes in the kernel about fixing a bug where more stack space was needed to swap over NFS, it wouldn't surprise me if there were similar potential problems in the case of iSCSI; probably not deadlocks like you were talking about, but still something that might cause problems. Since the iSCSI initiator is fairly new, and there was already a way for diskless workstations to swap without iSCSI, I don't know that anyone has tried swapping to an iSCSI-attached block device, and thus exercised that code, found and fixed any similar bugs, etc. Thus, I'd prefer to use the established way of doing it rather than fool with iSCSI. Nor can I see why one would even bother with (non-bootable software-based) iSCSI (rather than NFS) at all for a diskless Solaris system, unless one was running a DBMS that wanted a block device on them, in which case I'd probably want enough memory onboard to avoid the need to swap, since swap over the network (esp. via NFS, but probably also to a lesser degree via iSCSI) is likely to have higher latency than the local swap one was doing without, and in any event, most big DBMSs love all the memory they can get. So I simply don't see any need to swap over iSCSI on Solaris, even if it's possible; a much more mature solution (swapping to a regular file over NFS) already exists. As for an iSCSI HBA, the only point I can see to that is for booting over iSCSI. Since SPARC can boot over NFS without any local storage at all (and x86 could probably do it with reasonably small flash storage for grub and a miniroot), I can't see going that route very often. The only use I see for iSCSI initiator on Solaris is if one already has an iSCSI infrastructure (or has compelling reason to contemplate that rather than an NFS server). (OTOH, using Solaris as an iSCSI _target_ (server) with some cheap SATA JBOD being mirrored or raidz'd by zfs, and zvols actually being served out, that could be kind of cool.) So I'd say, in the case of Solaris, swap over NFS and just forget about iSCSI for swapping, even if it's _possible_, because absent an iSCSI HBA, you won't be _booting_ over iSCSI anyway, so you'll still need an NFS server. Which approach is better for other operating systems, that's your problem... This message posted from opensolaris.org _______________________________________________ networking-discuss mailing list [email protected]
