Re: vn_fullpath() again
On Tue, 6 Sep 2005, Matthew Dillon wrote: NFS views from the client are one of those shades of gray, since files and directories can be ripped up by other clients or the server, but since clients have to assume a certain level of consistency anyway it's hardly a show stopper from the point of view of any real-life use or need. The NFS server problem I was referring to specifically came up in the case of the DTE work is as follows, and can't be solved through modifications to the name cache: - An NFS client uses lookups to retrieve the file handle of /foo/bar/baz on an NFS server. It sets the cwd of a process on the client to that handle. - The NFS server reboots, flushing its name cache and other state. - The NFS client begins to look up and access files relative to the handle of /foo/bar/baz, which is fine because it has a file handle. All further file access by the NFS client is done relative to directories that don't appear in the name cache unless faulted in through further access to that directory via the file system root, and access to existing open files may be done without access to any directory at all. It may even happen if the file isn't open on the client but the path data is cached in the client avoiding server access. This doesn't just apply to direct remote access -- local access by file handle for rpc.lockd and friends is similarly affected, such as access via fhopen() in order to proxy locking operations. In the case of DTE, the file server was required to reconstruct a path to the directory and files, which in the case of a pure directory access required a number of I/O's to walk up the tree to the root. In the case of simply accessing a file and not a directory, it required an O(1) search of directories on the disk to find a directory that referenced the file. Both of these mechanisms fall ratherly firmly into the undesirable category, but are necessary if you want reliable and arbitrary vnode-path conversion. Hence my asking about who the consumer is, when they need the data, and how timely the data needs to be. If you're only interested in the actions of local processes, then the name cache can be a somewhat reliable source of name data. If you're also interested in identifying the actions of remote consumers of files accessing by file handle, which may account for the majority of activity on a file system on an NFS file server -- i.e., for a server side access monitoring tool, then that may cause significant difficulty. This is because, as has been discussed extensively, paths are traversals of a name space at a particular time, and not properties of files themselves. Robert N M Watson ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Re: vn_fullpath() again
From: Giorgos Keramidas [EMAIL PROTECTED] On 2005-09-06 19:27, Igor Shmukler [EMAIL PROTECTED] wrote: Perhaps, I do not get it or maybe you are do not getting my point. There are times when resolving would not be possible or a name returned is not necessarily the one used when file was first accessed. We have discussed it here and everyone agreed on that. The hardlinks or files unlinked while vnode is still open are corner cases. The unlink is a bit more difficult to deal with, but hardlinks are probably not a big issue. As long as we can get A name, we may not even need to know THE name. Why does it make sense to get name A in the following scenario then? user 1 creates file A user 1 hardlinks this to B user 1 gets the real name of A user 2 deletes file A user 2 creates a new file called A user 1 tries to access A and gets something unexpected Corner cases and their handling *is* important. Find another way to do whatever it is you're thinking you can do with the real name of a vnode. This particular case is easy to handle: all that user 1 needs to do is to do fstat() after opening the file and check that the returned inode field is still the same. On the otehr hand, if there is this new interface to access files directly by inode numbers, why bother with any names at all? If a name is so desirable, then just create a pseudo-fs translation that would convert the text-formatted device and inode number given to it as a file name to an access to the identified file. Though both this and access directly by vnode number open a security hole: in many cases the access to files mught be actually limited by making the directories unreadable to the unwanted users. Bypassing the directories bypasses this security by obscurity. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Re: vn_fullpath() again
On 2005-09-06 11:29, Sergey Babkin [EMAIL PROTECTED] wrote: Giorgos Keramidas [EMAIL PROTECTED] On 2005-09-06 19:27, Igor Shmukler [EMAIL PROTECTED] wrote: Perhaps, I do not get it or maybe you are do not getting my point. There are times when resolving would not be possible or a name returned is not necessarily the one used when file was first accessed. We have discussed it here and everyone agreed on that. The hardlinks or files unlinked while vnode is still open are corner cases. The unlink is a bit more difficult to deal with, but hardlinks are probably not a big issue. As long as we can get A name, we may not even need to know THE name. Why does it make sense to get name A in the following scenario then? user 1 creates file A user 1 hardlinks this to B user 1 gets the real name of A user 2 deletes file A user 2 creates a new file called A user 1 tries to access A and gets something unexpected Corner cases and their handling *is* important. Find another way to do whatever it is you're thinking you can do with the real name of a vnode. This particular case is easy to handle: all that user 1 needs to do is to do fstat() after opening the file and check that the returned inode field is still the same. This sounds still a bit hackish. Why require that user 1 reopens the file when a) he already has an open descriptor to, for example, i-node 12345, b) reopening may have unwanted side-effects? On the otehr hand, if there is this new interface to access files directly by inode numbers, why bother with any names at all? That's more like it. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Re: vn_fullpath() again
On Tue, 6 Sep 2005, Sergey Babkin wrote: Corner cases and their handling *is* important. Find another way to do whatever it is you're thinking you can do with the real name of a vnode. This particular case is easy to handle: all that user 1 needs to do is to do fstat() after opening the file and check that the returned inode field is still the same. On the otehr hand, if there is this new interface to access files directly by inode numbers, why bother with any names at all? If a name is so desirable, then just create a pseudo-fs translation that would convert the text-formatted device and inode number given to it as a file name to an access to the identified file. Though both this and access directly by vnode number open a security hole: in many cases the access to files mught be actually limited by making the directories unreadable to the unwanted users. Bypassing the directories bypasses this security by obscurity. FYI, Mac OS X has a service called volfs, mounted as /.vol, which does exactly this. It was required because Mac OS 9 allowed applications to open a file by path, and then get back a persistent reference ID that would work across reboots. This is how applications like Microsoft Word can open a file that appears in the recently opened file list even after it has been renamed or moved from one directory to another, and how Mac OS 9 shortcuts could span file systems even after renames and moves. In Mac OS X, these file handles are converted into requests against volfs, and nominally bypass the directory structure. In earlier Mac OS X versions, this resulted in possible security problems, if folder permissions were assumed to protect their contents. As I understand it, in newer versions, they attempt to construct a path which will then be followed to open the file, in order to cause appropriate permission checks to occur. This process works OK on HFS+, but very poorly on other file systems. You'll notice, though, that if you look at the audit event stream, during first use you'll often see normal path names, but that as the system peters along, the vn_getpath() call in XNU will start to return volfs names -- typically when that is the most recent or only path that has been used to reach a file, since the name mechanism is based on the assumption that the name cache holds the name you want. In practice, systems like Solaris and Irix assume that names are a path that is important at the time of lookup for audit purposes, since the path you follow affects the permission checks that take place, but that once you get to the file what you really want is the device and inode number, since only that combination usefully persistently identifies the file. Ideally you have the generation number in there also, as well as mount events, etc. This is why, if you look at Solaris BSM streams, you get the lookup path only at time of open (or stat or whatever), and after that, just a description of the vnode/inode properties, not a path. In our implementation of audit for Mac OS X, we try a bit harder to generate paths, since HFS+ is fairly good at it, but you will likely get no path, or a volfs path, for files that reside in a non-HFS+ file system, have only been accessed using Carbon APIs, and so on. Robert N M Watson ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: vn_fullpath() again
:The NFS server problem I was referring to specifically came up in the case :of the DTE work is as follows, and can't be solved through modifications :to the name cache: : :- An NFS client uses lookups to retrieve the file handle of /foo/bar/baz : on an NFS server. It sets the cwd of a process on the client to that : handle. : :- The NFS server reboots, flushing its name cache and other state. : :- The NFS client begins to look up and access files relative to the handle : of /foo/bar/baz, which is fine because it has a file handle. : :All further file access by the NFS client is done relative to directories :that don't appear in the name cache unless faulted in through further :access to that directory via the file system root, and access to existing :open files may be done without access to any directory at all. It may :even happen if the file isn't open on the client but the path data is :cached in the client avoiding server access. This doesn't just apply to :direct remote access -- local access by file handle for rpc.lockd and :friends is similarly affected, such as access via fhopen() in order to :proxy locking operations. In the case of DTE, the file server was :required to reconstruct a path to the directory and files, which in the :case of a pure directory access required a number of I/O's to walk up the :tree to the root. In the case of simply accessing a file and not a :directory, it required an O(1) search of directories on the disk to find a :directory that referenced the file. Both of these mechanisms fall :ratherly firmly into the undesirable category, but are necessary if you :want reliable and arbitrary vnode-path conversion. I reviewed the Dragonfly implementation and I was incorrect about the NFS(v3) server being able to resolve the namecache/path for file handles representing regular files. It only can do it for directories, but since all namespace operations (lookup, rename, remove, etc) pass a directory file handle in the RPC, this is all the NFS server needs to satisfy DragonFly's VOP requirements for namespace operations. So in DragonFly it is still possible to have a dangling regular file vnode on the NFS server, but it is not possible for a dangling vnode to exist with regard to any namespace operation (since DragonFly is required to pass namecache pointers for namespace VOPs like REMOVE, RENAME, LOOKUP, etc, and the directory vnode is available in all such cases). In anycase, the resolution case for directories is pretty trivial. I have debugging code which prints out to the console whenever DFly has to walk the directory tree backwards to resolve a directory from fhopen, and most of the time the namecache entry is either already cached from prior operations, or the walk only has to scan one directory level because namecache entries for higher levels are already cached from prior operations. Even when working on a very large directory tree after a server reboot, the directory nodes in the namecache are very quickly repopulated and very quickly become optimal. I tested that too and it wasn't a big deal. I consider the reverse resolution case for directories to be a solved problem. The regular file resolution issue for changes made by an NFS server is something I need to fix for our journaling code, to reduce the number of instances where the userland side of the journaling stream will have to index by the inode number. I think the regular file case can be mostly solved by embedding the directory inode number in the file handle for any file that is looked up, at least for NFSv3. NFSv2 is a lost cause, the file handles simply are not big enough. Lookups would mask out the directory inode part of the handle (i.e. you wouldn't get ESTALE if the file happened to be renamed to another directory), but namepath resolution would be able to use the directory inode data as a hint. This would result in nearly all such files being resolvable with the scan of only a single directory per file, and the other files encountered during the scan could be cached, as well. :Hence my asking about who the consumer is, when they need the data, and :how timely the data needs to be. If you're only interested in the actions :of local processes, then the name cache can be a somewhat reliable source :of name data. If you're also interested in identifying the actions of :remote consumers of files accessing by file handle, which may account for :the majority of activity on a file system on an NFS file server -- i.e., :for a server side access monitoring tool, then that may cause significant :difficulty. This is because, as has been discussed extensively, paths are :traversals of a name space at a particular time, and not properties of :files themselves. : :Robert N M Watson Yes, I agree. It just so happens
Re: vn_fullpath() again
Igor Shmukler [EMAIL PROTECTED] writes: You are correct about the Unix file system organization, but does it mean reliable vnode to fullname conversation is not possible? Yes. Get over it. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: vn_fullpath() again
Igor Shmukler [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: Igor Shmukler [EMAIL PROTECTED] writes: You are correct about the Unix file system organization, but does it mean reliable vnode to fullname conversation is not possible? Yes. Get over it. Well, I do not think it is a Yes. I very much think it is a No. You should have continued reading my email 'til the middle or even farther. I did. You just don't get it. A file may be associated with zero, one or more names and none of these names are more correct or authoritative than any of the others. If a user does 'ln /bin/ls /tmp' (assuming /bin and /tmp are on the same filesystem), it may be obvious to you that /bin/ls is the real name is /tmp/ls is just an alias, but it is not obvious to the kernel. In fact, the kernel is unable to see any difference at all between these two names. Storing the name that was used to access a file in the vnode does not solve anything, because the vnode is shared by all users of that file, regardless of which name they used to access it, and there is no guarantee that the name that was used to access a file two seconds ago still references the same file, or any file at all; the file may have been renamed or deleted, or a new filesystem may have been mounted that covers the namespace that file was in. In summary: THERE IS NO WAY TO UNIQUELY AND RELIABLY MAP A VNODE BACK TO A NAME, and I wish people would stop insisting that there must be. All the world is not MS-DOS. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: vn_fullpath() again
On 2005-09-06 19:27, Igor Shmukler [EMAIL PROTECTED] wrote: Perhaps, I do not get it or maybe you are do not getting my point. There are times when resolving would not be possible or a name returned is not necessarily the one used when file was first accessed. We have discussed it here and everyone agreed on that. The hardlinks or files unlinked while vnode is still open are corner cases. The unlink is a bit more difficult to deal with, but hardlinks are probably not a big issue. As long as we can get A name, we may not even need to know THE name. Why does it make sense to get name A in the following scenario then? user 1 creates file A user 1 hardlinks this to B user 1 gets the real name of A user 2 deletes file A user 2 creates a new file called A user 1 tries to access A and gets something unexpected Corner cases and their handling *is* important. Find another way to do whatever it is you're thinking you can do with the real name of a vnode. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: vn_fullpath() again
[snip] I did. You just don't get it. A file may be associated with zero, one or more names and none of these names are more correct or authoritative than any of the others. If a user does 'ln /bin/ls /tmp' (assuming /bin and /tmp are on the same filesystem), it may be obvious to you that /bin/ls is the real name is /tmp/ls is just an alias, but it is not obvious to the kernel. In fact, the kernel is unable to see any difference at all between these two names. Yes -but when a user requests a mapping of vnode to pathname, he is asking in the context of files he has accessed (recently). In this context, the name cache does suffice -but unfortunately not every unix variant provides access to it. regards -kamal ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: vn_fullpath() again
On Tue, 6 Sep 2005, Kamal R. Prasad wrote: one or more names and none of these names are more correct or authoritative than any of the others. If a user does 'ln /bin/ls /tmp' (assuming /bin and /tmp are on the same filesystem), it may be obvious to you that /bin/ls is the real name is /tmp/ls is just an alias, but it is not obvious to the kernel. In fact, the kernel is unable to see any difference at all between these two names. Yes -but when a user requests a mapping of vnode to pathname, he is asking in the context of files he has accessed (recently). In this context, the name cache does suffice -but unfortunately not every unix variant provides access to it. regards -kamal I suppose it depends a lot on what it is you plan to use the path for. For the purposes of audit, we've concluded that in fact we don't even need to re-generate the path, we'll simply use the path as requested by the user when a path was entered. I.e., our definition of recently is immediately. Other less immediate definitions of recently are a bit of a problem though, since file access often happens over the course of months or years after the initial lookup took place, at which point things an be very, very stale. For example, programs often open log files and hold them open for extended periods; likewise libraries, data files, directories, and so on. How recently is recently? One tenth of a second? One second? Ten seconds? Ten minutes? Ten hours? Ten days? Ten weeks? Ten years? FreeBSD operates on all of these time scales... Right now the FreeBSD name cache reliably returns paths for a short period of time after they are looked up -- the further you get from the initial lookup, the less reliable they become. The reliability is largely affected by two factors: the fact that this is a cache, and things fall out of the cache, and the fact that the name space is quite mutable and invalidates entries in the cache. The third case of unreliability which is addressable is cases where synthetic file systems simply don't use the cache -- i.e., devfs. My proposal for dealing with this is simply to allow those synthetic file systems to return a name if they can. I think modifying them to use the name cache doesn't make sense however, since in many cases you don't want to cache: i.e., lookups of /dev/ptmx with the pts driver, or /proc/curproc, and caching would defeat the point. Robert N M Watson ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: vn_fullpath() again
At the cost of drawing ire from FreeBSD core developers, I will point out that reverse-resolution is hardly a black-and-white issue. There are many shades of grey, and there is a huge problem set that can either be solved or 99.99% of the way solved (greatly reducing the time required to solve the remainder) with a more robust namecache implementation. FreeBSD's implementation is basically at the lowest rung on the ladder. DragonFly's is a couple of rungs up. DragonFly can returned a guarenteed consistent path (even through renames of any component) to any open vnode as well as tell you whether the namespace used to access the vnode was remove()'d. For an auditing program or for generating a high level journaling stream to generate a mirror on a remote host, that covers 99.99% of the filesystem. NFS views from the client are one of those shades of gray, since files and directories can be ripped up by other clients or the server, but since clients have to assume a certain level of consistency anyway it's hardly a show stopper from the point of view of any real-life use or need. Hardlinks are one of those shades of gray, but they hardly invalidate the many uses that namepath resolution can be put to. 99.99% of the files on most filesystems are either not hardlinked or not removed once acccessed, after all, and at least with UFS a directory CAN'T be hardlinked. The important thing is to reduce the problem set to something manageable. For a mirroring program or an auditing program, being able to get valid paths in real time for nearly all the changes made to a filesystem is no small thing, and you at least get a definitive red flag for any hardlinks (simply by the fact that st_nlink is greater then one) and can do it the slow way (aka scan/index all files with st_nlink 1) for the remaining few, and track realtime namespace operations on hardlinked files after that (which is very easy to do). As to how to solve the basic problem in FreeBSD... well, it basically isn't solvable to the degree that DFly has solved it unless someone good spends a lot of time rewriting the namecache code and the VFS API. BUT, short of doing that, I think it *IS* possible to rewrite enough of the namecache to at least make the namecache records consistent against the active vnodes and to not throw away namecache records for the directory chain leading up to any vnode. It's even possible to generate the chain for vnodes generated from file handles (inode numbers), which an NFS server op has to do quite often, because the directory is available in those cases (DragonFly does this for NFS server operations so I know it's possible). It is even possible to do even less work to maintain the associations... you don't even NEED to have a working namecache, in fact. All you need are ref'd directory vnodes in a chain from any leaf leading to the mount point... basically taking the vnode-v_dd field and changing it from a verifier heuristic to a real, ref'd directory vnode, with appropriate feedback from filesystem to fix things up for rename(), and mark the namecache entry as invalid for remove(). Given a valid directory vnode chain, you can ALWAYS regenerate a valid path (maybe not the only path, but a *VALID* path) to any vnode for all cases except the case where you have a hardlinked file that you have open()'d and remove()'d. Very few programs care about open but completely unlinked files. DragonFly can provide the original path to such a file, but it flags it as having been removed and one can almost certainly ignore such files for, e.g. filesystem mirroring and even for auditing if the file is not otherwise important. Considering the rarity of the case, it would be sufficient to simply red-flag the condition (which you can reliably do with a v_dd directory chain implementation). In summary, the implementation would be: * Maintain vref'd v_dd pointers in leaf vnodes representing the directory tree to a leaf so they can't go away until the leaf vnode goes away. * Handle NFS server based file handle - vnode translation by resolving the chain to root (doable because the NFS server has access to the related directory vnode for all such translations). * Use the namecache when it exists, and * Create related namecache records when asked to resolve a full path when it doesn't by recursing upwards through the directory chains and scanning the directory to locate the name translation for the underlying vnode. DragonFly does this for the NFS server (see the cache_inefficient_scan() procedure in kern/vfs_cache.c in the DFly source for an example). That is more achievable in FreeBSD. In fact, I would say that it is
Re: vn_fullpath() again
On 9/6/05, Robert Watson [EMAIL PROTECTED] wrote: On Tue, 6 Sep 2005, Kamal R. Prasad wrote: one or more names and none of these names are more correct or authoritative than any of the others. If a user does 'ln /bin/ls /tmp' (assuming /bin and /tmp are on the same filesystem), it may be obvious to you that /bin/ls is the real name is /tmp/ls is just an alias, but it is not obvious to the kernel. In fact, the kernel is unable to see any difference at all between these two names. Yes -but when a user requests a mapping of vnode to pathname, he is asking in the context of files he has accessed (recently). In this context, the name cache does suffice -but unfortunately not every unix variant provides access to it. regards -kamal I suppose it depends a lot on what it is you plan to use the path for. For the purposes of audit, we've concluded that in fact we don't even need to re-generate the path, we'll simply use the path as requested by the user when a path was entered. I.e., our definition of recently is immediately. Thanks for the info detailed herewith. My purpose is similar to lsof utility. It looks up the namecache and provides a list of open filenames for a given process and it turns out that lsof is cross-platform and quite popular. Just curious -how huge would the name cache be -if all open files had a vnode--pathname mapping recorded? The mappings can be dropped when the file descriptor's reference count drops to 0. regards -kamal Other less immediate definitions of recently are a bit of a problem though, since file access often happens over the course of months or years after the initial lookup took place, at which point things an be very, very stale. For example, programs often open log files and hold them open for extended periods; likewise libraries, data files, directories, and so on. How recently is recently? One tenth of a second? One second? Ten seconds? Ten minutes? Ten hours? Ten days? Ten weeks? Ten years? FreeBSD operates on all of these time scales... Right now the FreeBSD name cache reliably returns paths for a short period of time after they are looked up -- the further you get from the initial lookup, the less reliable they become. The reliability is largely affected by two factors: the fact that this is a cache, and things fall out of the cache, and the fact that the name space is quite mutable and invalidates entries in the cache. The third case of unreliability which is addressable is cases where synthetic file systems simply don't use the cache -- i.e., devfs. My proposal for dealing with this is simply to allow those synthetic file systems to return a name if they can. I think modifying them to use the name cache doesn't make sense however, since in many cases you don't want to cache: i.e., lookups of /dev/ptmx with the pts driver, or /proc/curproc, and caching would defeat the point. Robert N M Watson ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: vn_fullpath() again
On Mon, 5 Sep 2005, Sergey Uvarov wrote: all knows that vn_fullpath() is unreliable. However I really need to get a filename for a given vnode. To simplify the task, I do not care of synthetic file systems or hardlinks. I have looked through archives in hope to find a better solution. It seems that linux_getcwd() approach could help. However to make that code work for me, I need to know a directory vnode where the file resides. vnode-v_dd field looks promising. But as I understand it did not help if file name is not in a name cache. So the question: is it ever possible to get directory vnode for a given file vnode? One way to look at the problem is from the perspective of how you might derive that information from an on-disk inode. If you look at the UFS layout on-disk, you'll see that there is no pointer to a directory back from a leaf inode; in kernel, you can have a reference to a vnode with no back pointer to a directory vnode. In order to find the parent, you potentially have to iterate through all directories on the hard disk looking for the parent, which is a potentially long-running activity. It's also not at all theoretical: vnodes are often accessed without any path lookup at all. For example, background fsck may pull inodes off disk without a name lookup, and the NFS server can receive file handle references following a reboot from a live client that maintains cached references -- it will service them without performing a lookup. So unfortunately, the answer is complex: (a) you may have to search the disk for a name, and (b) you may not even find one, since there can be files without any name at all (i.e., a temporary file that has been unlinked). On non-UFS style file systems, such as HFS+, it is possible to generate a path from the file system root without extensive disk I/O. However, all common UNIX-like file systems don't have this property -- Sun's version of UFS, ext2fs/ext3fs, and so on. If the child vnode is a directory, you can just follow it's '..' link or covered vnode, of course... Robert N M Watson ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
