Re: FreeBSD 5.1-Release freezes
On Wed, 25 Jun 2003 16:44:56 -0400 (EDT) Robert Watson [EMAIL PROTECTED] wrote: The code most likely to cause a memory leak in the MAC Framework is the label management code, since that's the only code that really does much in the way of memory allocaiton. Try compiling options MAC_DEBUG into your kernel, which causes the MAC Framework to track the number of labels it has allocated/free'd in a series of variables: static unsigned int nmacmbufs, nmaccreds, nmacifnets, nmacbpfdescs, nmacsockets, nmacmounts, nmactemp, nmacvnodes, nmacdevfsdirents, nmacipqs, nmacpipes, nmacprocs; You can inspect them using a series of sysctls in the security.mac.debug tree; I'd be interested to see how those values change as you approach the hang. Hmm, this is strange. I build the same kernel with: makeoptions DEBUG=-g options DDB #Enable the kernel debugger options INVARIANTS #Enable calls of extra sanity checking options INVARIANT_SUPPORT #Extra sanity checks of internal structures, required by INVARIANTS options WITNESS #Enable checks to detect deadlocks and cycles options MAC options MAC_DEBUG Now it doesn't hang and there is nothing in the logs. Btw here is some info: security.mac.debug.label_fallback: 0 security.mac.debug.counters.mbufs: 0 security.mac.debug.counters.creds: 17 security.mac.debug.counters.ifnets: 3 security.mac.debug.counters.ipqs: 0 security.mac.debug.counters.bpfdescs: 0 security.mac.debug.counters.sockets: 7 security.mac.debug.counters.pipes: 2 security.mac.debug.counters.procs: 63 security.mac.debug.counters.mounts: 6 security.mac.debug.counters.temp: 0 security.mac.debug.counters.vnodes: 429 security.mac.debug.counters.devfsdirents: 96 Again I am not running anything but the base system and ssh. br socketd ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-Release freezes
On Wed, 25 Jun 2003 16:44:56 -0400 (EDT) Robert Watson [EMAIL PROTECTED] wrote: The code most likely to cause a memory leak in the MAC Framework is the label management code, since that's the only code that really does much in the way of memory allocaiton. Try compiling options MAC_DEBUG into your kernel, which causes the MAC Framework to track the number of labels it has allocated/free'd in a series of variables: static unsigned int nmacmbufs, nmaccreds, nmacifnets, nmacbpfdescs, nmacsockets, nmacmounts, nmactemp, nmacvnodes, nmacdevfsdirents, nmacipqs, nmacpipes, nmacprocs; After running a few commands, ssh logins and loading mac_seeotheruids we now have: security.mac.debug.label_fallback: 0 security.mac.debug.counters.mbufs: 0 security.mac.debug.counters.creds: 17 security.mac.debug.counters.ifnets: 3 security.mac.debug.counters.ipqs: 0 security.mac.debug.counters.bpfdescs: 0 security.mac.debug.counters.sockets: 7 security.mac.debug.counters.pipes: 2 security.mac.debug.counters.procs: 63 security.mac.debug.counters.mounts: 6 security.mac.debug.counters.temp: 0 security.mac.debug.counters.vnodes: 524 security.mac.debug.counters.devfsdirents: 99 Loading mac_seeotheruids made vnodes increase a little, but it has (very slowly) been increasing and as you can see is now at 524 (don't know if this means anything). br socketd ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-Release freezes
On Thu, 26 Jun 2003, Socketd wrote: On Wed, 25 Jun 2003 16:44:56 -0400 (EDT) Robert Watson [EMAIL PROTECTED] wrote: The code most likely to cause a memory leak in the MAC Framework is the label management code, since that's the only code that really does much in the way of memory allocaiton. Try compiling options MAC_DEBUG into your kernel, which causes the MAC Framework to track the number of labels it has allocated/free'd in a series of variables: static unsigned int nmacmbufs, nmaccreds, nmacifnets, nmacbpfdescs, nmacsockets, nmacmounts, nmactemp, nmacvnodes, nmacdevfsdirents, nmacipqs, nmacpipes, nmacprocs; After running a few commands, ssh logins and loading mac_seeotheruids we now have: ... Loading mac_seeotheruids made vnodes increase a little, but it has (very slowly) been increasing and as you can see is now at 524 (don't know if this means anything). You can compare the number of labeled vnodes with the number of actual vnodes by looking at the vfs.numvnodes sysctl value -- ideally they should be the same. The fact that the problem went away with the addition of the debugging code is a little concerning. Here's what I'd like you to try, if you can: set up a serial console to another machine. Compile the kernel with options DDB, but without options MAC_DEBUG. Do whatever it is you do that triggers the hang. On the serial console, send a serial break and see if you can get into DDB. If you can, send the output of the following commands: ps show lockedvnods I may then ask you to run the trace command on various processes/threads, and use show locks to show which threads hold which locks (assuming you have WITNESS in the kernel, which also changes the timing). On the whole, enabling MAC doesn't substantially change the locking, but there are a couple of edge cases where we grab locks that were not previously grabbed. For example, we now grab vnode locks when polling a vnode so we can do a label check on the vnode; this generates WITNESS warnings as we hold the file descriptor lock over the call to VOP_POLL() (ouch). If we wend up concluding this is a locking problem, it might be that I send you some patches to modify the locking and see if we can get rid of the problem. thanks! Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Network Associates Laboratories ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-Release freezes
Date: Wed, 25 Jun 2003 19:16:52 +0200 From: Socketd [EMAIL PROTECTED] Sender: [EMAIL PROTECTED] Hi all I have just installed 5.1 Release on my laptop (6gb harddisk and 60 mb ram). I only installed the base system + man pages and the port collection. A little info: / 128M Swap 256M /var 128M /var/tmp 256M (/tmp is symlinked to this) /home 256M /usr The rest I compiled a new kernel, with no support for SCSI, RAID, IPv6 and all but my USB NIC (also removed a few USB things). I added: options SC_DISABLE_REBOOT options MAC device pcm It compiled fine and I installed it. Now, if I run with this kernel the system freezes! I can boot, login and run a few commands, but that's it. There is nothing in the logs, but when I tried ssh'ing to the mashine I got this before it froze: fork() - Not enough ressources available at this time (or something like that). Is there a memory leak somewhere or am I doing something wrong? (I compiled and installed the kernel with: make buildkernel KERNCONF=SOCKETD make installkernel KERNCONF=SOCKETD) Providing your complete configuration file and /var/run/dmesg.boot might give a bit of a clue. You do mention that you are using MAC. The MAC code is rather new and clearly labeled as experimental, so that might be a source of the problem. Can you try building a kernel without MAC? -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: [EMAIL PROTECTED] Phone: +1 510 486-8634 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-Release freezes
On Wed, 25 Jun 2003 10:24:43 -0700 Kevin Oberman [EMAIL PROTECTED] wrote: Providing your complete configuration file and /var/run/dmesg.boot might give a bit of a clue. Ok. /usr/src/sys/i386/conf/SOCKETD: machine i386 cpu I586_CPU ident SOCKETD options SCHED_4BSD #4BSD scheduler options INET#InterNETworking options FFS #Berkeley Fast Filesystem options SOFTUPDATES #Enable FFS soft updates support options UFS_ACL #Support for access control lists options UFS_DIRHASH #Improve performance on big directories options MD_ROOT #MD is a potential root device options MSDOSFS #MSDOS Filesystem options CD9660 #ISO 9660 Filesystem options PROCFS #Process filesystem (requires PSEUDOFS) options PSEUDOFS#Pseudo-filesystem framework options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_FREEBSD4 #Compatible with FreeBSD4 options SCSI_DELAY=15000#Delay (in ms) before probing SCSI options KTRACE #ktrace(1) support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time extensionsoptions KBD_INSTALL_CDEV# install a CDEV entry in /dev options AHC_REG_PRETTY_PRINT# Print register bitfields in debug # output. Adds ~128k to driver. options AHD_REG_PRETTY_PRINT# Print register bitfields in debug # output. Adds ~215k to driver. options INVARIANT_SUPPORT #Extra sanity checks of internal structures, required by INVARIANTS device isa device eisa device pci device fdc device ata device atadisk # ATA disk drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID #Static device numbering device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver device splash # Splash screen and screen saver support device sc device agp # support several AGP chipsets device npx device pmtimer device cbb # cardbus (yenta) bridge device pccard # PC Card (16-bit) bus device cardbus # CardBus (32-bit) bus device sio # 8250, 16[45]50 based serial ports device ppc device ppbus # Parallel port bus (required) device lpt # Printer device plip# TCP/IP over parallel device ppi # Parallel port interface device device miibus # MII bus support device random # Entropy device device loop# Network loopback device ether # Ethernet support device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory disks device bpf # Berkeley packet filter device uhci# UHCI PCI-USB interface device ohci# OHCI PCI-USB interface device usb # USB Bus (required) device ugen# Generic device uhid# Human Interface Devices device ukbd# Keyboard device ums # Mouse device aue # ADMtek USB ethernet device pcm options SC_DISABLE_REBOOT options MAC And /var/run/dmesg.boot Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.1-RELEASE #1: Sat Jan 2 03:46:39 CET 1999 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SOCKETD Preloaded elf kernel /boot/kernel/kernel at 0xc048f000. Preloaded elf module /boot/kernel/acpi.ko at 0xc048f1f4. Timecounter i8254 frequency 1193182 Hz Timecounter TSC frequency 448341906 Hz CPU: AMD-K6(tm) 3D processor (448.34-MHz 586-class CPU) Origin = AuthenticAMD Id = 0x58c Stepping = 12 Features=0x8021bfFPU,VME,DE,PSE,TSC,MSR,MCE,CX8,PGE,MMX AMD Features=0x8800SYSCALL,3DNow! real memory =
Re: FreeBSD 5.1-Release freezes
On Wed, 25 Jun 2003, Socketd wrote: I have just installed 5.1 Release on my laptop (6gb harddisk and 60 mb ram). I only installed the base system + man pages and the port collection. A little info: / 128M Swap 256M /var 128M /var/tmp 256M (/tmp is symlinked to this) /home 256M /usr The rest I compiled a new kernel, with no support for SCSI, RAID, IPv6 and all but my USB NIC (also removed a few USB things). I added: options SC_DISABLE_REBOOT options MAC device pcm It compiled fine and I installed it. Now, if I run with this kernel the system freezes! I can boot, login and run a few commands, but that's it. There is nothing in the logs, but when I tried ssh'ing to the mashine I got this before it froze: fork() - Not enough ressources available at this time (or something like that). Is there a memory leak somewhere or am I doing something wrong? (I compiled and installed the kernel with: make buildkernel KERNCONF=SOCKETD make installkernel KERNCONF=SOCKETD) btw please cc to me as I am not on the list. Sounds like a memory leak to me; it might also be a locking problem. Are you actually running with any MAC policies, or just with the framework? The code most likely to cause a memory leak in the MAC Framework is the label management code, since that's the only code that really does much in the way of memory allocaiton. Try compiling options MAC_DEBUG into your kernel, which causes the MAC Framework to track the number of labels it has allocated/free'd in a series of variables: static unsigned int nmacmbufs, nmaccreds, nmacifnets, nmacbpfdescs, nmacsockets, nmacmounts, nmactemp, nmacvnodes, nmacdevfsdirents, nmacipqs, nmacpipes, nmacprocs; You can inspect them using a series of sysctls in the security.mac.debug tree; I'd be interested to see how those values change as you approach the hang. If you have DDB compiled into the kernel (which you probably will want for debugging this problem), I'd be interested in seeing if you can break into DDB using a console break or serial console break. If so, print out the values of those variables and see if any are through the roof. Also, you might do a ps in DDB and see if processes are blocked on anything in particular. The most likely locking problem is a vnode deadlock; if you can get into DDB, try show lockedvnods to see whether any locks are held, etc. If you don't have a serial console set up, you might want to set one up because it will make copying and pasting all this stuff into e-mail a lot easier. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Network Associates Laboratories ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-Release freezes
Just FYI, there are actually some legitimate situations in which processes will appear to hang when using MAC. For example, if you have MLS or Biba turned on, and revocation support specifically enabled (not the default), you can see interactive processes appear to hang when they are interacting with a terminal that is not authorized to receive its output. For example, if you change the process labels so that they have access to top secret data, but your terminal label is set so it is only cleared for confidential data, the process can't write to the terminal. Given that you don't appear to have any policies loaded, that's presumably not what is going on. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Network Associates Laboratories ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 5.1-Release freezes
On Wed, 25 Jun 2003 16:44:56 -0400 (EDT) Robert Watson [EMAIL PROTECTED] wrote: Sounds like a memory leak to me; it might also be a locking problem. Are you actually running with any MAC policies, or just with the framework? Just the framework, no policies. The code most likely to cause a memory leak in the MAC Framework is the label management code, since that's the only code that really does much in the way of memory allocaiton. Try compiling options MAC_DEBUG into your kernel, which causes the MAC Framework to track the number of labels it has allocated/free'd in a series of variables: [snip] Ok, I will try and do the things du suggest (never tried that before), but am I the only one with this problem? If you'd like, I can give you root access to the laptop, then you can play all you want :-) br socketd ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
