Re: [Bug 726461] Re: sshd on lucid causes kernel panic
The box messed about again yesterday. This time I was still able to Ctrl+F1, Ctrl+F2 between terminals but it wouldn't accept any console input. Nor would it respond to anything except pings on the LAN. The keyboard Num/Caps/Scroll were flashing on the keyboard. On reset there doesn't seem to have been a core dump though. This time OCFS2 was definitely not involved as it wasn't in use at the time. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ocfs2-tools in ubuntu. https://bugs.launchpad.net/bugs/726461 Title: sshd on lucid causes kernel panic -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 726461] Re: sshd on lucid causes kernel panic
I'm not at work this week, but I'll look at it when I get back. Touch wood the extra RAM seems to have stopped it happening so far this week. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openssh in ubuntu. https://bugs.launchpad.net/bugs/726461 Title: sshd on lucid causes kernel panic -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 726461] Re: sshd on lucid causes kernel panic
It's using the native DLM. However, it's the only node in the cluster. Our live VM environment is Ubuntu server running KVM and OCFS2 and we have 2 nodes. This box is just a backup server which takes an lvm2 snapshot of the OCFS2 filesystem, exports it over iSCSI and then as a separate cluster mounts the filesystem and copies the VM images off. Alex -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openssh in ubuntu. https://bugs.launchpad.net/bugs/726461 Title: sshd on lucid causes kernel panic -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 726461] Re: sshd on lucid causes kernel panic
The OCFS2 filesystem is held on an OpenFiler box. Cluster A is our live VM system with two nodes. Cluster B is the backup system (the box in question). Once a week, the OpenFiler box takes an LVM snapshot of the filesystem normally used by cluster A and exports it as a new iSCSI target. The backup server, which is the sole member of cluster B then connects to the new iSCSI target and mounts the OCFS2 filesystem. It then copies the VM images from inside the OCFS2 filesystem to local disk It then unmounts the OCFS2 filesystem, disconnects from the iSCSI target. The OpenFiler box then stops exporting the backup target, and destroys the snapshot. So yes, the backup server is using OCFS2 properly, but not in the production environment cluster since we're dealing with historical data in the snapshot. Alex -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openssh in ubuntu. https://bugs.launchpad.net/bugs/726461 Title: sshd on lucid causes kernel panic -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 726461] Re: sshd on lucid causes kernel panic
Has every instance of this bug involved ssh in the stack trace? All the ones we've had to date. I initially thought it was apparmor related because it mentions memory allocation etc but having removed the packages and rebuilt initramfs I guess not. If it happens again from now on, I'll make sure we always get a photo. Can you describe the software raid stack in detail? It's really simple. 4x 2TB drives, each with a 2 partitons, the first being 2GB and the remainder. sda1 and sdb1 are RAID1 as /boot, sdc1 and sdd1 are RAID1 for swap and the sda2,b2,c2,d2 are RAID5 as / Currently the / filesystem is about 70% full. My first guess would be that OCFS is to blame. Would it be possible to run this server for awhile without it, or is that impossible? Can you either do without the SAN, or mount it as another fs type? It's not possible really. The machine only mounts a snapshot of our live SAN via iSCSI once a week to copy over virtual machine images. It TENDS to lock up at those times, however it's also by far the busiest time on the box as it'll be running rsnapshot backups at the same time too. It's not unsual for the box to have 180% IO WAIT across the two CPUs at those times. We've put more RAM in the box again this week to see if that solves it. I'm wondering if it's starting to swap to the point that the box is getting overloaded. Time will tell if that has any effect. It's hard because munin gives some idea of things like load/memory usage as the box loads up, but isn't frequent enough to catch a spike! Cheers Alex -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openssh in ubuntu. https://bugs.launchpad.net/bugs/726461 Title: sshd on lucid causes kernel panic -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 726461] [NEW] sshd on lucid causes kernel panic
Public bug reported: I have an HP ML115 running Lucid 64 bit. It's got 4 2TB Samsung drives in a software RAID 5 and is used to backup a bunch of servers (using rsnapshot), and also to mount a snapshot of our SAN (using OCFS2) and copy virtual machine images over. About once per week it hard locks with the attached messages. I've tried changing the RAM in the box (went from 2GB to 4GB, both HP branded sets). I've also removed AppArmor completely, but still get this message. This may well be unrelated to sshd, but it seems to be the process that's listed in the error message as causing the lockup. ** Affects: openssh (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openssh in ubuntu. https://bugs.launchpad.net/bugs/726461 Title: sshd on lucid causes kernel panic -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 726461] Re: sshd on lucid causes kernel panic
** Attachment added: Photo of the error output https://bugs.launchpad.net/bugs/726461/+attachment/1876255/+files/IMG_20110228_101124.jpg -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openssh in ubuntu. https://bugs.launchpad.net/bugs/726461 Title: sshd on lucid causes kernel panic -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 632314] Re: slapd Too many open files
Thanks for the pointer! Alex This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
I will do next time I do updates on that server. I don't want to take the service down at the moment given the trouble we've had up until now. Cheers Alex -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
After a couple of days with: ulimit -n 8192 in /etc/defaults/slapd we've not had a recurrence so I'll cautiously say it looks like that solves it. It's really strange however that adding the same limit via /etc/security/limits doesn't have the same effect? -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
Ended up with adding openldap soft nofile 8096 openldap hard nofile 8096 root soft nofile 8096 root hard nofile 8096 Rebooted. ulimit -a as root/openldap shows the change but still the system goes unresponsive and I get the too many open files message. Does anyone have anything else to suggest? -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
http://readlist.com/lists/openldap.org/openldap-software/2/12904.html http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=378261 Both worth reading. I've tried adding a ulimit -n 8192 as suggested in /etc/defaults/slapd so I'll see if that fixes things. ** Bug watch added: Debian Bug tracker #378261 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=378261 -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
r...@core01:~# su openldap r...@core01:~# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited On Dapper however we found that increasing the open files only prolonged the time before we had this problem. I'm already running the package I compiled up yesterday. I'll let that run for a few days or until the ldap server stops responding and then look at changing the limits here too. Alex -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
slapd fell over again this morning. I'm trying editing /etc/security/limits.conf openldap hard nofile 8096 root hard nofile 8096 in conjunction with modified packages. I'll report back. -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] [NEW] slapd Too many open files
Public bug reported: I reported this back in 2007 on Dapper and we've been rolling custom debs since then. https://bugs.launchpad.net/ubuntu/+source/openldap2.2/+bug/164303 Recently we've upgraded to lucid and are having the same issue again. After a short period of time with heavy load, we get: Nov 19 10:23:03 core01 slapd[24753]: warning: cannot open /etc/hosts.deny: Too many open files We've only got a few thousand users and 700 workstations using the directory so I can't see why nobody else would be seeing this issue. Any chance of having the compile options changed on the official package so that this doesn't happen, or am I facing a further 5 years of rolling my own every time there's an update? ** Affects: openldap (Ubuntu) Importance: Undecided Status: New ** Affects: openldap2.3 (Ubuntu) Importance: Undecided Status: Invalid ** Also affects: openldap (Ubuntu) Importance: Undecided Status: New ** Changed in: openldap2.3 (Ubuntu) Status: New = Invalid -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
Sorry. Marked against the wrong package. It's 2.4.21 -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 632314] Re: slapd Too many open files
I've built new packages with the debian.rules file patched as attached and will test over the next couple of days. ** Patch added: debian_rules.patch https://bugs.launchpad.net/ubuntu/+source/openldap2.3/+bug/632314/+attachment/1553586/+files/debian_rules.patch -- slapd Too many open files https://bugs.launchpad.net/bugs/632314 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to openldap2.3 in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
