Re: [Vserver] vserver enter eating cpu and vkill problem
Hey, [...] Could you try http://svn.linux-vserver.org/projects/util-vserver/changeset/2514?format=diffnew=2514 instead? Essentially the same thing, just in the (IMHO) correct function. It works very well. Jarek DylÄ…g ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
Jarek Dylag wrote: Hey [...] Yeah, I know where it happens, but why it happens and how to fix it is still a mystery. Thus far, every solution I've tried has failed. http://people.linux-vserver.org/~dhozac/p/uv/experimental/delta-vlogin-efds.diff has two of the things I've tried to fix it. I added additional check in termina_copy() functon: Could you try http://svn.linux-vserver.org/projects/util-vserver/changeset/2514?format=diffnew=2514 instead? Essentially the same thing, just in the (IMHO) correct function. --- vlogin.c.old2006-12-09 17:15:22.0 +0100 +++ vlogin.c2007-03-14 13:47:13.0 +0100 @@ -139,6 +139,11 @@ } else if (len == -1) return; + if (len == 0) { +terminal_kill(SIGTERM); +exit(1); + } + /* write activity to user */ EwriteAll(dst, buf, len); } It fixes the problem, but i don't know if i haven't broken something else. Jarek Dylag -- Daniel Hokka Zakrisson ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
Hey [...] Yeah, I know where it happens, but why it happens and how to fix it is still a mystery. Thus far, every solution I've tried has failed. http://people.linux-vserver.org/~dhozac/p/uv/experimental/delta-vlogin-efds.diff has two of the things I've tried to fix it. I added additional check in termina_copy() functon: --- vlogin.c.old2006-12-09 17:15:22.0 +0100 +++ vlogin.c2007-03-14 13:47:13.0 +0100 @@ -139,6 +139,11 @@ } else if (len == -1) return; + if (len == 0) { +terminal_kill(SIGTERM); +exit(1); + } + /* write activity to user */ EwriteAll(dst, buf, len); } It fixes the problem, but i don't know if i haven't broken something else. Jarek Dylag ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
Hey, [...] Apparently it happens when the session is not ended properly (disconnections, etc) I could reproduce the problem as follow: * ssh to the server as normal user * sudo su - * vserver XX enter * now from another term, kill -9 on either of the two su processes (su - or -su) At that point the session in the vserver ends but vcontext is still hooked to the current terminal so cpu is fine. * now quit the current terminal At that point vcontext is not hooked to any terminal (vps ax = ? on the tty column) and using top you see it's running 100% cpu, ~ 80% system and 20% user. I'll look in to it. I am unable to reproduce that here. The vlogin process does stick around, which it probably shouldn't, but its resource usage is the same as it's always been. I can reproduce it on debian sid with 2.6.20.1-vs2.3.0.11 kernel. ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2716 82.5 0.0 116 40 ?R15:17 7:04 login root 2774 0.0 0.3 5072 1660 pts/6Ss+ 15:17 0:00 \_ /bin/bash -login I straced login process, strace shows: Process 2716 attached - interrupt to quit select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, \r, 64) = 1 write(3, \r, 1) = 1 select(4, [0 3], NULL, NULL, NULL) = 1 (in [3]) read(3, \r\n, 64) = 2 write(1, \r\n, 2) = 2 select(4, [0 3], NULL, NULL, NULL) = 1 (in [3]) read(3, \33[1m/ \33[37;0m(\33[32;1m1525\33[37;0m..., 64) = 47 write(1, \33[1m/ \33[37;0m(\33[32;1m1525\33[37;0m..., 47) = 47 Here session ended: select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) Jarek Dylag ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
Jarek Dylag wrote: Hey, [...] Apparently it happens when the session is not ended properly (disconnections, etc) I could reproduce the problem as follow: * ssh to the server as normal user * sudo su - * vserver XX enter * now from another term, kill -9 on either of the two su processes (su - or -su) At that point the session in the vserver ends but vcontext is still hooked to the current terminal so cpu is fine. * now quit the current terminal At that point vcontext is not hooked to any terminal (vps ax = ? on the tty column) and using top you see it's running 100% cpu, ~ 80% system and 20% user. I'll look in to it. I am unable to reproduce that here. The vlogin process does stick around, which it probably shouldn't, but its resource usage is the same as it's always been. I can reproduce it on debian sid with 2.6.20.1-vs2.3.0.11 kernel. ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2716 82.5 0.0 116 40 ?R15:17 7:04 login root 2774 0.0 0.3 5072 1660 pts/6Ss+ 15:17 0:00 \_ /bin/bash -login I straced login process, strace shows: Process 2716 attached - interrupt to quit select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, \r, 64) = 1 write(3, \r, 1) = 1 select(4, [0 3], NULL, NULL, NULL) = 1 (in [3]) read(3, \r\n, 64) = 2 write(1, \r\n, 2) = 2 select(4, [0 3], NULL, NULL, NULL) = 1 (in [3]) read(3, \33[1m/ \33[37;0m(\33[32;1m1525\33[37;0m..., 64) = 47 write(1, \33[1m/ \33[37;0m(\33[32;1m1525\33[37;0m..., 47) = 47 Here session ended: select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) read(0, , 64) = 0 select(4, [0 3], NULL, NULL, NULL) = 1 (in [0]) Yeah, I know where it happens, but why it happens and how to fix it is still a mystery. Thus far, every solution I've tried has failed. http://people.linux-vserver.org/~dhozac/p/uv/experimental/delta-vlogin-efds.diff has two of the things I've tried to fix it. -- Daniel Hokka Zakrisson ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
On Sun, Jan 14, 2007 at 12:09:36PM +0100, Philippe Teuwen wrote: I'll look in to it. Thanks, I just tried on the vanilla setup 2.6.17.14 + vs2.0.2.1-grsec2.1.9 and I have the same cpu burning issue. So it has nothing to do with Debian kernels. Did someone else experienced the same problem when following the steps I presented in my first email? haven't checked your steps yet, but it is very likely that a vanilla (completely unpatched) kernel from kernel.org will expose similar effects ... http://people.linux-vserver.org/%7Eharry/patch-2.6.17.14-vs2.0.2.1-grsec2.1.9.diff Lots of programs overwrite the process name to get nicer ps/top/etc. output. Ok vkill needs the context too, i.e. vkill --xid devel 17111. Ok but why killing from context 1 does not work? because context 1 is the so called 'spectator context' and not the admin context (which is xid=0 for now) chcontext --ctx 1 kill = silently fails chcontext --ctx kill = ok yep, intentional HTH, Herbert Phil ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
I'll look in to it. Thanks, I just tried on the vanilla setup 2.6.17.14 + vs2.0.2.1-grsec2.1.9 and I have the same cpu burning issue. So it has nothing to do with Debian kernels. Did someone else experienced the same problem when following the steps I presented in my first email? http://people.linux-vserver.org/%7Eharry/patch-2.6.17.14-vs2.0.2.1-grsec2.1.9.diff Lots of programs overwrite the process name to get nicer ps/top/etc. output. Ok vkill needs the context too, i.e. vkill --xid devel 17111. Ok but why killing from context 1 does not work? chcontext --ctx 1 kill = silently fails chcontext --ctx kill = ok Phil ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
On Sat, Jan 13, 2007 at 12:31:52AM +0100, Philippe Teuwen wrote: This is on a remote server in a datacenter so I'm not that fan to try a vanilla home-patched remotely :-/ i know ... i had same problem ;-) Nevertheless I tried some upgrade: the last kernel and the last util-vserver tools of etch and sid: Linux version 2.6.18-3-vserver-amd64 (Debian 2.6.18-7) ([EMAIL PROTECTED]) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-20)) #1 SMP Mon Dec 4 17:19:24 CET 2006 vserver patch: 2.0.2.2-rc8 util-vserver: 0.30.212-1 I have the same problems with this single difference: now killing a process of a guest with a single kill in context 0 does not work anymore (as expected), so the only way is # vserver exec kill as vkill still does not work. yes 2.6.18-3-vserver-amd64 was bad too ... -- 5o Peter.Mann at tuke.sk ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
yes 2.6.18-3-vserver-amd64 was bad too ... Are the kernels here better? http://linux-vserver.derjohn.de/debian-etch-sid/debian/dists/sid/main/binary-amd64/ as seen in the faq http://linux-vserver.org/Frequently_Asked_Questions#Were_can_I_get_newer_versions_of_VServer_as_ready_made_packages_for_Debian.3F Phil ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
On Sat, Jan 13, 2007 at 12:44:05PM +0100, Philippe Teuwen wrote: yes 2.6.18-3-vserver-amd64 was bad too ... Are the kernels here better? http://linux-vserver.derjohn.de/debian-etch-sid/debian/dists/sid/main/binary-amd64/ as seen in the faq http://linux-vserver.org/Frequently_Asked_Questions#Were_can_I_get_newer_versions_of_VServer_as_ready_made_packages_for_Debian.3F i don't know ... you can make own kernel with vanilla source + vserver patch by using your kernel config and make-kpkg ... IMHO it's easy and safe ... i sent to this list mail about Etch vserver kernels a few days ago: Subject: Re: [Vserver] stop vserver cause error/oops - with new debian 4.0 vserver kernel version Date: Tue, 9 Jan 2007 09:47:36 +0100 On Tue, Jan 09, 2007 at 06:11:10AM +0100, [EMAIL PROTECTED] wrote: On Mon, Jan 08, 2007 at 11:05:50AM +0100, Oliver Paulus wrote: I have used Linux kernel 2.6.17.13 with VServer 2.0.2.1 before - everything worked correctly. Now I have installed the Debian 4.0 (Etch) vserver kernel 2.6.18-3-vserver-amd64. If I stop a specific vserver instance with vserver name stop I get the following error: please try with a mainstream kernel, preferable one of the 2.2.0-rc* kernels, and mainstream util-vserver (0.30.212) if the issue remains, please let us know, otherwise please contact the debian maintainers about those issues ... TIA, Herbert The Debian maintainers are aware, and it's the bug fixed in 2.0.2.2-rc9 (off by one cacct). From what I gathered, the most recent Debian snapshot kernel should work fine, but I didn't quite understand how or where to get it. Debian Etch is now frozen, so every package update into Etch is processed through debian-release team ... some new linux-2.6 (kernel) is scheduled: http://lists.debian.org/debian-release/2007/01/msg00085.html -- 5o Peter.Mann at tuke.sk ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
Philippe Teuwen wrote: Hello, Here are some problems I have with vserver. My environment: Kernel: Debian vserver_pre-patched kernel for AMD64: Linux version 2.6.17-2-vserver-amd64 (Debian 2.6.17-9) ([EMAIL PROTECTED]) (gcc version 4.1.2 20060901 (prerelease) (Debian 4.1.1-13)) #1 SMP Wed Sep 13 18:02:36 CEST 2006 According to changelog.Debian, this version is using vserver patch v2.0.2 util-vserver: 0.30.211-6 First problem: In some situations my CPU was burning like hell for days before I noticed it. This was the vcontext/login process, the one running when using the command vserver XX enter. If it happens that the process is disconnected from its terminal it eats all the CPU :-( Apparently it happens when the session is not ended properly (disconnections, etc) I could reproduce the problem as follow: * ssh to the server as normal user * sudo su - * vserver XX enter * now from another term, kill -9 on either of the two su processes (su - or -su) At that point the session in the vserver ends but vcontext is still hooked to the current terminal so cpu is fine. * now quit the current terminal At that point vcontext is not hooked to any terminal (vps ax = ? on the tty column) and using top you see it's running 100% cpu, ~ 80% system and 20% user. I'll look in to it. Second problem (cosmetic): Depending on the tool, the process I was talking about in the previous problem appears as vcontext (top, pgrep,... cat /proc/NNN/status) or as login (ps ax, pgrep -f,... cat /proc/NNN/cmdline) or should I say login\0\0\0\0\0\0\0\0\0\0\0... This is quite disturbing using processes mgmt tools like when I wrote a cron to detect cpu hangry vcontext processes to be killed :-) Lots of programs overwrite the process name to get nicer ps/top/etc. output. Third problem I discovered when writing that watchdog: Processes from guests are not displayed using ps/top/etc in context 0 (ps ax from host) but well in context 1 (e.g. vps ax). Fine. But if I want to kill one of them: host:~# vserver devel enter devel:~# top --- in another host term --- host:~# ps -C top PID TTY TIME CMD host:~# vps -C top PID CONTEXT TTY TIME CMD 17111 31022 devel pts/13 00:00:00 top host:~# vkill 17111 vkill: vc_ctx_kill(): No such process vkill needs the context too, i.e. vkill --xid devel 17111. host:~# chcontext --ctx 1 ps -C top PID TTY TIME CMD 17111 pts/13 00:00:00 top host:~# chcontext --ctx 1 kill 17111 no error but nothing happens neither: host:~# vps -C top PID CONTEXT TTY TIME CMD 17111 31022 devel pts/13 00:00:00 top host:~# kill 17111 host:~# vps -C top PID CONTEXT TTY TIME CMD worked!! This is not the behaviour I expected. Apart from that I'm happy running vserver for almost a year with now 6 guests. Phil PS: nothing particular here when I ran testme.sh Linux-VServer Test [V0.17] Copyright (C) 2003-2006 H.Poetzl chcontext is working. chbind is working. Linux 2.6.17-2-vserver-amd64 #1 SMP Wed Sep 13 18:02:36 CEST 2006 x86_64 Ea 0.30.211 236/glibc (DSa) compat,v11,fscompat,v13,net,v21,oldproc,olduts VCI: 0002:0002 236 0316 (TbLgnP) --- [000]# succeeded. [001]# succeeded. [011]# succeeded. [031]# succeeded. [101]# succeeded. [102]# succeeded. [201]# succeeded. [202]# succeeded. -- Daniel Hokka Zakrisson ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
On Fri, Jan 12, 2007 at 07:18:42PM +0100, Philippe Teuwen wrote: Here are some problems I have with vserver. My environment: Kernel: Debian vserver_pre-patched kernel for AMD64: Linux version 2.6.17-2-vserver-amd64 (Debian 2.6.17-9) ([EMAIL PROTECTED]) (gcc version 4.1.2 20060901 (prerelease) (Debian 4.1.1-13)) #1 SMP Wed Sep 13 18:02:36 CEST 2006 According to changelog.Debian, this version is using vserver patch v2.0.2 util-vserver: 0.30.211-6 i had same problem with debian kernels ... new kernel are waiting to enter Etch (which is now frozen) ... so it's (temporary) debian specific problem Etch is still 'testing' so try your own kernel ... i recompiled vanilla + vserver patch and all works fine ... -- 5o Peter.Mann at tuke.sk ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver
Re: [Vserver] vserver enter eating cpu and vkill problem
Here are some problems I have with vserver. Linux version 2.6.17-2-vserver-amd64 (Debian 2.6.17-9) ([EMAIL PROTECTED]) (gcc version 4.1.2 20060901 (prerelease) (Debian 4.1.1-13)) #1 SMP Wed Sep 13 18:02:36 CEST 2006 According to changelog.Debian, this version is using vserver patch v2.0.2 util-vserver: 0.30.211-6 i had same problem with debian kernels ... new kernel are waiting to enter Etch (which is now frozen) ... so it's (temporary) debian specific problem Etch is still 'testing' so try your own kernel ... i recompiled vanilla + vserver patch and all works fine ... Thanks for the advice, This is on a remote server in a datacenter so I'm not that fan to try a vanilla home-patched remotely :-/ Nevertheless I tried some upgrade: the last kernel and the last util-vserver tools of etch and sid: Linux version 2.6.18-3-vserver-amd64 (Debian 2.6.18-7) ([EMAIL PROTECTED]) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-20)) #1 SMP Mon Dec 4 17:19:24 CET 2006 vserver patch: 2.0.2.2-rc8 util-vserver: 0.30.212-1 I have the same problems with this single difference: now killing a process of a guest with a single kill in context 0 does not work anymore (as expected), so the only way is # vserver exec kill as vkill still does not work. Phil ___ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver