Re: Preventing a single student programmer from choking the whole server.
Hi Yedidyah Loved the SIGSTOP/SIGKILL approach - simple but awsome. I seems to be working. Thanks for everyone's input. Josh On Sun, Apr 23, 2017 at 9:18 AM, Yedidyah Bar Davidwrote: > On Sat, Apr 22, 2017 at 12:51 PM, Josh Roden wrote: > > Hi > > I see some explanation about cgroups on Centos 6 here: > > https://www.digitalocean.com/community/tutorials/how-to- > limit-resources-using-cgroups-on-centos-6 > > I will check about its implementation for Centos 6 and update my post. > > Thanks for giving a direction. > > Josh > > > > On Fri, Apr 21, 2017 at 4:27 PM, guy keren > wrote: > >> > >> > >> how about using cgroups, putting each user's login shell in a cgroup > that > >> cannot use more then X% of the whole CPUs. this will affect all > processes > >> spawned under the user's shell. > >> > >> --guy > >> > >> > >> On 04/21/2017 03:07 PM, Josh Roden wrote: > >>> > >>> Hi > >>> > >>> server setup: > >>> -- > >>> Centos 6 > >>> 32GB RAM > >>> 16 Cpu's > >>> 70 students max > >>> > >>> I am using /etc/security/limits.conf to prevent the students from > >>> choking the whole > >>> server but sometimes one student will write a very bad program that > >>> somehow > >>> runs itself again and again - so fast that "killall" and "pkill -9 -u" > >>> can't stop/remove > >>> the user fast enough before the student's program is run again and > >>> again... > >>> > >>> Here is my definition in limits: > >>> > >>> @stud hardcpu 8 > >>> @stud hardnproc 256 > >>> @stud hardnofile 1024 > >>> @stud - maxlogins 6 > >>> > >>> I can't reduce cpu time below 8min because eclipse will be killed every > >>> hour or so. > >>> My problem seems to be that the student can run up to 256 processes > that > >>> each > >>> uses 100% of a single CPU and we only have 16 CPU''s. > >>> > >>> Thanks for any suggestions. > >>> Josh > > Also, to handle actual specific cases, you can: > 1. Send all processes SIGSTOP (or SIGTSTP) > 2. Then send them SIGKILL > > Why does this work? > > First, why doesn't simply sending all SIGKILL works? > The kernel already limits the user to 256 procs. So when they try to create > more, they fail. But when you kill some of them, the others will now > succeed > creating more. > > With above, you first suspend all procs, so that they won't manage to try > to > create new ones. Then, when you kill some of them, the others do not create > new ones and eventually all are killed. > > Best, > > >>> > >>> > >>> ___ > >>> Linux-il mailing list > >>> Linux-il@cs.huji.ac.il > >>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > >>> > >> > >> > >> ___ > >> Linux-il mailing list > >> Linux-il@cs.huji.ac.il > >> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > > > > > > > ___ > > Linux-il mailing list > > Linux-il@cs.huji.ac.il > > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > > ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Preventing a single student programmer from choking the whole server.
Hi I see some explanation about cgroups on Centos 6 here: https://www.digitalocean.com/community/tutorials/how-to-limit-resources-using-cgroups-on-centos-6 I will check about its implementation for Centos 6 and update my post. Thanks for giving a direction. Josh On Fri, Apr 21, 2017 at 4:27 PM, guy kerenwrote: > > how about using cgroups, putting each user's login shell in a cgroup that > cannot use more then X% of the whole CPUs. this will affect all processes > spawned under the user's shell. > > --guy > > > On 04/21/2017 03:07 PM, Josh Roden wrote: > >> Hi >> >> server setup: >> -- >> Centos 6 >> 32GB RAM >> 16 Cpu's >> 70 students max >> >> I am using /etc/security/limits.conf to prevent the students from >> choking the whole >> server but sometimes one student will write a very bad program that >> somehow >> runs itself again and again - so fast that "killall" and "pkill -9 -u" >> can't stop/remove >> the user fast enough before the student's program is run again and >> again... >> >> Here is my definition in limits: >> >> @stud hardcpu 8 >> @stud hardnproc 256 >> @stud hardnofile 1024 >> @stud - maxlogins 6 >> >> I can't reduce cpu time below 8min because eclipse will be killed every >> hour or so. >> My problem seems to be that the student can run up to 256 processes that >> each >> uses 100% of a single CPU and we only have 16 CPU''s. >> >> Thanks for any suggestions. >> Josh >> >> >> ___ >> Linux-il mailing list >> Linux-il@cs.huji.ac.il >> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il >> >> > > ___ > Linux-il mailing list > Linux-il@cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Preventing a single student programmer from choking the whole server.
how about using cgroups, putting each user's login shell in a cgroup that cannot use more then X% of the whole CPUs. this will affect all processes spawned under the user's shell. --guy On 04/21/2017 03:07 PM, Josh Roden wrote: Hi server setup: -- Centos 6 32GB RAM 16 Cpu's 70 students max I am using /etc/security/limits.conf to prevent the students from choking the whole server but sometimes one student will write a very bad program that somehow runs itself again and again - so fast that "killall" and "pkill -9 -u" can't stop/remove the user fast enough before the student's program is run again and again... Here is my definition in limits: @stud hardcpu 8 @stud hardnproc 256 @stud hardnofile 1024 @stud - maxlogins 6 I can't reduce cpu time below 8min because eclipse will be killed every hour or so. My problem seems to be that the student can run up to 256 processes that each uses 100% of a single CPU and we only have 16 CPU''s. Thanks for any suggestions. Josh ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il