Re: Preventing a single student programmer from choking the whole server.

[Josh Roden]

Hi Yedidyah
Loved the SIGSTOP/SIGKILL approach - simple but awsome.
I seems to be working.
Thanks for everyone's input.
Josh

On Sun, Apr 23, 2017 at 9:18 AM, Yedidyah Bar David 
wrote:

> On Sat, Apr 22, 2017 at 12:51 PM, Josh Roden  wrote:
> > Hi
> > I see some explanation about cgroups on Centos 6 here:
> > https://www.digitalocean.com/community/tutorials/how-to-
> limit-resources-using-cgroups-on-centos-6
> > I will check about its implementation for Centos 6 and update my post.
> > Thanks for giving a direction.
> > Josh
> >
> > On Fri, Apr 21, 2017 at 4:27 PM, guy keren 
> wrote:
> >>
> >>
> >> how about using cgroups, putting each user's login shell in a cgroup
> that
> >> cannot use more then X% of the whole CPUs. this will affect all
> processes
> >> spawned under the user's shell.
> >>
> >> --guy
> >>
> >>
> >> On 04/21/2017 03:07 PM, Josh Roden wrote:
> >>>
> >>> Hi
> >>>
> >>> server setup:
> >>> --
> >>> Centos 6
> >>> 32GB RAM
> >>> 16 Cpu's
> >>> 70 students max
> >>>
> >>> I am using /etc/security/limits.conf to prevent the students from
> >>> choking the whole
> >>> server but sometimes one student will write a very bad program that
> >>> somehow
> >>> runs itself again and again - so fast that "killall" and "pkill -9 -u"
> >>> can't stop/remove
> >>> the user fast enough before the student's program is run again and
> >>> again...
> >>>
> >>> Here is my definition in limits:
> >>>
> >>>  @stud   hardcpu 8
> >>>  @stud   hardnproc   256
> >>>  @stud   hardnofile  1024
> >>>  @stud   -   maxlogins   6
> >>>
> >>> I can't reduce cpu time below 8min because eclipse will be killed every
> >>> hour or so.
> >>> My problem seems to be that the student can run up to 256 processes
> that
> >>> each
> >>> uses 100% of a single CPU and we only have 16 CPU''s.
> >>>
> >>> Thanks for any suggestions.
> >>> Josh
>
> Also, to handle actual specific cases, you can:
> 1. Send all processes SIGSTOP (or SIGTSTP)
> 2. Then send them SIGKILL
>
> Why does this work?
>
> First, why doesn't simply sending all SIGKILL works?
> The kernel already limits the user to 256 procs. So when they try to create
> more, they fail. But when you kill some of them, the others will now
> succeed
> creating more.
>
> With above, you first suspend all procs, so that they won't manage to try
> to
> create new ones. Then, when you kill some of them, the others do not create
> new ones and eventually all are killed.
>
> Best,
>
> >>>
> >>>
> >>> ___
> >>> Linux-il mailing list
> >>> Linux-il@cs.huji.ac.il
> >>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
> >>>
> >>
> >>
> >> ___
> >> Linux-il mailing list
> >> Linux-il@cs.huji.ac.il
> >> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
> >
> >
> >
> > ___
> > Linux-il mailing list
> > Linux-il@cs.huji.ac.il
> > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
> >
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Preventing a single student programmer from choking the whole server.

[Josh Roden]

Hi
I see some explanation about cgroups on Centos 6 here:
https://www.digitalocean.com/community/tutorials/how-to-limit-resources-using-cgroups-on-centos-6
I will check about its implementation for Centos 6 and update my post.
Thanks for giving a direction.
Josh

On Fri, Apr 21, 2017 at 4:27 PM, guy keren  wrote:

>
> how about using cgroups, putting each user's login shell in a cgroup that
> cannot use more then X% of the whole CPUs. this will affect all processes
> spawned under the user's shell.
>
> --guy
>
>
> On 04/21/2017 03:07 PM, Josh Roden wrote:
>
>> Hi
>>
>> server setup:
>> --
>> Centos 6
>> 32GB RAM
>> 16 Cpu's
>> 70 students max
>>
>> I am using /etc/security/limits.conf to prevent the students from
>> choking the whole
>> server but sometimes one student will write a very bad program that
>> somehow
>> runs itself again and again - so fast that "killall" and "pkill -9 -u"
>> can't stop/remove
>> the user fast enough before the student's program is run again and
>> again...
>>
>> Here is my definition in limits:
>>
>>  @stud   hardcpu 8
>>  @stud   hardnproc   256
>>  @stud   hardnofile  1024
>>  @stud   -   maxlogins   6
>>
>> I can't reduce cpu time below 8min because eclipse will be killed every
>> hour or so.
>> My problem seems to be that the student can run up to 256 processes that
>> each
>> uses 100% of a single CPU and we only have 16 CPU''s.
>>
>> Thanks for any suggestions.
>> Josh
>>
>>
>> ___
>> Linux-il mailing list
>> Linux-il@cs.huji.ac.il
>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>
>>
>
> ___
> Linux-il mailing list
> Linux-il@cs.huji.ac.il
> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Re: Preventing a single student programmer from choking the whole server.

[guy keren]

how about using cgroups, putting each user's login shell in a cgroup 
that cannot use more then X% of the whole CPUs. this will affect all 
processes spawned under the user's shell.


--guy

On 04/21/2017 03:07 PM, Josh Roden wrote:

Hi

server setup:
--
Centos 6
32GB RAM
16 Cpu's
70 students max

I am using /etc/security/limits.conf to prevent the students from
choking the whole
server but sometimes one student will write a very bad program that somehow
runs itself again and again - so fast that "killall" and "pkill -9 -u"
can't stop/remove
the user fast enough before the student's program is run again and again...

Here is my definition in limits:

 @stud   hardcpu 8
 @stud   hardnproc   256
 @stud   hardnofile  1024
 @stud   -   maxlogins   6

I can't reduce cpu time below 8min because eclipse will be killed every
hour or so.
My problem seems to be that the student can run up to 256 processes that
each
uses 100% of a single CPU and we only have 16 CPU''s.

Thanks for any suggestions.
Josh


___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il




___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

Preventing a single student programmer from choking the whole server.

[Josh Roden]

Hi

server setup:
--
Centos 6
32GB RAM
16 Cpu's
70 students max

I am using /etc/security/limits.conf to prevent the students from choking
the whole
server but sometimes one student will write a very bad program that somehow
runs itself again and again - so fast that "killall" and "pkill -9 -u"
can't stop/remove
the user fast enough before the student's program is run again and again...

Here is my definition in limits:

 @stud   hardcpu 8
 @stud   hardnproc   256
 @stud   hardnofile  1024
 @stud   -   maxlogins   6

I can't reduce cpu time below 8min because eclipse will be killed every
hour or so.
My problem seems to be that the student can run up to 256 processes that
each
uses 100% of a single CPU and we only have 16 CPU''s.

Thanks for any suggestions.
Josh
___
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il