Re: Putting Limits on Memory and CPU Usage in python ?

[SHINTO PETER]

Hi ,
Erik Cederstrand , i just want to limit the memory (maximum heap size for
your process) with in the python application/Service ie in it client Socket
Program.

I found a solution like below

import resource
rsrc = resource.RLIMIT_DATA
soft, hard = resource.getrlimit(rsrc)
resource.setrlimit(rsrc, (1024, hard))
Link :
http://stackoverflow.com/questions/2308091/how-to-limit-python-heap-size

On Tue, Apr 21, 2015 at 1:18 PM, Erik Cederstrand  wrote:

>
> > Den 21/04/2015 kl. 09.20 skrev SHINTO PETER :
> >
> > Hi
> > François Schiettecatte , limit memory and CPU usage for python socket
> client service
>
> Really, if you want qualified help, you need to be more verbose.
>
> Do you want to kill a process violating your limits? Or can the process be
> expected to cooperate, so you can communicate to a process that it needs to
> cut down on memory usage / CPU? Are these hard limits, or can the process
> use available resources if they are available? Do you want to build some
> sort of accounting, like AWS pay-per-use?
>
> Anyway, this is not really a Django or Python problem. It's a hard problem
> to solve in most operating systems, so you may have better luck asking this
> in a forum related to your operating system.
>
> Erik
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-users/LE6J4sRujbQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/63B8E65D-E694-4F22-AA89-4732430FD840%40cederstrand.dk
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
SHINTO PETER
NIRAPPIL
09591040010

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAKggVnBz7uSVyxvtuj%2BRDhVU_zm_OsjB2o3-BoYVSGd2dt5j-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[Erik Cederstrand]

> Den 21/04/2015 kl. 09.20 skrev SHINTO PETER :
> 
> Hi 
> François Schiettecatte , limit memory and CPU usage for python socket client 
> service

Really, if you want qualified help, you need to be more verbose.

Do you want to kill a process violating your limits? Or can the process be 
expected to cooperate, so you can communicate to a process that it needs to cut 
down on memory usage / CPU? Are these hard limits, or can the process use 
available resources if they are available? Do you want to build some sort of 
accounting, like AWS pay-per-use?

Anyway, this is not really a Django or Python problem. It's a hard problem to 
solve in most operating systems, so you may have better luck asking this in a 
forum related to your operating system.

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/63B8E65D-E694-4F22-AA89-4732430FD840%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[David Palao]

Hi,
Have you had a look at "psutil"? It is a python module to manage processes.

Best

2015-04-21 9:20 GMT+02:00 SHINTO PETER :
> Hi
> François Schiettecatte , limit memory and CPU usage for python socket client
> service
>
> On Mon, Apr 20, 2015 at 11:24 PM, François Schiettecatte
>  wrote:
>>
>> Bummer, shows how long it has been since I have used that.
>>
>> Peter, why do you want to limit memory and CPU usage? To prevent someone
>> from killing the machine ?
>>
>> F.
>>
>> > On Apr 20, 2015, at 1:47 PM, Tim Chase 
>> > wrote:
>> >
>> > On 2015-04-20 13:13, François Schiettecatte wrote:
>> >> You don’t say what platform you are running on, but on linux/unix
>> >> you can use ulimit, see http://ss64.com/bash/ulimit.html
>> >
>> > I've wanted this occasionally, but "ulimit -m" doesn't appear to work
>> > on most modern Linux installations.  See
>> >
>> >
>> > http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux
>> >
>> > for more info.
>> >
>> > I'm not sure *why* it was removed, as there have been times I wanted
>> > this functionality (I'm looking at you, Firefox, and your
>> > memory-gorging habits) so it would be awfully helpful to have
>> > around.  I also just tested "ulimit -m" on OpenBSD and it too seems
>> > to let the RSS of the process grow beyond the specified limit.  I've
>> > got a FreeBSD laptop around here that I could test as well if I can
>> > find the power adaptor.
>> >
>> > -tkc
>> >
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Django users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an email to django-users+unsubscr...@googlegroups.com.
>> > To post to this group, send email to django-users@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/django-users.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msgid/django-users/20150420124711.0f49d04d%40bigbox.christie.dr.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Django users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/django-users/LE6J4sRujbQ/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-users.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-users/E8425E9D-C9B9-4474-B6BC-3B719C599D23%40gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> SHINTO PETER
> NIRAPPIL
> 09591040010
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CAKggVnC%3DJGo%2BKxfE21%3DGAJY%2BTS%3DhTCJ8Ba1ZAVupLAHFanA1Yw%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAKUKWznQC3K0E044R4a1G-%2B067X1a-j9twrT_TBpear%2BS47cOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[SHINTO PETER]

Hi
François Schiettecatte , limit memory and CPU usage for python socket
client service


On Mon, Apr 20, 2015 at 11:24 PM, François Schiettecatte <
fschietteca...@gmail.com> wrote:

> Bummer, shows how long it has been since I have used that.
>
> Peter, why do you want to limit memory and CPU usage? To prevent someone
> from killing the machine ?
>
> F.
>
> > On Apr 20, 2015, at 1:47 PM, Tim Chase 
> wrote:
> >
> > On 2015-04-20 13:13, François Schiettecatte wrote:
> >> You don't say what platform you are running on, but on linux/unix
> >> you can use ulimit, see http://ss64.com/bash/ulimit.html
> >
> > I've wanted this occasionally, but "ulimit -m" doesn't appear to work
> > on most modern Linux installations.  See
> >
> >
> http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux
> >
> > for more info.
> >
> > I'm not sure *why* it was removed, as there have been times I wanted
> > this functionality (I'm looking at you, Firefox, and your
> > memory-gorging habits) so it would be awfully helpful to have
> > around.  I also just tested "ulimit -m" on OpenBSD and it too seems
> > to let the RSS of the process grow beyond the specified limit.  I've
> > got a FreeBSD laptop around here that I could test as well if I can
> > find the power adaptor.
> >
> > -tkc
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users+unsubscr...@googlegroups.com.
> > To post to this group, send email to django-users@googlegroups.com.
> > Visit this group at http://groups.google.com/group/django-users.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/20150420124711.0f49d04d%40bigbox.christie.dr
> .
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-users/LE6J4sRujbQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/E8425E9D-C9B9-4474-B6BC-3B719C599D23%40gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
SHINTO PETER
NIRAPPIL
09591040010

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAKggVnC%3DJGo%2BKxfE21%3DGAJY%2BTS%3DhTCJ8Ba1ZAVupLAHFanA1Yw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[Erik Cederstrand]

> Den 20/04/2015 kl. 19.46 skrev SHINTO PETER :
> 
> Working with linux/unix

If unix includes FreeBSD, check out 
https://www.freebsd.org/doc/en/books/handbook/security-resourcelimits.html

Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/452CBB5B-7837-45A8-94B9-0CBBAA62ECAD%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[Tim Chase]

On 2015-04-20 16:01, François Schiettecatte wrote:
> I did a little research on this and found cpulimit, you can also
> use nice and renice,

Yes, nice/renice and their I/O friend ionice (in my Debian
installation, it's part of the "util-linux" package) work quite
nicely.  It's the memory-limiting that I wish would work.

> and (on fedora core 21 at least) you can do
> 'cat /proc/###/status’ where ### is the process ID which will give
> you memory stats on a process. Not ideal but you can wrap all of
> these in some python to make it easier to manage. 

Yes, I suppose you could have a watch-dog that will monitor memory
usage and then kill the process.  I'd rather the process get an
out-of-memory error if it tried surpassing a set threshold (what
ulimit is *supposed* to do).

-tkc


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20150420160642.62433676%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[François Schiettecatte]

I did a little research on this and found cpulimit, you can also use nice and 
renice, and (on fedora core 21 at least) you can do 'cat /proc/###/status’ 
where ### is the process ID which will give you memory stats on a process. Not 
ideal but you can wrap all of these in some python to make it easier to manage. 

François

> On Apr 20, 2015, at 1:58 PM, Tim Chase  wrote:
> 
> On 2015-04-20 13:54, François Schiettecatte wrote:
>> Peter, why do you want to limit memory and CPU usage? To prevent
>> someone from killing the machine ?
> 
> I know I've wanted it to help prevent malicious data that could suck
> up my RAM and push things into swap.  Absent "ulimit -a", I've had to
> work around it with extra care to ensure that huge files or pessimal
> instructions don't drag my machine to the ground.
> 
> -tkc
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/20150420125834.3faf7102%40bigbox.christie.dr.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/47D4DCD3-F55D-41A2-A4D9-4B6746C7BEA9%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[Tim Chase]

On 2015-04-20 13:54, François Schiettecatte wrote:
> Peter, why do you want to limit memory and CPU usage? To prevent
> someone from killing the machine ?

I know I've wanted it to help prevent malicious data that could suck
up my RAM and push things into swap.  Absent "ulimit -a", I've had to
work around it with extra care to ensure that huge files or pessimal
instructions don't drag my machine to the ground.

-tkc


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20150420125834.3faf7102%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[François Schiettecatte]

Bummer, shows how long it has been since I have used that.

Peter, why do you want to limit memory and CPU usage? To prevent someone from 
killing the machine ?

F.

> On Apr 20, 2015, at 1:47 PM, Tim Chase  wrote:
> 
> On 2015-04-20 13:13, François Schiettecatte wrote:
>> You don’t say what platform you are running on, but on linux/unix
>> you can use ulimit, see http://ss64.com/bash/ulimit.html
> 
> I've wanted this occasionally, but "ulimit -m" doesn't appear to work
> on most modern Linux installations.  See
> 
> http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux
> 
> for more info.
> 
> I'm not sure *why* it was removed, as there have been times I wanted
> this functionality (I'm looking at you, Firefox, and your
> memory-gorging habits) so it would be awfully helpful to have
> around.  I also just tested "ulimit -m" on OpenBSD and it too seems
> to let the RSS of the process grow beyond the specified limit.  I've
> got a FreeBSD laptop around here that I could test as well if I can
> find the power adaptor.
> 
> -tkc
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/20150420124711.0f49d04d%40bigbox.christie.dr.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/E8425E9D-C9B9-4474-B6BC-3B719C599D23%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[SHINTO PETER]

Working with linux/unix

On Monday, 20 April 2015 22:44:17 UTC+5:30, François Schiettecatte wrote:
>
> You don’t say what platform you are running on, but on linux/unix you can 
> use ulimit, see http://ss64.com/bash/ulimit.html 
>
> François 
>
> > On Apr 20, 2015, at 1:09 PM, SHINTO PETER  > wrote: 
> > 
> > how to limit memory utilization for a particular python script ? 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django users" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-users...@googlegroups.com . 
> > To post to this group, send email to django...@googlegroups.com 
> . 
> > Visit this group at http://groups.google.com/group/django-users. 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/8bed3762-1dde-45e2-8573-4be58ab4d090%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9e12b909-40e2-48c5-9c7e-58918a4d7b2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[Tim Chase]

On 2015-04-20 13:13, François Schiettecatte wrote:
> You don’t say what platform you are running on, but on linux/unix
> you can use ulimit, see http://ss64.com/bash/ulimit.html

I've wanted this occasionally, but "ulimit -m" doesn't appear to work
on most modern Linux installations.  See

http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux

for more info.

I'm not sure *why* it was removed, as there have been times I wanted
this functionality (I'm looking at you, Firefox, and your
memory-gorging habits) so it would be awfully helpful to have
around.  I also just tested "ulimit -m" on OpenBSD and it too seems
to let the RSS of the process grow beyond the specified limit.  I've
got a FreeBSD laptop around here that I could test as well if I can
find the power adaptor.

-tkc



-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/20150420124711.0f49d04d%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

Re: Putting Limits on Memory and CPU Usage in python ?

[François Schiettecatte]

You don’t say what platform you are running on, but on linux/unix you can use 
ulimit, see http://ss64.com/bash/ulimit.html

François

> On Apr 20, 2015, at 1:09 PM, SHINTO PETER  wrote:
> 
> how to limit memory utilization for a particular python script ?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-users/8bed3762-1dde-45e2-8573-4be58ab4d090%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/704F864E-CD2D-4A31-9D83-5E7BD35F83FA%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Putting Limits on Memory and CPU Usage in python ?

[SHINTO PETER]

how to limit memory utilization for a particular python script ?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/8bed3762-1dde-45e2-8573-4be58ab4d090%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.