Re: [web2py] Re: Memory leak - followup
I don't have permission to post in web2py-developers but I added a comment to the bug report. I think the problem is the cached object is the class definition, not an instance of the class. The class definition likely has references to the environment or global namespace in order it to function properly (inheritance, attribute enhancement etc.). In the bug posting I changed the sample code to create an instance of class Blah using blah1 = Blah(), cached that and the before and after reference counts for the gluon objects did not double according to guppy. Ron
Re: [web2py] Re: Memory leak - followup
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Let' move this discussion to web2py-developers. If you are not already > there, please join. OK, the discussion continues there: http://groups.google.com/group/web2py-developers/t/136534ec35b48af8 and this is relevant issue: https://code.google.com/p/web2py/issues/detail?id=146 David - -- David Zejda, Open-IT cz web development & services http://www.o-it.info -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAk0jvOsACgkQ3oCkkciamVFe5gCgqT2vUFPNcJptDrwNAVhk7FYt nEcAnR0UEjp0JRCsrl4AXRt+lVmzOAVB =jJDx -END PGP SIGNATURE-
Re: [web2py] Re: Memory leak - followup
David,
please open a ticket here also:
https://code.google.com/p/web2py/issues/entry
2011/1/4 mdipierro :
> This is odd. I can reproduce the problem. What is even stranger is
> that if I call blahstuff once the count doubles from 24 to 48 but if I
> blahstuff more than once (even if with lower cache time) it does not
> increase the counter more than 48.
>
> I also tried caching a lambda:repr(Blah()) as opposed to Blah and the
> problem does not occur.
>
> Looks like when caching an instance it keep a copy in cache of the
> entire environment, which includes db.
>
> I do not understand why that happens since there is not reference from
> the cache.py code to the environment nor any reference from the Blah
> class.
>
> Let' move this discussion to web2py-developers. If you are not already
> there, please join.
>
> Massimo
>
>
> On Jan 4, 12:10 pm, David Zejda wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Hi :)
>>
>> You may check the issue even with the default simple application created
>> by web admin. Simply add this to the default controller:
>>
>> class Blah():
>> def __init__(self):
>> pass
>>
>> def blahstuff():
>> p = cache.ram('blahblah',Blah,time_expire=30)
>> return dict(p=BEAUTIFY(p))
>>
>> def guppy():
>> from guppy import hpy
>> h = hpy()
>> label='h.heap()'
>> form = FORM(LABEL("Guppy code: "),INPUT(_name="code", _size='35',
>> _value=label),INPUT(_type="submit", _value="Execute.."))
>> if form.accepts(request.vars, session):
>> heap = eval(request.vars.code)
>> label = request.vars.code
>> else:
>> heap = h.heap()
>> fullstack = h.heap().parts
>> return dict(heap=PRE(heap), fullstack=PRE(BEAUTIFY(fullstack)),
>> label=label, form=form)
>>
>> If you visit heapy() first time, the results contains:
>>
>> Partition of a set of 24 objects. Total size = 80448 bytes.
>> Index Count % Size % Cumulative % Kind (class / dict of class)
>> 0 24 100 80448 100 80448 100 dict of gluon.dal.Field
>>
>> Once you run blahstuff(), heapy() reports:
>>
>> Partition of a set of 48 objects. Total size = 160896 bytes.
>> Index Count % Size % Cumulative % Kind (class / dict of class)
>> 0 48 100 160896 100 160896 100 dict of gluon.dal.Field
>>
>> David
>>
>> mdipierro wrote:
>> > can you show us the guppy stats before and after caching? without
>> > caching any db object?
>> > can you also email me the entire app code?
>>
>> > On Jan 4, 9:15 am, David Zejda wrote:
>> > Whenever in controller/model is the class declared, the same result.
>>
>> > Michele Comitini wrote:
>> Try to put the Blah class in the global scope of the controller. Do
>> you get same result?
>>
>> - --
>> David Zejda, Open-IT cz
>> web development & serviceshttp://www.o-it.info
>> -BEGIN PGP SIGNATURE-
>> Version: GnuPG v1.4.9 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org
>>
>> iEYEARECAAYFAk0jYp0ACgkQ3oCkkciamVEyngCfeFLsLcFyo3+97O0wc0w/cbPM
>> oI8AoLY5t0URVVk2+ehOFomsMAjZzlyv
>> =0hmM
>> -END PGP SIGNATURE-
[web2py] Re: Memory leak - followup
This is odd. I can reproduce the problem. What is even stranger is
that if I call blahstuff once the count doubles from 24 to 48 but if I
blahstuff more than once (even if with lower cache time) it does not
increase the counter more than 48.
I also tried caching a lambda:repr(Blah()) as opposed to Blah and the
problem does not occur.
Looks like when caching an instance it keep a copy in cache of the
entire environment, which includes db.
I do not understand why that happens since there is not reference from
the cache.py code to the environment nor any reference from the Blah
class.
Let' move this discussion to web2py-developers. If you are not already
there, please join.
Massimo
On Jan 4, 12:10 pm, David Zejda wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi :)
>
> You may check the issue even with the default simple application created
> by web admin. Simply add this to the default controller:
>
> class Blah():
> def __init__(self):
> pass
>
> def blahstuff():
> p = cache.ram('blahblah',Blah,time_expire=30)
> return dict(p=BEAUTIFY(p))
>
> def guppy():
> from guppy import hpy
> h = hpy()
> label='h.heap()'
> form = FORM(LABEL("Guppy code: "),INPUT(_name="code", _size='35',
> _value=label),INPUT(_type="submit", _value="Execute.."))
> if form.accepts(request.vars, session):
> heap = eval(request.vars.code)
> label = request.vars.code
> else:
> heap = h.heap()
> fullstack = h.heap().parts
> return dict(heap=PRE(heap), fullstack=PRE(BEAUTIFY(fullstack)),
> label=label, form=form)
>
> If you visit heapy() first time, the results contains:
>
> Partition of a set of 24 objects. Total size = 80448 bytes.
> Index Count % Size % Cumulative % Kind (class / dict of class)
> 0 24 100 80448 100 80448 100 dict of gluon.dal.Field
>
> Once you run blahstuff(), heapy() reports:
>
> Partition of a set of 48 objects. Total size = 160896 bytes.
> Index Count % Size % Cumulative % Kind (class / dict of class)
> 0 48 100 160896 100 160896 100 dict of gluon.dal.Field
>
> David
>
> mdipierro wrote:
> > can you show us the guppy stats before and after caching? without
> > caching any db object?
> > can you also email me the entire app code?
>
> > On Jan 4, 9:15 am, David Zejda wrote:
> > Whenever in controller/model is the class declared, the same result.
>
> > Michele Comitini wrote:
> Try to put the Blah class in the global scope of the controller. Do
> you get same result?
>
> - --
> David Zejda, Open-IT cz
> web development & serviceshttp://www.o-it.info
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org
>
> iEYEARECAAYFAk0jYp0ACgkQ3oCkkciamVEyngCfeFLsLcFyo3+97O0wc0w/cbPM
> oI8AoLY5t0URVVk2+ehOFomsMAjZzlyv
> =0hmM
> -END PGP SIGNATURE-
Re: [web2py] Re: Memory leak - followup
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi :)
You may check the issue even with the default simple application created
by web admin. Simply add this to the default controller:
class Blah():
def __init__(self):
pass
def blahstuff():
p = cache.ram('blahblah',Blah,time_expire=30)
return dict(p=BEAUTIFY(p))
def guppy():
from guppy import hpy
h = hpy()
label='h.heap()'
form = FORM(LABEL("Guppy code: "),INPUT(_name="code", _size='35',
_value=label),INPUT(_type="submit", _value="Execute.."))
if form.accepts(request.vars, session):
heap = eval(request.vars.code)
label = request.vars.code
else:
heap = h.heap()
fullstack = h.heap().parts
return dict(heap=PRE(heap), fullstack=PRE(BEAUTIFY(fullstack)),
label=label, form=form)
If you visit heapy() first time, the results contains:
Partition of a set of 24 objects. Total size = 80448 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 24 10080448 100 80448 100 dict of gluon.dal.Field
Once you run blahstuff(), heapy() reports:
Partition of a set of 48 objects. Total size = 160896 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 48 100 160896 100160896 100 dict of gluon.dal.Field
David
mdipierro wrote:
> can you show us the guppy stats before and after caching? without
> caching any db object?
> can you also email me the entire app code?
>
> On Jan 4, 9:15 am, David Zejda wrote:
> Whenever in controller/model is the class declared, the same result.
>
> Michele Comitini wrote:
Try to put the Blah class in the global scope of the controller. Do
you get same result?
- --
David Zejda, Open-IT cz
web development & services
http://www.o-it.info
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAk0jYp0ACgkQ3oCkkciamVEyngCfeFLsLcFyo3+97O0wc0w/cbPM
oI8AoLY5t0URVVk2+ehOFomsMAjZzlyv
=0hmM
-END PGP SIGNATURE-
[web2py] Re: Memory leak - followup
can you show us the guppy stats before and after caching? without caching any db object? can you also email me the entire app code? On Jan 4, 9:15 am, David Zejda wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Whenever in controller/model is the class declared, the same result. > > Michele Comitini wrote: > > Try to put the Blah class in the global scope of the controller. Do > > you get same result? > > - -- > David Zejda, Open-IT cz > web development & serviceshttp://www.o-it.info > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org > > iEYEARECAAYFAk0jOaoACgkQ3oCkkciamVH++gCgpR/w98HwRVUgTqOpCbj+/ITE > LrwAn3b34zURtjOXDb3rXIsX3bwDXc4v > =2l85 > -END PGP SIGNATURE-
Re: [web2py] Re: Memory leak - followup
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Whenever in controller/model is the class declared, the same result. Michele Comitini wrote: > Try to put the Blah class in the global scope of the controller. Do > you get same result? - -- David Zejda, Open-IT cz web development & services http://www.o-it.info -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAk0jOaoACgkQ3oCkkciamVH++gCgpR/w98HwRVUgTqOpCbj+/ITE LrwAn3b34zURtjOXDb3rXIsX3bwDXc4v =2l85 -END PGP SIGNATURE-
[web2py] Re: Memory leak - followup
I am puzzled. Let me think about this.
Massimo
On Jan 4, 2:47 am, David Zejda wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi :)
>
> Yes, I am really sure. This is exact code of my controller.
>
> def load_pers():
>
> class Blah():
> def __init__(self):
> pass
>
> def blah2():
> return Blah()
>
> p = cache.ram('blahblah',blah2,time_expire=30)
>
> return dict(p=BEAUTIFY(p))
>
> and this has the same effect:
>
> def load_pers():
>
> class Blah():
> def __init__(self):
> pass
>
> p = cache.ram('blahblah',Blah,time_expire=30)
>
> return dict(p=BEAUTIFY(p))
>
> Simply, if I store an object declared in controller or model in cache
> (regardless of db involvement), whole db model is being duplicated in
> RAM, which is really unwanted behaviour.
>
> As I mentioned in one of previous messages, I have strong suspiction
> that the issue is related to another one discussed here:
>
> http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
>
> D.
>
>
>
> mdipierro wrote:
> > Sorry. I got confused too. You say this leads to the leak:
>
> > class Blah():
> > def __init__(self):
> > pass
> > def blah2():
> > return Blah()
> > p = cache.ram('blahblah',blah2,time_expire=30)
>
> > so there is o db involved? Are you sure 'blahblah' is a constant in
> > your code and not a variable? Is this the exact code you are running?
>
> > On Jan 3, 5:42 pm, David Zejda wrote:
> > Hello Massimo,
>
> > sorry, but I really do not understand. :-|
>
> > How do I store whole record if the only thing I'm trying to store is
> > instance of a dummy class? In the last example with empty __init__ there
> > is no record involved at all!
>
> > If I do exactly what Michele suggested:
>
> > class Blah():
> > def __init__(self):
> > self.nick = db.person[1].nick
>
> > def blah_f():
> > return Blah()
>
> > p = cache.ram('blahblah', blah_f,time_expire=30)
>
> > result is the same, of course.
>
> > David
>
> - --
> David Zejda, Open-IT cz
> web development & serviceshttp://www.o-it.info
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org
>
> iEYEARECAAYFAk0i3rUACgkQ3oCkkciamVEz2wCdGXIgszz1wjDCR6/gHMOSqpda
> T+QAnjbShkt47zF8D1NLzrjm0zrXEyTL
> =88aY
> -END PGP SIGNATURE-
Re: [web2py] Re: Memory leak - followup
Try to put the Blah class in the global scope of the controller. Do
you get same result?
2011/1/4 David Zejda :
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi :)
>
> Yes, I am really sure. This is exact code of my controller.
>
> def load_pers():
>
> class Blah():
> def __init__(self):
> pass
>
> def blah2():
> return Blah()
>
> p = cache.ram('blahblah',blah2,time_expire=30)
>
> return dict(p=BEAUTIFY(p))
>
> and this has the same effect:
>
> def load_pers():
>
> class Blah():
> def __init__(self):
> pass
>
> p = cache.ram('blahblah',Blah,time_expire=30)
>
> return dict(p=BEAUTIFY(p))
>
> Simply, if I store an object declared in controller or model in cache
> (regardless of db involvement), whole db model is being duplicated in
> RAM, which is really unwanted behaviour.
>
> As I mentioned in one of previous messages, I have strong suspiction
> that the issue is related to another one discussed here:
>
> http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
>
> D.
>
> mdipierro wrote:
>> Sorry. I got confused too. You say this leads to the leak:
>>
>> class Blah():
>> def __init__(self):
>> pass
>> def blah2():
>> return Blah()
>> p = cache.ram('blahblah',blah2,time_expire=30)
>>
>> so there is o db involved? Are you sure 'blahblah' is a constant in
>> your code and not a variable? Is this the exact code you are running?
>>
>>
>> On Jan 3, 5:42 pm, David Zejda wrote:
>> Hello Massimo,
>>
>> sorry, but I really do not understand. :-|
>>
>> How do I store whole record if the only thing I'm trying to store is
>> instance of a dummy class? In the last example with empty __init__ there
>> is no record involved at all!
>>
>> If I do exactly what Michele suggested:
>>
>> class Blah():
>> def __init__(self):
>> self.nick = db.person[1].nick
>>
>> def blah_f():
>> return Blah()
>>
>> p = cache.ram('blahblah', blah_f,time_expire=30)
>>
>> result is the same, of course.
>>
>> David
>
> - --
> David Zejda, Open-IT cz
> web development & services
> http://www.o-it.info
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAk0i3rUACgkQ3oCkkciamVEz2wCdGXIgszz1wjDCR6/gHMOSqpda
> T+QAnjbShkt47zF8D1NLzrjm0zrXEyTL
> =88aY
> -END PGP SIGNATURE-
>
Re: [web2py] Re: Memory leak - followup
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi :)
Yes, I am really sure. This is exact code of my controller.
def load_pers():
class Blah():
def __init__(self):
pass
def blah2():
return Blah()
p = cache.ram('blahblah',blah2,time_expire=30)
return dict(p=BEAUTIFY(p))
and this has the same effect:
def load_pers():
class Blah():
def __init__(self):
pass
p = cache.ram('blahblah',Blah,time_expire=30)
return dict(p=BEAUTIFY(p))
Simply, if I store an object declared in controller or model in cache
(regardless of db involvement), whole db model is being duplicated in
RAM, which is really unwanted behaviour.
As I mentioned in one of previous messages, I have strong suspiction
that the issue is related to another one discussed here:
http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
D.
mdipierro wrote:
> Sorry. I got confused too. You say this leads to the leak:
>
> class Blah():
> def __init__(self):
> pass
> def blah2():
> return Blah()
> p = cache.ram('blahblah',blah2,time_expire=30)
>
> so there is o db involved? Are you sure 'blahblah' is a constant in
> your code and not a variable? Is this the exact code you are running?
>
>
> On Jan 3, 5:42 pm, David Zejda wrote:
> Hello Massimo,
>
> sorry, but I really do not understand. :-|
>
> How do I store whole record if the only thing I'm trying to store is
> instance of a dummy class? In the last example with empty __init__ there
> is no record involved at all!
>
> If I do exactly what Michele suggested:
>
> class Blah():
> def __init__(self):
> self.nick = db.person[1].nick
>
> def blah_f():
> return Blah()
>
> p = cache.ram('blahblah', blah_f,time_expire=30)
>
> result is the same, of course.
>
> David
- --
David Zejda, Open-IT cz
web development & services
http://www.o-it.info
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAk0i3rUACgkQ3oCkkciamVEz2wCdGXIgszz1wjDCR6/gHMOSqpda
T+QAnjbShkt47zF8D1NLzrjm0zrXEyTL
=88aY
-END PGP SIGNATURE-
[web2py] Re: Memory leak - followup
Sorry. I got confused too. You say this leads to the leak:
class Blah():
def __init__(self):
pass
def blah2():
return Blah()
p = cache.ram('blahblah',blah2,time_expire=30)
so there is o db involved? Are you sure 'blahblah' is a constant in
your code and not a variable? Is this the exact code you are running?
On Jan 3, 5:42 pm, David Zejda wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hello Massimo,
>
> sorry, but I really do not understand. :-|
>
> How do I store whole record if the only thing I'm trying to store is
> instance of a dummy class? In the last example with empty __init__ there
> is no record involved at all!
>
> If I do exactly what Michele suggested:
>
> class Blah():
> def __init__(self):
> self.nick = db.person[1].nick
>
> def blah_f():
> return Blah()
>
> p = cache.ram('blahblah', blah_f,time_expire=30)
>
> result is the same, of course.
>
> David
>
>
>
> mdipierro wrote:
> > This is not what Michele suggested. You are still storing the entire
> > record.
>
> > On Jan 3, 12:22 pm, David Zejda wrote:
> > Dear Massimo,
>
> > thank you for the reply, but Michele's advice makes no difference.
>
> > I just noticed that even this one leads to "the leak":
>
> > class Blah():
> > def __init__(self):
> > pass
>
> > def blah2():
> > return Blah()
>
> > p = cache.ram('blahblah',blah2,time_expire=30)
>
> > Since I avoid storing objects of classes declared in model or
> > controllers I do not have problems. But still I do not understand why
> > when I store such object in cache, whole copy of db model goes into
> > memory. IMO if there is no solution for the behaviour, at least the
> > danger should be emphasized in cache docs, because I doubt that anybody
> > would anticipate it.
>
> > Wising you a nice day!
> > David
>
> > mdipierro wrote:
> yes but you are not following Michele's advice. If you cache a record
> or a an object like a reference, you are storing in ram of copy db
> (the leak). You should cache values or dictionaries. As long as those
> values are not reference fields there should be no leaks.
> I am not sure the term leak is appropriate here. When you cache you
> store something and storage takes space, until you clear cache. The
> bigger the object you cache, the more space it takes.
> On Jan 3, 4:21 am, David Zejda wrote:
> Hi,
> nick is not reference field. To make it sure that the issue is not
> dependent on the field definition I checked it with as simple Field as
> possible:
> Field('trustworthiness', 'double', default=0),
> defined in my model and added conversion to str:
> class Blah():
> def __init__(self):
> self.nick = str(db.person[1].trustworthiness)
> p = cache.ram('blahblah',Blah,time_expire=30)
> Also,
> def blah_f():
> return Blah()
> makes no change, leads to the same leak.
> The leak appears regardless to the place of the declaration of Blah
> class. I tried to declare it in top level of model, in controller, and
> inside function of controller, no difference.
> It is possible that this IMO weird behaviour is related to another issue
> which teased me months ago:
> http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
> With regards,
> David
> mdipierro wrote:
> >>> It depends on whether nick is a reference field.
> >>> On Jan 2, 5:31 pm, Michele Comitini
> >>> wrote:
> what if you do this?
> class Blah():
> def __init__(self):
> self.nick = db.person[1].nick
> def blah_f():
> return Blah()
> p = cache.ram('blahblah', blah_f,time_expire=30)
> 2011/1/2 David Zejda :
>
> - --
> David Zejda, Open-IT cz
> web development & serviceshttp://www.o-it.info
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org
>
> iEYEARECAAYFAk0iXskACgkQ3oCkkciamVHYZgCdFltRuwqLh1y3NPmS45VxNfnh
> 9iUAniwUeoXlSbtCniEnc99Q0sKzIm9+
> =SRTR
> -END PGP SIGNATURE-
Re: [web2py] Re: Memory leak - followup
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hello Massimo,
sorry, but I really do not understand. :-|
How do I store whole record if the only thing I'm trying to store is
instance of a dummy class? In the last example with empty __init__ there
is no record involved at all!
If I do exactly what Michele suggested:
class Blah():
def __init__(self):
self.nick = db.person[1].nick
def blah_f():
return Blah()
p = cache.ram('blahblah', blah_f,time_expire=30)
result is the same, of course.
David
mdipierro wrote:
> This is not what Michele suggested. You are still storing the entire
> record.
>
> On Jan 3, 12:22 pm, David Zejda wrote:
> Dear Massimo,
>
> thank you for the reply, but Michele's advice makes no difference.
>
> I just noticed that even this one leads to "the leak":
>
> class Blah():
> def __init__(self):
> pass
>
> def blah2():
> return Blah()
>
> p = cache.ram('blahblah',blah2,time_expire=30)
>
> Since I avoid storing objects of classes declared in model or
> controllers I do not have problems. But still I do not understand why
> when I store such object in cache, whole copy of db model goes into
> memory. IMO if there is no solution for the behaviour, at least the
> danger should be emphasized in cache docs, because I doubt that anybody
> would anticipate it.
>
> Wising you a nice day!
> David
>
>
>
> mdipierro wrote:
yes but you are not following Michele's advice. If you cache a record
or a an object like a reference, you are storing in ram of copy db
(the leak). You should cache values or dictionaries. As long as those
values are not reference fields there should be no leaks.
I am not sure the term leak is appropriate here. When you cache you
store something and storage takes space, until you clear cache. The
bigger the object you cache, the more space it takes.
On Jan 3, 4:21 am, David Zejda wrote:
Hi,
nick is not reference field. To make it sure that the issue is not
dependent on the field definition I checked it with as simple Field as
possible:
Field('trustworthiness', 'double', default=0),
defined in my model and added conversion to str:
class Blah():
def __init__(self):
self.nick = str(db.person[1].trustworthiness)
p = cache.ram('blahblah',Blah,time_expire=30)
Also,
def blah_f():
return Blah()
makes no change, leads to the same leak.
The leak appears regardless to the place of the declaration of Blah
class. I tried to declare it in top level of model, in controller, and
inside function of controller, no difference.
It is possible that this IMO weird behaviour is related to another issue
which teased me months ago:
http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
With regards,
David
mdipierro wrote:
>>> It depends on whether nick is a reference field.
>>> On Jan 2, 5:31 pm, Michele Comitini
>>> wrote:
what if you do this?
class Blah():
def __init__(self):
self.nick = db.person[1].nick
def blah_f():
return Blah()
p = cache.ram('blahblah', blah_f,time_expire=30)
2011/1/2 David Zejda :
- --
David Zejda, Open-IT cz
web development & services
http://www.o-it.info
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAk0iXskACgkQ3oCkkciamVHYZgCdFltRuwqLh1y3NPmS45VxNfnh
9iUAniwUeoXlSbtCniEnc99Q0sKzIm9+
=SRTR
-END PGP SIGNATURE-
[web2py] Re: Memory leak - followup
This is not what Michele suggested. You are still storing the entire
record.
On Jan 3, 12:22 pm, David Zejda wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Dear Massimo,
>
> thank you for the reply, but Michele's advice makes no difference.
>
> I just noticed that even this one leads to "the leak":
>
> class Blah():
> def __init__(self):
> pass
>
> def blah2():
> return Blah()
>
> p = cache.ram('blahblah',blah2,time_expire=30)
>
> Since I avoid storing objects of classes declared in model or
> controllers I do not have problems. But still I do not understand why
> when I store such object in cache, whole copy of db model goes into
> memory. IMO if there is no solution for the behaviour, at least the
> danger should be emphasized in cache docs, because I doubt that anybody
> would anticipate it.
>
> Wising you a nice day!
> David
>
>
>
> mdipierro wrote:
> > yes but you are not following Michele's advice. If you cache a record
> > or a an object like a reference, you are storing in ram of copy db
> > (the leak). You should cache values or dictionaries. As long as those
> > values are not reference fields there should be no leaks.
>
> > I am not sure the term leak is appropriate here. When you cache you
> > store something and storage takes space, until you clear cache. The
> > bigger the object you cache, the more space it takes.
>
> > On Jan 3, 4:21 am, David Zejda wrote:
> > Hi,
>
> > nick is not reference field. To make it sure that the issue is not
> > dependent on the field definition I checked it with as simple Field as
> > possible:
>
> > Field('trustworthiness', 'double', default=0),
>
> > defined in my model and added conversion to str:
>
> > class Blah():
> > def __init__(self):
> > self.nick = str(db.person[1].trustworthiness)
>
> > p = cache.ram('blahblah',Blah,time_expire=30)
>
> > Also,
>
> > def blah_f():
> > return Blah()
>
> > makes no change, leads to the same leak.
>
> > The leak appears regardless to the place of the declaration of Blah
> > class. I tried to declare it in top level of model, in controller, and
> > inside function of controller, no difference.
>
> > It is possible that this IMO weird behaviour is related to another issue
> > which teased me months ago:
>
> >http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
>
> > With regards,
> > David
>
> > mdipierro wrote:
> It depends on whether nick is a reference field.
> On Jan 2, 5:31 pm, Michele Comitini
> wrote:
> > what if you do this?
> > class Blah():
> > def __init__(self):
> > self.nick = db.person[1].nick
> > def blah_f():
> > return Blah()
> > p = cache.ram('blahblah', blah_f,time_expire=30)
> > 2011/1/2 David Zejda :
>
> - --
> David Zejda, Open-IT cz
> web development & serviceshttp://www.o-it.info
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org
>
> iEYEARECAAYFAk0iE/4ACgkQ3oCkkciamVHfNwCdFkpF3okgulfxi7qV1EZA0P1/
> VhAAn0Rd3EXTwwn+fttzIvc83K8ri8bK
> =3lwE
> -END PGP SIGNATURE-
Re: [web2py] Re: Memory leak - followup
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Dear Massimo,
thank you for the reply, but Michele's advice makes no difference.
I just noticed that even this one leads to "the leak":
class Blah():
def __init__(self):
pass
def blah2():
return Blah()
p = cache.ram('blahblah',blah2,time_expire=30)
Since I avoid storing objects of classes declared in model or
controllers I do not have problems. But still I do not understand why
when I store such object in cache, whole copy of db model goes into
memory. IMO if there is no solution for the behaviour, at least the
danger should be emphasized in cache docs, because I doubt that anybody
would anticipate it.
Wising you a nice day!
David
mdipierro wrote:
> yes but you are not following Michele's advice. If you cache a record
> or a an object like a reference, you are storing in ram of copy db
> (the leak). You should cache values or dictionaries. As long as those
> values are not reference fields there should be no leaks.
>
> I am not sure the term leak is appropriate here. When you cache you
> store something and storage takes space, until you clear cache. The
> bigger the object you cache, the more space it takes.
>
> On Jan 3, 4:21 am, David Zejda wrote:
> Hi,
>
> nick is not reference field. To make it sure that the issue is not
> dependent on the field definition I checked it with as simple Field as
> possible:
>
> Field('trustworthiness', 'double', default=0),
>
> defined in my model and added conversion to str:
>
> class Blah():
> def __init__(self):
> self.nick = str(db.person[1].trustworthiness)
>
> p = cache.ram('blahblah',Blah,time_expire=30)
>
> Also,
>
> def blah_f():
> return Blah()
>
> makes no change, leads to the same leak.
>
> The leak appears regardless to the place of the declaration of Blah
> class. I tried to declare it in top level of model, in controller, and
> inside function of controller, no difference.
>
> It is possible that this IMO weird behaviour is related to another issue
> which teased me months ago:
>
> http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
>
> With regards,
> David
>
>
>
> mdipierro wrote:
It depends on whether nick is a reference field.
On Jan 2, 5:31 pm, Michele Comitini
wrote:
> what if you do this?
> class Blah():
> def __init__(self):
> self.nick = db.person[1].nick
> def blah_f():
> return Blah()
> p = cache.ram('blahblah', blah_f,time_expire=30)
> 2011/1/2 David Zejda :
- --
David Zejda, Open-IT cz
web development & services
http://www.o-it.info
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAk0iE/4ACgkQ3oCkkciamVHfNwCdFkpF3okgulfxi7qV1EZA0P1/
VhAAn0Rd3EXTwwn+fttzIvc83K8ri8bK
=3lwE
-END PGP SIGNATURE-
[web2py] Re: Memory leak - followup
yes but you are not following Michele's advice. If you cache a record
or a an object like a reference, you are storing in ram of copy db
(the leak). You should cache values or dictionaries. As long as those
values are not reference fields there should be no leaks.
I am not sure the term leak is appropriate here. When you cache you
store something and storage takes space, until you clear cache. The
bigger the object you cache, the more space it takes.
On Jan 3, 4:21 am, David Zejda wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi,
>
> nick is not reference field. To make it sure that the issue is not
> dependent on the field definition I checked it with as simple Field as
> possible:
>
> Field('trustworthiness', 'double', default=0),
>
> defined in my model and added conversion to str:
>
> class Blah():
> def __init__(self):
> self.nick = str(db.person[1].trustworthiness)
>
> p = cache.ram('blahblah',Blah,time_expire=30)
>
> Also,
>
> def blah_f():
> return Blah()
>
> makes no change, leads to the same leak.
>
> The leak appears regardless to the place of the declaration of Blah
> class. I tried to declare it in top level of model, in controller, and
> inside function of controller, no difference.
>
> It is possible that this IMO weird behaviour is related to another issue
> which teased me months ago:
>
> http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
>
> With regards,
> David
>
>
>
> mdipierro wrote:
> > It depends on whether nick is a reference field.
>
> > On Jan 2, 5:31 pm, Michele Comitini
> > wrote:
> >> what if you do this?
>
> >> class Blah():
> >> def __init__(self):
> >> self.nick = db.person[1].nick
>
> >> def blah_f():
> >> return Blah()
>
> >> p = cache.ram('blahblah', blah_f,time_expire=30)
>
> >> 2011/1/2 David Zejda :
>
> - --
> David Zejda, Open-IT cz
> web development & serviceshttp://www.o-it.info
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org
>
> iEYEARECAAYFAk0hoyEACgkQ3oCkkciamVGE0gCfROfrJo7SaeDxIbPco58LHzt9
> mesAoK9JfcQKnGyAlkKfoeqIGKLJ812Z
> =IBqT
> -END PGP SIGNATURE-
Re: [web2py] Re: Memory leak - followup
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi,
nick is not reference field. To make it sure that the issue is not
dependent on the field definition I checked it with as simple Field as
possible:
Field('trustworthiness', 'double', default=0),
defined in my model and added conversion to str:
class Blah():
def __init__(self):
self.nick = str(db.person[1].trustworthiness)
p = cache.ram('blahblah',Blah,time_expire=30)
Also,
def blah_f():
return Blah()
makes no change, leads to the same leak.
The leak appears regardless to the place of the declaration of Blah
class. I tried to declare it in top level of model, in controller, and
inside function of controller, no difference.
It is possible that this IMO weird behaviour is related to another issue
which teased me months ago:
http://www.mail-archive.com/web2py@googlegroups.com/msg34333.html
With regards,
David
mdipierro wrote:
> It depends on whether nick is a reference field.
>
>
> On Jan 2, 5:31 pm, Michele Comitini
> wrote:
>> what if you do this?
>>
>> class Blah():
>> def __init__(self):
>> self.nick = db.person[1].nick
>>
>> def blah_f():
>> return Blah()
>>
>> p = cache.ram('blahblah', blah_f,time_expire=30)
>>
>> 2011/1/2 David Zejda :
>>
>>
- --
David Zejda, Open-IT cz
web development & services
http://www.o-it.info
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAk0hoyEACgkQ3oCkkciamVGE0gCfROfrJo7SaeDxIbPco58LHzt9
mesAoK9JfcQKnGyAlkKfoeqIGKLJ812Z
=IBqT
-END PGP SIGNATURE-
[web2py] Re: Memory leak - followup
It depends on whether nick is a reference field.
On Jan 2, 5:31 pm, Michele Comitini
wrote:
> what if you do this?
>
> class Blah():
> def __init__(self):
> self.nick = db.person[1].nick
>
> def blah_f():
> return Blah()
>
> p = cache.ram('blahblah', blah_f,time_expire=30)
>
> 2011/1/2 David Zejda :
>
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA1
>
> > Finally I had enough time to find out, where is the main root of my
> > memory leak.
>
> > Consider two scenarios:
>
> > 1. start web2py server
>
> > 2. execute controller function with contents:
>
> > def blah():
> > return Storage(dict(nick=db.person[1].nick))
>
> > p = cache.ram('blahblah',blah,time_expire=30)
>
> > 3. guppy stats:
>
> > Partition of a set of 119902 objects. Total size = 17240104 bytes.
> > Index Count % Size % Cumulative % Kind (class / dict of class)
> > 0 64658 54 5946560 34 5946560 34 str
> > 1 24649 21 2172768 13 8119328 47 tuple
> > 2 624 1 2091648 12 10210976 59 dict of gluon.dal.Field
> > 3 706 1 888112 5 11099088 64 dict (no owner)
> > 4 282 0 876912 5 11976000 69 dict of module
> > 5 6701 6 804120 5 12780120 74 types.CodeType
> > 6 6359 5 763080 4 13543200 79 function
> > 7 652 1 571680 3 14114880 82 type
> > 8 652 1 501280 3 14616160 85 dict of type
> > 9 451 0 454984 3 15071144 87 dict of class
> > <342 more rows. Type e.g. '_.more' to view.>
>
> > 1. start web2py server
>
> > 2. execute controller function with contents (nick is plain string):
>
> > class Blah():
> > def __init__(self):
> > self.nick = db.person[1].nick
>
> > p = cache.ram('blahblah',Blah,time_expire=30)
>
> > 3. guppy stats:
>
> > Partition of a set of 13 objects. Total size = 21017568 bytes.
> > Index Count % Size % Cumulative % Kind (class / dict of class)
> > 0 69722 53 6343688 30 6343688 30 str
> > 1 1248 1 4183296 20 10526984 50 dict of gluon.dal.Field
> > 2 26193 20 2334008 11 12860992 61 tuple
> > 3 756 1 987360 5 13848352 66 dict (no owner)
> > 4 282 0 876912 4 14725264 70 dict of module
> > 5 7056 5 846720 4 15571984 74 types.CodeType
> > 6 6686 5 802320 4 16374304 78 function
> > 7 654 0 573440 3 16947744 81 type
> > 8 654 0 501840 2 17449584 83 dict of type
> > 9 459 0 471048 2 17920632 85 dict of class
> > <356 more rows. Type e.g. '_.more' to view.>
>
> > Please, look at dict of gluon.dal.Field. In the second scenario the
> > count is doubled, which means that db model remains in the memory twice,
> > which is bad.
>
> > I did not try the same with simplier model, so I do not have idea
> > whether it matters.
>
> > In my case it will not be too difficult to replace my own classes with
> > dicts or Storages, but maybe the findings will help somebody else facing
> > similar leak..
>
> > Nice year to all :)
> > David
>
> > - --
> > David Zejda, Open-IT cz
> > web development & services
> >http://www.o-it.info
> > -BEGIN PGP SIGNATURE-
> > Version: GnuPG v1.4.9 (GNU/Linux)
> > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org
>
> > iEYEARECAAYFAk0gqW4ACgkQ3oCkkciamVH0JACfWPEaSQX+7/OqeGeWNhtJr4/R
> > JzwAn3Z17zjKv93dxjDTuM2KorzI6LB/
> > =OKMq
> > -END PGP SIGNATURE-
>
>
