Re: Subtle Memory Leak Finally Found! (DEBUG is off)
On Wed, 2008-11-19 at 11:53 +1100, Malcolm Tredinnick wrote: [...] > Once memory gets to the over-allocated stage, > it will be reclaimed... Thinking about it further, this is rubbish. Details (if you care): Memory allocated with malloc(), etc, won't necessarily be reclaimed, since it can only be given back to the operating system via a call to brk(), which means it can only be removed from the end of a block of memory reserved for that process and there's no guarantee that the memory at the end isn't in use. The gist of my answer still stands, though: the memory isn't necessarily returned to the operating system until the process exits. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Subtle Memory Leak Finally Found! (DEBUG is off)
On Tue, 2008-11-18 at 08:24 -0800, 7timesTom wrote: [...] > Maybe this post will help someone else also one thing I don't > understand: > > Why wasn't the memory returned to me after each page view? Why was an > apache off/on necessary to clear memory? And is there anything I can > do to help clear memory that been used, but not returned to the system? I'm going to guess a bit here, since it's almost impossible to tell without seeing the system in action. But there's one common situation where it looks like a process is "leaking memory" when it's actually just running on a Unix-like system. Normal Unix-like process operation is that malloc'd memory is not returned to the system immediately (it's the difference between the library call malloc() and the system call brk(), in effect). This is because the process will often need to request the same memory again in a short while, so it makes sense to leave it hanging around. Once memory gets to the over-allocated stage, it will be reclaimed and once the process exits, it will also be reclaimed, so there's no long-term memory leak with that model of operation (which has been around for decades -- it isn't a new thing). The problem is that software-layer memory monitoring tools don't know about this and can't tell the system that it is essentially over-allocated. So there's a conflict between normal operation and those sorts of monitoring processes. I don't really know what a solution is there. Been a while since I've had to play with things at that level and I can't remember if ulimit settings (setrlimit() at the C library level) is helpful or a hinderance there, for example. Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Subtle Memory Leak Finally Found! (DEBUG is off)
7timesTom wrote: > cars = Car.objects.filter(...) #upto 2000 items > msg =... len(cars) # can you see the massive memory use here? This is actually well documented: http://docs.djangoproject.com/en/dev/ref/models/querysets/#when-querysets-are-evaluated --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Subtle Memory Leak Finally Found! (DEBUG is off)
Are you using MySQL? It's been implicated in these sorts of issues before: http://nedbatchelder.com/blog/200809/a_server_memory_leak.html --Ned. http://nedbatchelder.com 7timesTom wrote: > Some time ago Webfaction notified me "Memory usage over your > limit" (80MB) It's taken me ages to find this darn memory leak! It was > one short line that didn't even add functionality to my users. > > I have 2000 items (cars) in a database table, which my users can > search through. After the webfaction memory warning, I optimised the > code so that my pagination was in place so that in theory only 10 > items were loaded into memory per page view. But still my memory usage > was massive. > > Anyway today I notice the line: > msg =cgi.escape(str(len(cars)))+'\n' # which stores a debugging > message for me > > That line was between these two lines (spread out by other stuff and > comments): > cars = Car.objects.filter(...) #upto 2000 items > msg =... len(cars) # can you see the massive memory use here? > paginator = Paginator(cars, per_page=10) > > As you all know Django has been well designed to be lazy at executing > database queries, this is how the Paginator can take a queryset > representing all 2000 items, and return just 10 having loaded only > those 10 into memory. > > So what's wrong with len(cars)? Well, it forces the db query to be > executed in full (before the Paginator line) and loaded into memory. > This caused massive memory use and made me go over my limit. > > If I'd wanted the total number of cars, or number of results, I > should've done Car.objects.filter(...).count() which actually executes > a SELECT COUNT(*) which returns a single number, rather than, in > essence, a huge list. > > Maybe this post will help someone else also one thing I don't > understand: > > Why wasn't the memory returned to me after each page view? Why was an > apache off/on necessary to clear memory? And is there anything I can > do to help clear memory that been used, but not returned to the system? > > > > -- Ned Batchelder, http://nedbatchelder.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Subtle Memory Leak Finally Found! (DEBUG is off)
Some time ago Webfaction notified me "Memory usage over your limit" (80MB) It's taken me ages to find this darn memory leak! It was one short line that didn't even add functionality to my users. I have 2000 items (cars) in a database table, which my users can search through. After the webfaction memory warning, I optimised the code so that my pagination was in place so that in theory only 10 items were loaded into memory per page view. But still my memory usage was massive. Anyway today I notice the line: msg =cgi.escape(str(len(cars)))+'\n' # which stores a debugging message for me That line was between these two lines (spread out by other stuff and comments): cars = Car.objects.filter(...) #upto 2000 items msg =... len(cars) # can you see the massive memory use here? paginator = Paginator(cars, per_page=10) As you all know Django has been well designed to be lazy at executing database queries, this is how the Paginator can take a queryset representing all 2000 items, and return just 10 having loaded only those 10 into memory. So what's wrong with len(cars)? Well, it forces the db query to be executed in full (before the Paginator line) and loaded into memory. This caused massive memory use and made me go over my limit. If I'd wanted the total number of cars, or number of results, I should've done Car.objects.filter(...).count() which actually executes a SELECT COUNT(*) which returns a single number, rather than, in essence, a huge list. Maybe this post will help someone else also one thing I don't understand: Why wasn't the memory returned to me after each page view? Why was an apache off/on necessary to clear memory? And is there anything I can do to help clear memory that been used, but not returned to the system? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
