On 2/25/07, Jens Vagelpohl <[EMAIL PROTECTED]> wrote:
On 25 Feb 2007, at 00:05, Jean Lagarde wrote: > I fully understand that Zope has to be restarted regularly This may be needed in a situation where you have a memory-hungry application (like Plone), but otherwise there is no general need to restart Zope regularly. My websites run months without restart, and they don't balloon in memory footprint.
Thanks Roché and Jens. Yes, of course I meant this in the context of a public Plone application.
That setting is a little bit misleading. These values represent targets, not hard limits. If a request pulls in a much larger number of objects into memory then that chunk of memory is now held by Python and won't be released, small target value or not.
I also realize that. Although here the part that is not clear to me: Python holds that memory and won't release it to the OS, but can't it reuse it internally for future object loads? If it just keeps piling up objects in memory then even talking of a target is meaningless.
> You would need to study all the inefficiencies inherent in the > content types you use, or all the nasty stuff that gets pulled in > for viewing the site furniture shown for every page etc. Very few > product authors out there put memory efficiency onto their priority > list. P.S.: Memory is cheap. Cheap enough even for a non-profit. Go get some, or at least compare the cost of buying meory with the cost of countless hours you already have and may continue to spend on this. I'm not saying it's not a worthwhile task to try and get to the bottom of code inefficiencies, but keep the practical side in mind.
Yep, that's what I keep on telling my co-administrators. As I said we actually have a lot more physical memory available, but I need their buy-in to increase the process limits on FreeBSD (I'm not succeeding by the way -- got an email this morning suggesting that Zope/Plone was a bad idea instead). And yes, I do allow crawlers and they are poking at all kinds of things. However, I was still hoping that someone would address the specific examples I provided to tell me if they are in the ballpark of what is commonly observed. I understand all the generalities, but I want to find out if something is broken in my particular implementation (possibly because of a bad practice on my part, etc.). I realize that looking at requests only does not give the whole picture (although I think it should be the largest source of Zope activity by far, am I wrong?), and that the descriptions I have provided are far from sufficient to do a complete determination, but let's just take the first one: Memory size was at 417640 (Kb) from 03:01:40 to 03:04:41 (for three minutes). During these three minutes, there were requests for our front page and apparently everything that accompanies it in Plone, i.e. the Plone scripts.jss, the style sheets, bunch of icon gifs, image thumbs the logo.jpg, etc. No memory increase during that time. At 03:05:41 I logged a memory increase to 422704. Between 03:04:41 and 03:05:41, there was a single request at 03:05:02 for an image_view_fullscreen. For those unfamiliar with Plone, that is one of the barest page served: Just an image and a link to go back to Plone (none of the Plone portlets, etc.). The access log shows that the number of bytes returned was 1137. There was no GET for the actual image. There were no memory increase whatsoever for at least the next 30 minutes, and there were a number of other requests during that time. Now I'm looking for hypotheses to explain the 5Mb memory increase. Here are some: 1) It's impossible. I must have made a mistake when I created that merged log (i.e. I must be looking at bad data). 2) The 5 Mb necessarily had to be used by the object cache to load one or more objects. Note that the image was 2 Mb at most, and the object itself can't be much larger, so even if it was loaded in the cache then, my question is what else was loaded? Or, does Zope/Python grab more memory for the cache in chunks? (I know that when appending to a Python list for example, the list will grow in chunks, but that's only space for pointers and not the actual objects pointed to.) 3) Part of the 5 Mb was probably used loading into the object cache, but a significant part could have been used elsewhere (but the increase was still due to the request). So my question then is: Where else would previously unallocated megabytes of memory be needed to serve a page? I am using the default CacheFu behavior for images, so I know it wasn't also RAM cached. 4) Could have been a coincidence, unrelated to the request. My question then is: What else could Zope be doing in the background, other than serve requests, that consumes megabytes of new memory. (We don't use Webdav, or ftp; I'm quite positive in any case that no imports were being made. I do use external editor, but that would show up in the HTTP access log). Cheers, Jean _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )