Hi, On Tue, Dec 21, 2010 at 02:03:17PM -0800, Vincent wrote: > I think the problem is when varnish uses more than 4GB virtual memory on > 32-bit, it will restart. I started varnish with "-s malloc,1.5G". Varnish > will gradually increase memory usage and stop at about 1.7G. However, the > virtual memory size is growing continuously and it doesn't seem to stop. We > restarted varnish yesterday and currently the VM size is 2.7G, and it is > still growing. Is this normal?
The virtual memory usage depends on a large number of factors, and when you are limited to 3GB of it (assuming a user/kernel split at 3G/1G) trying to determine if there's a leak or not is just not worth the effort. Varnish is a massively parallel application, each thread has an overhead, each session has an overhead, each VCL file has an overhead, each object cached has an overhead. And so forth. These components were all designed for a 64-bit architecture where address space is a non-issue. Over the years, we've monitored real memory usage to determine memory leaks and changes in memory overhead. This effort was, for example, the reason obj_workspace was removed. Every reported memory leak I've investigated to date has turned out to be a wild goose chase. It's taken us quite some time to determine a reasonable estimate for _real_ memory overhead on 64-bit as it changed from Varnish 2.0 to 2.1, and I don't think you'll see us try to do the same thing for address space just because it would be a funny little thing to have on a platform nobody really need. If it is of interest, the per-object overhead we are seeing is currently roughly 1kB for each object - but that's resident memory, not virtual. With a million objects, that means 1GB of overhead just for the objects - I wouldn't be surprised if it was twice that in virtual memory. If you really really need 32-bit for some silly application, spend 100$ buying a dedicated 64-bit-box for Varnish. Because Varnish really really needs 64-bit. You'll save yourself a great deal of pain and you'll be running on the platform Varnish was designed for. > Yes, I know we should have used 32-bit in the first place. The reason is > that we have some other applications work on 32-bit only. Anyway, we will > build a 64-bit box dedicated for Varnish. However, isn't it nice to have > varnish run without any problems on 32-bit machines? No, we don't care much about 32-bit. Use 64-bit. If you got patches for 32-bit, we'll take a look, but trying to fix a legacy-platform isn't time well spent. If you fix it, we can check your conclusions, but that's about it. And in this case, it doesn't look like there's anything to fix. If I were you, I'd investigate the "32-bit only" application holding you back, not the 64-bit application actually using your hardware to its potential. You are trying to tune your Ferrari for a race track in your back yard: it'll never work well in that scenario and you'll never be happy with it like that. I usually call 32-bit platforms "Nintendo platforms" in the context of Varnish (and compare it to Varnish running on an iPhone, for example): It's fun if it works, but we wont spend any considerably time tuning it for the platform. But that's not really a fair thing to say - after all, the Nintendo 64 came out in 1996. - Kristian _______________________________________________ varnish-misc mailing list [email protected] http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
