Without some information about what kind of traffic you are trying to
handle it is very tough to make recommendations. I have experience with
our traffic for blogs.sun.com, but if you are getting more traffic then
us then any advice I can give is obviously less proven.
IMHO you need more RAM for caching and possibly a full hardware refresh
including a second machine. You said yourself that the heap fills up
with Strings, likely related to velocity and hence the rendering system.
If this is the case then you need more RAM both for rendering and for
caching. We can run BSC fine on a single machine (SunFire T2000) even
when restarting everything from scratch (we actually tested this
yesterday :( ), but we have a 3.6G jvm heap and 6G of additional memory
allocated to memcached for caching, so that could be the differentiating
factor here.
One other thing to note which may be part of your memory problems. The
rendering system in Roller is designed to cache whole pages, and the
only way to do that is to buffer the entire page while rendering
happens. This means that if you get a lot of traffic causing rendering
at the same time then your heap can fill up with buffered output content
(both pages & feeds). So again, the reality here is that if you are
running a large site then you need more than 1.5G of RAM, period. I
would be the first person to say that it's unrealistic to run a Roller
site with 10K blogs on a single machine with only a 1.5G heap.
If you want you can completely disable the rendering caches by setting
these properties in your config, although you're likely to run your cpu
a bit crazy and still run out of memory ...
cache.sitewide.enabled=false
cache.weblogpage.enabled=false
cache.weblogfeed.enabled=false
-- Allen
Matthew Schmidt wrote:
We have heap dumps both before and after it goes sour. The primarily
point to a large increase in strings, with many references to velocity
tokens. We didn't really notice any significant change in the traffic
when it happened. It happens so fast that the system is quickly
non-responsive.
-Matt
Anil Gangolli wrote:
Can you get a heap image at a point when things have gone sour?
Do you have async referrer processing enabled?
If you are capturing access logs, can you see if there is an increase in
any bot/crawler activity at the times when the heap grows?
--a.
Matthew Schmidt wrote:
Hi guys. Since our upgrade to Roller 3.1, things have not been good
in the land of JRoller. The system will occasionally run for a few
hours and then suddenly add nearly a gig of objects to the heap, all
of which are instanly pushed to the old gen space and cannot be
garbage collected. Restarting the application server does not
correct this problem, as it often comes right back immediately. We
can see nothing in the access logs that are different from when its
running fine and then its suddenly not. The search is currenly
disabled, so that's not causing it. The referer processing is also
disabled and our caches in Roller are not too large. We've upgraded
both Velocity and Hibernate to their latest released versions and
that had no effect. Looking into the heap dump, it seems that most
of the space is in Strings, probably related to Velocity. We've
disabled velocity caching completely right now so see if that is the
cause.
Has anyone else seen anything like this? We're running in a 1.5G
heap and it just runs normally for a while then flakes out and then
won't even operate normally after a restart for a while. Eventually,
enough restarts get is back into regular operation. I'd appreciate
any help you guys can give as without something, JRoller will likely
plunge even further into dispair.
Thanks,
Matt
