First of all, I wonder why you need to "init 6" the VM every morning? If it is "needed" due to performance degradations, this may be your main clue, and the causes got worse over time (i.e. some memory leaks, etc.)?
An OS-related idea is that some resource becomes locked up and its requesters pile up and bring the system down with starvation. This can be remedied to an extent with locking, so that only one instance of a "requester" runs (i.e. not more than one backup is done at a time, if one takes longer to complete than your estimated timeframe which you enforced with crontabs). I also have some non-illumos ideas :) You can look for "JVMStat/VisualGC" (may be part of JDK now, I believe) - a GUI program that nicely displays memory layouts of a JVM process, interpreting about the same data that you can receive via "kill -3", I think. You can see if garbage collections are occurring, and if they help (i.e. in case of leaks, you ultimately run out of expungable memory and need the interim JVM restarts, indeed - but better rewrite the code if possible). One way that an appserver leaks often is the dynamic reload of web applications (old instances remain referenced and are not expiring from heap). For Apache Tomcat I also found Lambda Probe quite useful. It is a web application (rather a servlet) which allows you to trace what your tomcat is doing, which pages take long to complete - or even hang, etc. It has helped us weed out a few coding errors that ultimately brought websites down; in one case - by launching thousands of copies of an unrenderable page while a site was crawled by honest search robots ;) Finally, you can bump up logging of GC, heap dumps on JVM deaths, etc. and try to debug with that data. HTH, //Jim ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
