Ok, I spent day experimenting with our P2 bug, that is very specific of OOM: it has a "peak" of memory consumption that is immediately reclaimed too.
In short, I tried following: * UncaughtExceptionHandler installed with my own "instrumented QTP". Did not have quite much effect, to me, seems like it was not even invoked in case of OOM. * wrapping Jetty's _runnable (from QTP) with my own and logging from there... same thing, like my code was not invoked at all. But, as I realized, _other threads_ in system kept living happily during OOM, it was only the failing thread that was heavily disrupted. I assume this is due to "type" or "case" of the OOM I have. Again, this was not a leak, this was one processing step allocating way too much and immediately releasing the same stuff. I assume this would not help for real leaks, as whole JVM would be disrupted (just to emphasize: my app was around 150MB of heap while "working/under load", one HTTP call causes to have heap jump to 850-900MB, but the very next GC also takes it back to 150-200MB). So, I went will 3rd approach that proved working for me: * a separated "watchdog" daemon thread, that watches for QTP thread deaths, and start you nagging, and does it endlessly once death detected. https://github.com/sonatype/sisu-jetty8/commit/0cf9572896d4f0cf77bbaa57c41359db86cf9586 I think this was my goal, to have _any_ feedback in case of OOMs, as threads were just disappearing, and users had NO whatsoever feedback, that they might running the system "on the edge". I'd really like to see some of these (if only the "instrumented QTP" and ability to pass in any ThreadFactory!) in Jetty! Thanks, ~t~ On Tue, Jun 26, 2012 at 7:14 PM, Tamás Cservenák <[email protected]>wrote: > I totally agree with you and rational. > > But still, my question stands: what is the simplest or intended way > (asking, since you guys might have something similar in Jetty already), to > have some UncaughtExceptionHandler (or maybe some Jetty specific class) > installed in Jetty threads (when running embedded)? > > Point is, that _nothing_ got logged after OOM, nothing is visible "from > outside", and my Nexus continued to live and happily served (!) things > after the incident. I have to point out that OOM is not due to a leak, but > one request in buggy code causes it, and in a moment it died and thread > removed, JVM got plenty of memory... > > So, to be able to emit at least a big bold warning, to try to shut down > Jetty, _something_ ... , as leaving OOM unnoticed -- even if I agree with > rationale -- is not okay for me in this case... > > Do I have access to ThreadGroup? How can I instrument threads and/or > thread creation in Jetty? > > Thanks, > ~t~ > > > On Tue, Jun 26, 2012 at 6:09 PM, Jesse McConnell < > [email protected]> wrote: > >> well, after an OOME all bets are generally off and your jvm is going >> to have to go down to be predictable again...I suspect that is part of >> the rational >> >> jesse >> >> -- >> jesse mcconnell >> [email protected] >> >>
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
