RE: [JBoss-dev] Implementing a Resource Protection System?
The real issue with what you're asking is, though, that I don't know that there is a way to, within a Java runtime, break down memory usage at a fine-grained level to spot a memory hog. One possible tack to take would be to make each classloader keep tabs on how many times it gets requests for object creation, but even that doesn't get you what's needed, because I'm not aware of a mechanism in place to actually chart how much memory is getting consumed by the classes instantiated out of that classloader. As far as I see, the API does not support this. The memory reporting methods are for the entire JVM, and breaking down that aggregate information by subsystem in the JVM would be tricky, if not downright impossible. And yet you can run the JVM with a setting that reports that memory consumption per class in memory at the end of the run. It is the easiest way to find a leak. You are saying that trace is not available through an API? Even JDK1.4? But to answer the previous question, seriously why don't you run various VMs? One per client? Describe the hardware problem before you dive too deep in useless code, marcf -Original Message- From: Hunter Hillegas [mailto:[EMAIL PROTECTED]] Sent: Wednesday, July 10, 2002 4:12 PM To: JBoss Dev Subject: [JBoss-dev] Implementing a Resource Protection System? How hard would it be to implement some kind of resource protection system for deployed JBoss apps? I'm thinking mainly memory... Not allowing an app to continue to grab memory until the container dies, instead, shutting down that app and sending notification to an admin... Could be useful for a hosting environment... Possible? Cheers, Hunter --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Implementing a Resource Protection System?
I've thought about doing this in some of the other architectures I've written from time to time. It's possible to keep an eye on memory usage and track its stats over time, so you can know when memory's becoming scarce and start telling different parts of the system Memory's tight. Can you do what you can do to lighten the load? That wasn't all that hard to do- every time this architecture deployed a job to run, it kept a handle to them and would asynchronously call a method on them that contained best effort code to lighten up the load and call the GC. That works fine when you just know that you're using more and more memory and just want to politely ask deployed code to attempt to help out. Rhet, LOL, do you realize how dumb this is??? You are saying that in the hosted environment, when one module is a hog, the system asks politely for the other well behaved parts, to please let go of some memory, cause the hog over there wants some... Dude, you were sleeping when you wrote this. I'll sue you if you commit this to our tree :) Marcf PS: for your own application it was probably fine --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: PRIVATE:(RE: [JBoss-dev] Implementing a Resource Protection System?)
Rhett, I am sorry man, I didn't mean it that way. In fact automatic tracking of memory hogs would be very interesting for us, I could use it right now on our own production site. Please feel free to contribute, we need more guys like you, I am sorry for making fun of the idea, as I said it was probably right for the problem you were trying to solve. PLgC marcf -Original Message- From: Rhett Aultman [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 11, 2002 12:41 PM To: [EMAIL PROTECTED] Subject: PRIVATE:(RE: [JBoss-dev] Implementing a Resource Protection System?) Marc, I really do admire you, and as such I know about your brutal honesty and sharp sense of humor, but am I really deserving of that kind of treatment? This wasn't for a hosted environment for tracking inadvertant hogs. It was for an environment of scheduled jobs being written by me and everyone on my team that were simple but intense and that were tested before being left in the scheduled environment (personally, I didn't even want to write the damn thing...but people around here have never heard of cron and won't accpet a non-Java answer anyway). The only purpose was to alert jobs that they needed to lighten the load or the heap would run out. It was necessary because team members were writing jobs of heavy memory utilization that would sometimes be running concurrently with other memory intensive jobs. Yes, it worked there, and it worked because the people developing jobs for it knew that if one of their jobs used up the heap and didn't complete, their heads would roll the next day. Yes, it would be retarded in a hosted environment trying to spot hogs. No, I am not that stupid. I may not be you or Dain or Juha or any of the other crowd, but I am not stupid. I may be all of 22, fresh out of college, and only now teaching myself about OS kernel development, but I still am not stupid. What I was attempting to do was to explain to the original author all the further I'd been able to get on such an issue, which was to spot when memory's getting seriously tight. As of now, I still don't see a way to get anything more fine-grained than that, and I really don't see any APIs out there for doing it inside the VM (though the JNI option is interesting). I know I'm not a JBoss contributor, nor am I Marc Fleury, but I thought that, at the time, I might have had something meaningful to add by pointing out the shortcomings of what appears to be readily available right now and by pointing out all the further that I can see someone getting with those tools, which clearly isn't far enough. I believe in JBoss. I use JBoss. I stand behind JBoss. I even do what I can do to try and make JBoss better (although I'm often out of my league and in the presence of those who have far more expertise and experience than me). If my input is really that unwelcome, just say so, and you won't have to be subjected to my commentary any further -Original Message- From: marc fleury [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 11, 2002 12:07 PM To: [EMAIL PROTECTED] Subject: RE: [JBoss-dev] Implementing a Resource Protection System? I've thought about doing this in some of the other architectures I've written from time to time. It's possible to keep an eye on memory usage and track its stats over time, so you can know when memory's becoming scarce and start telling different parts of the system Memory's tight. Can you do what you can do to lighten the load? That wasn't all that hard to do- every time this architecture deployed a job to run, it kept a handle to them and would asynchronously call a method on them that contained best effort code to lighten up the load and call the GC. That works fine when you just know that you're using more and more memory and just want to politely ask deployed code to attempt to help out. Rhet, LOL, do you realize how dumb this is??? You are saying that in the hosted environment, when one module is a hog, the system asks politely for the other well behaved parts, to please let go of some memory, cause the hog over there wants some... Dude, you were sleeping when you wrote this. I'll sue you if you commit this to our tree :) Marcf PS: for your own application it was probably fine --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss
RE: [JBoss-dev] Implementing a Resource Protection System?
Yeah...I wasn't suggesting that for a hosted environment with hog detections anyway, though. What I was trying to do was point out the limitations I'm aware of with standard APIs that are available. The current limitation I'm aware of is that you can see when memory's getting tight, but you cannot see why (this is, of course, not including the JNI options mentioned here), so the best you can do is to politely ask components to release memory or you can just throw the baby out with the bathwater by undeploying all loaded components and then redeploying them. My original intent was not to suggest we take that route, but to instead show that the idea of a memory monitor wasn't really feasible to begin with. I'm now recognizing the poor communication of that idea on my behalf by not starting off with I don't think it'll work and instead commenting on what worked for me in an environment far removed from the one in question. -Original Message- From: marc fleury [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 11, 2002 1:53 PM To: Jboss-Development@Lists. Sourceforge. Net Subject: FW: [JBoss-dev] Implementing a Resource Protection System? Dain helps me say it nicely, Marcf -Original Message- From: Dain Sundstrom [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 11, 2002 12:29 PM To: marc fleury Subject: Re: [JBoss-dev] Implementing a Resource Protection System? Here is a more friendly response: This is a bad idea. You are saying that in the hosted environment, when one module is a hog, the system asks politely for the other well behaved parts, to please let go of some memory, cause the hog over there wants some. More importantly you are talking about a huge amount of effort when you can simply run multiple JBoss instances. If you compare the cost of say an extra 20 megs of memory per JBoss instance to the cost of developing and maintaining this code, you will agree with me and buy an extra 256 MB chip for $50. -dain marc fleury wrote: Dain, How can I say this nicely to you? marcf -Original Message- From: Dain Sundstrom [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 11, 2002 12:22 PM To: marc fleury Subject: Re: [JBoss-dev] Implementing a Resource Protection System? I thought we were not going to call people (or their ideas) dumb anymore. I agree that is is a dumb idea, but we should be nicer. -dain marc fleury wrote: I've thought about doing this in some of the other architectures I've written from time to time. It's possible to keep an eye on memory usage and track its stats over time, so you can know when memory's becoming scarce and start telling different parts of the system Memory's tight. Can you do what you can do to lighten the load? That wasn't all that hard to do- every time this architecture deployed a job to run, it kept a handle to them and would asynchronously call a method on them that contained best effort code to lighten up the load and call the GC. That works fine when you just know that you're using more and more memory and just want to politely ask deployed code to attempt to help out. Rhet, LOL, do you realize how dumb this is??? You are saying that in the hosted environment, when one module is a hog, the system asks politely for the other well behaved parts, to please let go of some memory, cause the hog over there wants some... Dude, you were sleeping when you wrote this. I'll sue you if you commit this to our tree :) Marcf PS: for your own application it was probably fine --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development -- Dain Sundstrom Chief Architect JBossCMP JBoss Group, LLC --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Implementing a Resource Protection System?
marc fleury wrote: How about a PhD sub-project, Mr Akkerman, that would be a nice mbean in our system. It would have a little bit of everything, Some C, some JNI, java classes and mbean, You broadcast the information for the VM, Trust me, you will look like a king, marcf Sorry to disappoint you, Marc, but this does not scratch my itch at the moment (in fact, not pursuing PhD anymore but rather working for the research group here means I actually scratch the itches of someone else). As far as looking like a king ... well, who can look like you :) On a more serious not, if this becomes relevant to our work, me or someone else might implement this. Indeed, user-level (meaning w/o running the monitor as 'root'-equivalent) sandboxing of memory, CPU and network I/O on Win32 platform has been implemented by one of our former PhD students, so this strikes home somewhat. But we are more interested in monitoring and sandboxing invocations in component-based applications as the near-future goal (after I am done with what I am doing right now). Again, not being a good programmer, is a horrible fate in CS world, so I am slow and your guys might get to invocation flow-profiling and sandboxing before I do. Actually, have anyone done it already? (I mean writing a client-side and server-side interceptors that propagate information about the invocation itself, like which host/component the invocation is coming from?) Anatoly. --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Implementing a Resource Protection System?
I'm going to have to learn JVMPI first and dust off my JNI boots, but I'd pitch in on something like this. Memory profiling, resource protection, etc. are all itches of mine. Oh...no, wait. That's poison ivy. Seriously, though, I'd be more than willing to work on something like this, although I'm sure going it alone would not be wise. If one of you more senior project members wants an assistant on something like this, I'll step up. -Original Message- From: Anatoly Akkerman [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 11, 2002 3:11 PM To: [EMAIL PROTECTED] Subject: Re: [JBoss-dev] Implementing a Resource Protection System? marc fleury wrote: How about a PhD sub-project, Mr Akkerman, that would be a nice mbean in our system. It would have a little bit of everything, Some C, some JNI, java classes and mbean, You broadcast the information for the VM, Trust me, you will look like a king, marcf Sorry to disappoint you, Marc, but this does not scratch my itch at the moment (in fact, not pursuing PhD anymore but rather working for the research group here means I actually scratch the itches of someone else). As far as looking like a king ... well, who can look like you :) On a more serious not, if this becomes relevant to our work, me or someone else might implement this. Indeed, user-level (meaning w/o running the monitor as 'root'-equivalent) sandboxing of memory, CPU and network I/O on Win32 platform has been implemented by one of our former PhD students, so this strikes home somewhat. But we are more interested in monitoring and sandboxing invocations in component-based applications as the near-future goal (after I am done with what I am doing right now). Again, not being a good programmer, is a horrible fate in CS world, so I am slow and your guys might get to invocation flow-profiling and sandboxing before I do. Actually, have anyone done it already? (I mean writing a client-side and server-side interceptors that propagate information about the invocation itself, like which host/component the invocation is coming from?) Anatoly. --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Implementing a Resource Protection System?
If Anatoly is not a good programmer, most of us are indeed wasting our time and should probably take up carpentry or flower arranging. Anatoly wrote: Again, not being a good programmer, is a horrible fate in CS world... --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Implementing a Resource Protection System?
If Anatoly is not a good programmer, most of us are indeed wasting our time and should probably take up carpentry or flower arranging. Hiram is taking up carpentry and he is a really good programmer, Anatoly is not a good/bad programmer, he's a russian programmer, marcf Anatoly wrote: Again, not being a good programmer, is a horrible fate in CS world... --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek PC Mods, Computing goodies, cases more http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Implementing a Resource Protection System?
I've thought about doing this in some of the other architectures I've written from time to time. It's possible to keep an eye on memory usage and track its stats over time, so you can know when memory's becoming scarce and start telling different parts of the system Memory's tight. Can you do what you can do to lighten the load? That wasn't all that hard to do- every time this architecture deployed a job to run, it kept a handle to them and would asynchronously call a method on them that contained best effort code to lighten up the load and call the GC. That works fine when you just know that you're using more and more memory and just want to politely ask deployed code to attempt to help out. The real issue with what you're asking is, though, that I don't know that there is a way to, within a Java runtime, break down memory usage at a fine-grained level to spot a memory hog. One possible tack to take would be to make each classloader keep tabs on how many times it gets requests for object creation, but even that doesn't get you what's needed, because I'm not aware of a mechanism in place to actually chart how much memory is getting consumed by the classes instantiated out of that classloader. As far as I see, the API does not support this. The memory reporting methods are for the entire JVM, and breaking down that aggregate information by subsystem in the JVM would be tricky, if not downright impossible. -Original Message- From: Hunter Hillegas [mailto:[EMAIL PROTECTED]] Sent: Wednesday, July 10, 2002 4:12 PM To: JBoss Dev Subject: [JBoss-dev] Implementing a Resource Protection System? How hard would it be to implement some kind of resource protection system for deployed JBoss apps? I'm thinking mainly memory... Not allowing an app to continue to grab memory until the container dies, instead, shutting down that app and sending notification to an admin... Could be useful for a hosting environment... Possible? Cheers, Hunter --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Implementing a Resource Protection System?
From what I know, one can use JVMPI hooks to do fine-grained memory monitoring (http://java.sun.com/j2se/1.3/docs/guide/jvmpi/jvmpi.html). But this would involve writing some C, JNI, helper Java classes to query your custom profiling library for collected info. Depending on how well you design your library and how many profiling events you end up collecting, this may substantially slow down the JVM (just imagine the fact that you would have to call into the library every time an object is allocated or deleted, yikes). Also, once you load your library, you cannot debug this jvm remotely or profile anything in it using the standard profiling tools. - Anatoly Akkerman Computer Science Dept. Courant Institute of Mathematical Sciences, NYU 715 Broadway, #719 Tel: 212 998-3493 New York, NY 10003 Fax: 212 995-4123 - Rhett Aultman wrote: I've thought about doing this in some of the other architectures I've written from time to time. It's possible to keep an eye on memory usage and track its stats over time, so you can know when memory's becoming scarce and start telling different parts of the system Memory's tight. Can you do what you can do to lighten the load? That wasn't all that hard to do- every time this architecture deployed a job to run, it kept a handle to them and would asynchronously call a method on them that contained best effort code to lighten up the load and call the GC. That works fine when you just know that you're using more and more memory and just want to politely ask deployed code to attempt to help out. The real issue with what you're asking is, though, that I don't know that there is a way to, within a Java runtime, break down memory usage at a fine-grained level to spot a memory hog. One possible tack to take would be to make each classloader keep tabs on how many times it gets requests for object creation, but even that doesn't get you what's needed, because I'm not aware of a mechanism in place to actually chart how much memory is getting consumed by the classes instantiated out of that classloader. As far as I see, the API does not support this. The memory reporting methods are for the entire JVM, and breaking down that aggregate information by subsystem in the JVM would be tricky, if not downright impossible. -Original Message- From: Hunter Hillegas [mailto:[EMAIL PROTECTED]] Sent: Wednesday, July 10, 2002 4:12 PM To: JBoss Dev Subject: [JBoss-dev] Implementing a Resource Protection System? How hard would it be to implement some kind of resource protection system for deployed JBoss apps? I'm thinking mainly memory... Not allowing an app to continue to grab memory until the container dies, instead, shutting down that app and sending notification to an admin... Could be useful for a hosting environment... Possible? Cheers, Hunter --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Implementing a Resource Protection System?
Interesting...I'll have to check that out. Honestly, though, it doesn't sound like much of a solution for something like JBoss or even most infrasructural items. I figured any fine-grained memory monitoring would require native code. -Original Message- From: Anatoly Akkerman [mailto:[EMAIL PROTECTED]] Sent: Wed 7/10/2002 5:02 PM To: [EMAIL PROTECTED] Cc: Subject: Re: [JBoss-dev] Implementing a Resource Protection System? From what I know, one can use JVMPI hooks to do fine-grained memory monitoring (http://java.sun.com/j2se/1.3/docs/guide/jvmpi/jvmpi.html). But this would involve writing some C, JNI, helper Java classes to query your custom profiling library for collected info. Depending on how well you design your library and how many profiling events you end up collecting, this may substantially slow down the JVM (just imagine the fact that you would have to call into the library every time an object is allocated or deleted, yikes). Also, once you load your library, you cannot debug this jvm remotely or profile anything in it using the standard profiling tools. - Anatoly Akkerman Computer Science Dept. Courant Institute of Mathematical Sciences, NYU 715 Broadway, #719 Tel: 212 998-3493 New York, NY 10003 Fax: 212 995-4123 - Rhett Aultman wrote: I've thought about doing this in some of the other architectures I've written from time to time. It's possible to keep an eye on memory usage and track its stats over time, so you can know when memory's becoming scarce and start telling different parts of the system Memory's tight. Can you do what you can do to lighten the load? That wasn't all that hard to do- every time this architecture deployed a job to run, it kept a handle to them and would asynchronously call a method on them that contained best effort code to lighten up the load and call the GC. That works fine when you just know that you're using more and more memory and just want to politely ask deployed code to attempt to help out. The real issue with what you're asking is, though, that I don't know that there is a way to, within a Java runtime, break down memory usage at a fine-grained level to spot a memory hog. One possible tack to take would be to make each classloader keep tabs on how many times it gets requests for object creation, but even that doesn't get you what's needed, because I'm not aware of a mechanism in place to actually chart how much memory is getting consumed by the classes instantiated out of that classloader. As far as I see, the API does not support this. The memory reporting methods are for the entire JVM, and breaking down that aggregate information by subsystem in the JVM would be tricky, if not downright impossible. -Original Message- From: Hunter Hillegas [mailto:[EMAIL PROTECTED]] Sent: Wednesday, July 10, 2002 4:12 PM To: JBoss Dev Subject: [JBoss-dev] Implementing a Resource Protection System? How hard would it be to implement some kind of resource protection system for deployed JBoss apps? I'm thinking mainly memory... Not allowing an app to continue to grab memory until the container dies, instead, shutting down that app and sending notification to an admin... Could be useful for a hosting environment... Possible? Cheers, Hunter --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http://thinkgeek.com/sf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development --- This sf.net email is sponsored by:ThinkGeek Two, two, TWO treats in one. http
