Re: [DISCUSS] Improving diagnostics
Um... sure... but now right now... too many beers. :-P But JCL or SLF4J, we still need some serious logging reform... --jason On Jun 15, 2007, at 6:36 AM, Jacek Laskowski wrote: On 6/15/07, Jason Dillon <[EMAIL PROTECTED]> wrote: And, I think we should seriously consider using SLF4J ( http:// www.slf4j.org/ ) and drop JCL. I've never used SLF4J before, but it's again I could hear about it from you (I'm pretty sure it was you who tried to persuade us to use it). Could you tell a bit more why slf4j is better than jcl. Why is it worth to move to slf4j - I guess it'd mean a lot of changes in the code, wouldn't it? Jacek -- Jacek Laskowski http://www.JacekLaskowski.pl
Re: [DISCUSS] Improving diagnostics
On 6/15/07, Jason Dillon <[EMAIL PROTECTED]> wrote: And, I think we should seriously consider using SLF4J ( http:// www.slf4j.org/ ) and drop JCL. I've never used SLF4J before, but it's again I could hear about it from you (I'm pretty sure it was you who tried to persuade us to use it). Could you tell a bit more why slf4j is better than jcl. Why is it worth to move to slf4j - I guess it'd mean a lot of changes in the code, wouldn't it? Jacek -- Jacek Laskowski http://www.JacekLaskowski.pl
Re: [DISCUSS] Improving diagnostics
On Jun 15, 2007, at 3:24 AM, Jason Dillon wrote: I think that in general we need to go through some serious logging reform. IMO logging is critical and right now I think the state of logging in G is a mess. I would like to see more effort put into making logging more consistent and provide better user tools to control the servers logging as well as application logging. I totally agree. Was thinking the same thing last night... IMO, fixing this problem goes a long way towards addressing Matt's concerns... In most cases, it's *lack* of information that inhibits diagnosis, not how the information is presented... Some general architecture/standardization is needed on logging format (including Thread ids, etc -- start to address some of Matt's concerns). Followed by a concerted effort to redo/introduce logging into the majority of Geronimo's components. And, I think we should seriously consider using SLF4J ( http:// www.slf4j.org/ ) and drop JCL. Can't say that I'm super motivated to move off of JCL, but I'm not averse to it, either... --kevan
Re: [DISCUSS] Improving diagnostics
I think that in general we need to go through some serious logging reform. IMO logging is critical and right now I think the state of logging in G is a mess. I would like to see more effort put into making logging more consistent and provide better user tools to control the servers logging as well as application logging. And, I think we should seriously consider using SLF4J ( http:// www.slf4j.org/ ) and drop JCL. --jason On Jun 14, 2007, at 6:08 AM, Matt Hogstrom wrote: Lately I've been working with users in debugging various application problems. Some of the problems are merely configuration but others are deeper application / infrastructure problems. Regardless of the type of problem I've never personally been satisfied with the diagnostic information produced by the server (this isn't a Geronimo statement but really AppServers as a whole including WebSphere and WebLogic). Here are some thoughts that I want to pursue: I’ve been working with some customers lately and the work has centered around debugging some of the aspects of their server. In this case it was using Apache Geronimo but the problem really applies to most application servers in general. For the most part there is little diagnostic information available when an application fails. We get the ever popular nested Java Stack trace which is certainly a good indicator of where a failure occurred but is woefully inadequate in many instances of why a failure occurred. This get’s worse in that for the most part people need to recreate the problem with additional tracing and, in the worst case, additional diagnostic code in their application. Wouldn’t it be nice if some of the diagnostic capability that was needed was included in the server itself? Over the next few weeks I’m going to be doing some experimentation on how to improve server diagnostics through the use of Aspects and/ or Instrumentation. Since this is experimental we’ll see what the final result will be. Here are some of my initial goals: 1. Improve diagnostics by providing a Diagnostic Report when a Tx fails. 2. Provide better visualization of Java Stack traces so problem areas pop out. 3. Capture wait information For number 1 I’m going to focus on servlets to begin with given that they represent the preponderance of requests made in AppServers today. This information will include information from the request object, the servlet being invoked, invocation time, transaction ID (if it exists), enlisted connections (database and messaging), oh yeah, and the Thread ID of execution. This is a mouthful to begin with anyway. Number 2 is really just applying some template information on a Java Stack trace. I want application classes to standout so developers will be able to quickly see where their application is involved. Infrastructure pieces like the server, Hibernate, TopLink, etc. would also be highlighted in a different color and style and plain old java classes would be a boring style as their are merely pawns in the transactional game. Finally, wouldn’t it be nice to know how long a thread has been waiting and for what reason? Is it waiting on a request from another server or perhaps there is a database locking problem. Did a WebService go awry? Basically, I want to know what in the heck a thread is waiting on. Please chime in with your thoughts. I spect that Aspects or Instrumentation may be the only way to go for some of this as many of the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
Is there anyway we could intercept when exceptions are created? If we could then one possibility to improve diagnostics would be to keep a count of what exceptions are being thrown and from where. With that information, we could perhaps give an administrator hints on where trouble spots are in their applications. Or, we could automatically escalate the level of logging/information provided in areas that throw lots of exceptions (subject to some user configured threshold). And regarding XML serialization - Does anyone know where the XML serialization work was done or who did it (if there really was any work on this)? I am trying to rework some of the console screens to get better browser compatibility and sending information back and forth in XML will prevent some javascript security problems. If someone has already figured out the schema to describe GBeans then that would make my work on the console easier too. Jay Paul McMahan wrote: I like your ideas! Thanks for bringing up this important topic. One of my pet peeves in debugging geronimo apps is when I'm trying to step through a gbean call but the debugger doesn't have the source available since cglib has dynamically generated the proxy classes. I usually end up having to do cheesy things like setting breakpoints on both sides of the proxy, which only works if I can figure out what's on the other side of the proxy. Seems like I remember some talk a while ago about an experimental runtime mode which disables proxying. I don't think you would want to normally run in that mode but it would be useful for debugging purposes. I don't remember where that ended up. One other thing that I think would be very helpful for debug and diagnostics would be to serialize gbean info to xml rather than a serialized java object. Again, I think there was some experimental work on this a few months ago but where did it end up? Maybe these improvements are already and I just didn't know about it :-) Best wishes, Paul On Jun 14, 2007, at 9:08 AM, Matt Hogstrom wrote: Lately I've been working with users in debugging various application problems. Some of the problems are merely configuration but others are deeper application / infrastructure problems. Regardless of the type of problem I've never personally been satisfied with the diagnostic information produced by the server (this isn't a Geronimo statement but really AppServers as a whole including WebSphere and WebLogic). Here are some thoughts that I want to pursue: I’ve been working with some customers lately and the work has centered around debugging some of the aspects of their server. In this case it was using Apache Geronimo but the problem really applies to most application servers in general. For the most part there is little diagnostic information available when an application fails. We get the ever popular nested Java Stack trace which is certainly a good indicator of where a failure occurred but is woefully inadequate in many instances of why a failure occurred. This get’s worse in that for the most part people need to recreate the problem with additional tracing and, in the worst case, additional diagnostic code in their application. Wouldn’t it be nice if some of the diagnostic capability that was needed was included in the server itself? Over the next few weeks I’m going to be doing some experimentation on how to improve server diagnostics through the use of Aspects and/or Instrumentation. Since this is experimental we’ll see what the final result will be. Here are some of my initial goals: 1. Improve diagnostics by providing a Diagnostic Report when a Tx fails. 2. Provide better visualization of Java Stack traces so problem areas pop out. 3. Capture wait information For number 1 I’m going to focus on servlets to begin with given that they represent the preponderance of requests made in AppServers today. This information will include information from the request object, the servlet being invoked, invocation time, transaction ID (if it exists), enlisted connections (database and messaging), oh yeah, and the Thread ID of execution. This is a mouthful to begin with anyway. Number 2 is really just applying some template information on a Java Stack trace. I want application classes to standout so developers will be able to quickly see where their application is involved. Infrastructure pieces like the server, Hibernate, TopLink, etc. would also be highlighted in a different color and style and plain old java classes would be a boring style as their are merely pawns in the transactional game. Finally, wouldn’t it be nice to know how long a thread has been waiting and for what reason? Is it waiting on a request from another server or perhaps there is a database locking problem. Did a WebService go awry? Basically, I want to know what in the heck a thread is waiting on. Please chime in with
Re: [DISCUSS] Improving diagnostics
On Jun 14, 2007, at 10:51 AM, Paul McMahan wrote: I like your ideas! Thanks for bringing up this important topic. One of my pet peeves in debugging geronimo apps is when I'm trying to step through a gbean call but the debugger doesn't have the source available since cglib has dynamically generated the proxy classes. I usually end up having to do cheesy things like setting breakpoints on both sides of the proxy, which only works if I can figure out what's on the other side of the proxy. Seems like I remember some talk a while ago about an experimental runtime mode which disables proxying. I don't think you would want to normally run in that mode but it would be useful for debugging purposes. I don't remember where that ended up. Start the server with the additional option: -DXorg.apache.geronimo.gbean.NoProxy=true If you have time I think the only reason that inhibits us from running the server in this mode is that there are a few Portlets that depend on Proxies. It would be excellent to remove that dependency.
Re: [DISCUSS] Improving diagnostics
I saw the work you did in OEJB. This would be a HUGE help as well. On Jun 14, 2007, at 9:44 AM, Rick McGuire wrote: One thing that might be a useful first start would be something similar to the "Hungry Exception" cleanup effort that was done with openejb3. Basically, this just ensured that any place where an exception gets thrown because of another caught exception that the original exception was maintained as a cause. I've spend a lot of time grumbling about situations where the original exception information was thrown away, sometimes through multiple levels of failure. Rick Matt Hogstrom wrote: Lately I've been working with users in debugging various application problems. Some of the problems are merely configuration but others are deeper application / infrastructure problems. Regardless of the type of problem I've never personally been satisfied with the diagnostic information produced by the server (this isn't a Geronimo statement but really AppServers as a whole including WebSphere and WebLogic). Here are some thoughts that I want to pursue: I’ve been working with some customers lately and the work has centered around debugging some of the aspects of their server. In this case it was using Apache Geronimo but the problem really applies to most application servers in general. For the most part there is little diagnostic information available when an application fails. We get the ever popular nested Java Stack trace which is certainly a good indicator of where a failure occurred but is woefully inadequate in many instances of why a failure occurred. This get’s worse in that for the most part people need to recreate the problem with additional tracing and, in the worst case, additional diagnostic code in their application. Wouldn’t it be nice if some of the diagnostic capability that was needed was included in the server itself? Over the next few weeks I’m going to be doing some experimentation on how to improve server diagnostics through the use of Aspects and/or Instrumentation. Since this is experimental we’ll see what the final result will be. Here are some of my initial goals: 1. Improve diagnostics by providing a Diagnostic Report when a Tx fails. 2. Provide better visualization of Java Stack traces so problem areas pop out. 3. Capture wait information For number 1 I’m going to focus on servlets to begin with given that they represent the preponderance of requests made in AppServers today. This information will include information from the request object, the servlet being invoked, invocation time, transaction ID (if it exists), enlisted connections (database and messaging), oh yeah, and the Thread ID of execution. This is a mouthful to begin with anyway. Number 2 is really just applying some template information on a Java Stack trace. I want application classes to standout so developers will be able to quickly see where their application is involved. Infrastructure pieces like the server, Hibernate, TopLink, etc. would also be highlighted in a different color and style and plain old java classes would be a boring style as their are merely pawns in the transactional game. Finally, wouldn’t it be nice to know how long a thread has been waiting and for what reason? Is it waiting on a request from another server or perhaps there is a database locking problem. Did a WebService go awry? Basically, I want to know what in the heck a thread is waiting on. Please chime in with your thoughts. I spect that Aspects or Instrumentation may be the only way to go for some of this as many of the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
Paul, I believe there is a setting you can use to turn off the proxies and not have to deal with CGLib...perfect for debugging (I think Dain pu t it in). Jeff Paul McMahan wrote: > I like your ideas! Thanks for bringing up this important topic. One of > my pet peeves in debugging geronimo apps is when I'm trying to step > through a gbean call but the debugger doesn't have the source available > since cglib has dynamically generated the proxy classes. I usually end > up having to do cheesy things like setting breakpoints on both sides of > the proxy, which only works if I can figure out what's on the other side > of the proxy. Seems like I remember some talk a while ago about an > experimental runtime mode which disables proxying. I don't think you > would want to normally run in that mode but it would be useful for > debugging purposes. I don't remember where that ended up. One other > thing that I think would be very helpful for debug and diagnostics would > be to serialize gbean info to xml rather than a serialized java > object. Again, I think there was some experimental work on this a few > months ago but where did it end up? Maybe these improvements are > already and I just didn't know about it :-) > > Best wishes, > Paul > > On Jun 14, 2007, at 9:08 AM, Matt Hogstrom wrote: > >> Lately I've been working with users in debugging various application >> problems. Some of the problems are merely configuration but others >> are deeper application / infrastructure problems. Regardless of the >> type of problem I've never personally been satisfied with the >> diagnostic information produced by the server (this isn't a Geronimo >> statement but really AppServers as a whole including WebSphere and >> WebLogic). >> >> Here are some thoughts that I want to pursue: >> >> I’ve been working with some customers lately and the work has centered >> around debugging some of the aspects of their server. In this case it >> was using Apache Geronimo but the problem really applies to most >> application servers in general. For the most part there is little >> diagnostic information available when an application fails. We get >> the ever popular nested Java Stack trace which is certainly a good >> indicator of where a failure occurred but is woefully inadequate in >> many instances of why a failure occurred. This get’s worse in that >> for the most part people need to recreate the problem with additional >> tracing and, in the worst case, additional diagnostic code in their >> application. Wouldn’t it be nice if some of the diagnostic capability >> that was needed was included in the server itself? >> >> Over the next few weeks I’m going to be doing some experimentation on >> how to improve server diagnostics through the use of Aspects and/or >> Instrumentation. Since this is experimental we’ll see what the final >> result will be. Here are some of my initial goals: >> >>1. Improve diagnostics by providing a Diagnostic Report when a Tx >> fails. >>2. Provide better visualization of Java Stack traces so problem >> areas pop out. >>3. Capture wait information >> >> For number 1 I’m going to focus on servlets to begin with given that >> they represent the preponderance of requests made in AppServers >> today. This information will include information from the request >> object, the servlet being invoked, invocation time, transaction ID (if >> it exists), enlisted connections (database and messaging), oh yeah, >> and the Thread ID of execution. This is a mouthful to begin with anyway. >> >> Number 2 is really just applying some template information on a Java >> Stack trace. I want application classes to standout so developers >> will be able to quickly see where their application is involved. >> Infrastructure pieces like the server, Hibernate, TopLink, etc. would >> also be highlighted in a different color and style and plain old java >> classes would be a boring style as their are merely pawns in the >> transactional game. >> >> Finally, wouldn’t it be nice to know how long a thread has been >> waiting and for what reason? Is it waiting on a request from another >> server or perhaps there is a database locking problem. Did a >> WebService go awry? Basically, I want to know what in the heck a >> thread is waiting on. >> >> Please chime in with your thoughts. I spect that Aspects or >> Instrumentation may be the only way to go for some of this as many of >> the components we include won't have this capability in them. >
Re: [DISCUSS] Improving diagnostics
I like your ideas! Thanks for bringing up this important topic. One of my pet peeves in debugging geronimo apps is when I'm trying to step through a gbean call but the debugger doesn't have the source available since cglib has dynamically generated the proxy classes. I usually end up having to do cheesy things like setting breakpoints on both sides of the proxy, which only works if I can figure out what's on the other side of the proxy. Seems like I remember some talk a while ago about an experimental runtime mode which disables proxying. I don't think you would want to normally run in that mode but it would be useful for debugging purposes. I don't remember where that ended up. One other thing that I think would be very helpful for debug and diagnostics would be to serialize gbean info to xml rather than a serialized java object. Again, I think there was some experimental work on this a few months ago but where did it end up? Maybe these improvements are already and I just didn't know about it :-) Best wishes, Paul On Jun 14, 2007, at 9:08 AM, Matt Hogstrom wrote: Lately I've been working with users in debugging various application problems. Some of the problems are merely configuration but others are deeper application / infrastructure problems. Regardless of the type of problem I've never personally been satisfied with the diagnostic information produced by the server (this isn't a Geronimo statement but really AppServers as a whole including WebSphere and WebLogic). Here are some thoughts that I want to pursue: I’ve been working with some customers lately and the work has centered around debugging some of the aspects of their server. In this case it was using Apache Geronimo but the problem really applies to most application servers in general. For the most part there is little diagnostic information available when an application fails. We get the ever popular nested Java Stack trace which is certainly a good indicator of where a failure occurred but is woefully inadequate in many instances of why a failure occurred. This get’s worse in that for the most part people need to recreate the problem with additional tracing and, in the worst case, additional diagnostic code in their application. Wouldn’t it be nice if some of the diagnostic capability that was needed was included in the server itself? Over the next few weeks I’m going to be doing some experimentation on how to improve server diagnostics through the use of Aspects and/ or Instrumentation. Since this is experimental we’ll see what the final result will be. Here are some of my initial goals: 1. Improve diagnostics by providing a Diagnostic Report when a Tx fails. 2. Provide better visualization of Java Stack traces so problem areas pop out. 3. Capture wait information For number 1 I’m going to focus on servlets to begin with given that they represent the preponderance of requests made in AppServers today. This information will include information from the request object, the servlet being invoked, invocation time, transaction ID (if it exists), enlisted connections (database and messaging), oh yeah, and the Thread ID of execution. This is a mouthful to begin with anyway. Number 2 is really just applying some template information on a Java Stack trace. I want application classes to standout so developers will be able to quickly see where their application is involved. Infrastructure pieces like the server, Hibernate, TopLink, etc. would also be highlighted in a different color and style and plain old java classes would be a boring style as their are merely pawns in the transactional game. Finally, wouldn’t it be nice to know how long a thread has been waiting and for what reason? Is it waiting on a request from another server or perhaps there is a database locking problem. Did a WebService go awry? Basically, I want to know what in the heck a thread is waiting on. Please chime in with your thoughts. I spect that Aspects or Instrumentation may be the only way to go for some of this as many of the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
I like all your ideas!One thing I have been spending some time is to how to turn on different levels of logs at different projects (inside of core geronimo) or other geronimo dependent logs. It would be very nice if the console can provide that level of details so that a user can just turn on the logging for the few components they need. Even better, the system is smart to figure out which components might be interesting for the specific error, and turn on the logs for the user after the user requests it via a one click (instead of multiple clicks after figuring out which components they need to turn on). Lin Matt Hogstrom wrote: Lately I've been working with users in debugging various application problems. Some of the problems are merely configuration but others are deeper application / infrastructure problems. Regardless of the type of problem I've never personally been satisfied with the diagnostic information produced by the server (this isn't a Geronimo statement but really AppServers as a whole including WebSphere and WebLogic). Here are some thoughts that I want to pursue: I’ve been working with some customers lately and the work has centered around debugging some of the aspects of their server. In this case it was using Apache Geronimo but the problem really applies to most application servers in general. For the most part there is little diagnostic information available when an application fails. We get the ever popular nested Java Stack trace which is certainly a good indicator of where a failure occurred but is woefully inadequate in many instances of why a failure occurred. This get’s worse in that for the most part people need to recreate the problem with additional tracing and, in the worst case, additional diagnostic code in their application. Wouldn’t it be nice if some of the diagnostic capability that was needed was included in the server itself? Over the next few weeks I’m going to be doing some experimentation on how to improve server diagnostics through the use of Aspects and/or Instrumentation. Since this is experimental we’ll see what the final result will be. Here are some of my initial goals: 1. Improve diagnostics by providing a Diagnostic Report when a Tx fails. 2. Provide better visualization of Java Stack traces so problem areas pop out. 3. Capture wait information For number 1 I’m going to focus on servlets to begin with given that they represent the preponderance of requests made in AppServers today. This information will include information from the request object, the servlet being invoked, invocation time, transaction ID (if it exists), enlisted connections (database and messaging), oh yeah, and the Thread ID of execution. This is a mouthful to begin with anyway. Number 2 is really just applying some template information on a Java Stack trace. I want application classes to standout so developers will be able to quickly see where their application is involved. Infrastructure pieces like the server, Hibernate, TopLink, etc. would also be highlighted in a different color and style and plain old java classes would be a boring style as their are merely pawns in the transactional game. Finally, wouldn’t it be nice to know how long a thread has been waiting and for what reason? Is it waiting on a request from another server or perhaps there is a database locking problem. Did a WebService go awry? Basically, I want to know what in the heck a thread is waiting on. Please chime in with your thoughts. I spect that Aspects or Instrumentation may be the only way to go for some of this as many of the components we include won't have this capability in them.
Re: [DISCUSS] Improving diagnostics
One thing that might be a useful first start would be something similar to the "Hungry Exception" cleanup effort that was done with openejb3. Basically, this just ensured that any place where an exception gets thrown because of another caught exception that the original exception was maintained as a cause. I've spend a lot of time grumbling about situations where the original exception information was thrown away, sometimes through multiple levels of failure. Rick Matt Hogstrom wrote: Lately I've been working with users in debugging various application problems. Some of the problems are merely configuration but others are deeper application / infrastructure problems. Regardless of the type of problem I've never personally been satisfied with the diagnostic information produced by the server (this isn't a Geronimo statement but really AppServers as a whole including WebSphere and WebLogic). Here are some thoughts that I want to pursue: I’ve been working with some customers lately and the work has centered around debugging some of the aspects of their server. In this case it was using Apache Geronimo but the problem really applies to most application servers in general. For the most part there is little diagnostic information available when an application fails. We get the ever popular nested Java Stack trace which is certainly a good indicator of where a failure occurred but is woefully inadequate in many instances of why a failure occurred. This get’s worse in that for the most part people need to recreate the problem with additional tracing and, in the worst case, additional diagnostic code in their application. Wouldn’t it be nice if some of the diagnostic capability that was needed was included in the server itself? Over the next few weeks I’m going to be doing some experimentation on how to improve server diagnostics through the use of Aspects and/or Instrumentation. Since this is experimental we’ll see what the final result will be. Here are some of my initial goals: 1. Improve diagnostics by providing a Diagnostic Report when a Tx fails. 2. Provide better visualization of Java Stack traces so problem areas pop out. 3. Capture wait information For number 1 I’m going to focus on servlets to begin with given that they represent the preponderance of requests made in AppServers today. This information will include information from the request object, the servlet being invoked, invocation time, transaction ID (if it exists), enlisted connections (database and messaging), oh yeah, and the Thread ID of execution. This is a mouthful to begin with anyway. Number 2 is really just applying some template information on a Java Stack trace. I want application classes to standout so developers will be able to quickly see where their application is involved. Infrastructure pieces like the server, Hibernate, TopLink, etc. would also be highlighted in a different color and style and plain old java classes would be a boring style as their are merely pawns in the transactional game. Finally, wouldn’t it be nice to know how long a thread has been waiting and for what reason? Is it waiting on a request from another server or perhaps there is a database locking problem. Did a WebService go awry? Basically, I want to know what in the heck a thread is waiting on. Please chime in with your thoughts. I spect that Aspects or Instrumentation may be the only way to go for some of this as many of the components we include won't have this capability in them.
