Re: Multi processor issue
Thanks Mark. Gael Mark Thomas wrote: In 5.5.21 onwards, the RD will be thread safe so the OP's code should work without modification. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Mark Thomas wrote: I plan to look at modifying the current RD code over the holiday. In 5.5.21 onwards, the RD will be thread safe so the OP's code should work without modification. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Can you suggest a fix? I'm not sure how this kind of thing could be safely veto'd... for instance, it might actually be appropriate for a RequestDispatcher to be re-used in some context (say, repeating a request twice) or even to do so with different threads, as long as the access is done strictly serially. Do you mean submittingan enhancement request with some proposed code? Fantastic! I'm glad we were able to help, even if it did take a while. Yes and I want to thank all the people from the mailing list for their good advices and also for giving me additional motivation to investigate further. Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Caldarale, Charles R wrote: Since query information is normally unique to a request, I think the above could be construed that the RequestDispatcher is cognizant of the query string. If so, each RequestDispatcher would be associated with a specific request. I agree that the RD is cognizant of the query string but I don't see that this prevents the RD from being re-used. The RD merges its query string (specified on creation) with the query string of the request passed to the include/forward method. No reason it can't do this several times. SRV.15.2.8 ServletContext, getRequestDispatcher(String): The resource can be dynamic or static. Since dynamic resources can come and go and their existence can be dependent on specific requests, there is an implication that a RequestDispatcher is associated with the request it has been created for. I see where you are coming form here but this strikes me as an application issue. Send lawyers, guns, and money. (That's not in the spec, but perhaps should be.) He he. Indeed! What does it for me is: a) that you can obtain an RD from the context; b) SRV.8.1 that states: The method uses the path to look up a servlet ... wraps it with a RequestDispatcher object, and returns the resulting object. In my mind this aligns RDs more with servlets than requests. I plan to look at modifying the current RD code over the holiday. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Gael, Marziou, Gael wrote: Can you suggest a fix? I'm not sure how this kind of thing could be safely veto'd... for instance, it might actually be appropriate for a RequestDispatcher to be re-used in some context (say, repeating a request twice) or even to do so with different threads, as long as the access is done strictly serially. Do you mean submitting an enhancement request with some proposed code? Well, I'm sure the the developers would appreciate that, but meant just a suggestion of how it might be done -- just in words, not necessarily any code. Often, problems like this only affect a small number of people, and fixing them would break other people's code -- there's always someone who found a way to use it in an unexpected way. -chris signature.asc Description: OpenPGP digital signature
RE: Multi processor issue
From: Mark Thomas [mailto:[EMAIL PROTECTED] Subject: Re: Multi processor issue I have re-read the sections of the spec you quoted several times and I don't see anything that suggests to me that the RequestDispatcher has only request level scope that would suggest the OPs use is invalid. Could you elaborate? As I said yesterday, the statements in the spec are hints and by no means definitive. They are certainly subject to interpretation. The quoted items are from the spec, my comments follow each. SRV.8.1.1 Query Strings in Request Dispatcher Paths: The ServletContext and ServletRequest methods that create RequestDispatcher objects using path information allow the optional attachment of query string information to the path. Parameters specified in the query string used to create the RequestDispatcher take precedence over other parameters of the same name passed to the included servlet. The parameters associated with a RequestDispatcher are scoped to apply only for the duration of the include or forward call. Since query information is normally unique to a request, I think the above could be construed that the RequestDispatcher is cognizant of the query string. If so, each RequestDispatcher would be associated with a specific request. SRV.8.4.1 Query String: The request dispatching mechanism is responsible for aggregating query string parameters when forwarding or including requests. Again, I think the above can be viewed as tying a RequestDispatcher object to a specific request. SRV.15.2.8 ServletContext, getRequestDispatcher(String): The resource can be dynamic or static. Since dynamic resources can come and go and their existence can be dependent on specific requests, there is an implication that a RequestDispatcher is associated with the request it has been created for. Send lawyers, guns, and money. (That's not in the spec, but perhaps should be.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Chuck, Caldarale, Charles R wrote: Parameters specified in the query string used to create the RequestDispatcher take precedence over other parameters of the same name passed to the included servlet. The parameters associated with a RequestDispatcher are scoped to apply only for the duration of the include or forward call. That part right there pretty much closes the book on this one for me: if you pass (URL) parameters into the methods that return RequestDispatcher objects, then you get a customized RequestDispatcher. There's no reason to think that no parameters is any different than some parameters. I don't think this is a bug in TC's implementation. This is a relatively subtle mistake that seems both easy enough to make and easy to fix. Gael, what was the reasoning behind caching that RequestDispatcher? Was it simply because you would have been calling the same method with the same parameters repeatedly, and figured that you'd get some performance edge by saving the resulting object? Were you observing any performance degradation at any point? Or was this a case of premature optimization? ;) I'd really like to hear if simply eliminating the caching of this object fixes your problem. Does it? -chris signature.asc Description: OpenPGP digital signature
RE: Multi processor issue
I don't think this is a bug in TC's implementation. This is a relatively subtle mistake that seems both easy enough to make and easy to fix. Maybe Tomcat should enforce this like for instance when we get an exception trying to write into a response that was already committed. It would have saved weeks of effort on our side. Gael, what was the reasoning behind caching that RequestDispatcher? Was it simply because you would have been calling the same method with the same parameters repeatedly, and figured that you'd get some performance edge by saving the resulting object? Were you observing any performance degradation at any point? Or was this a case of premature optimization? ;) We did implement this 2 years ago, so details have vanished but it was mainly based on our spec interpretation and also I admit on premature optimization. In several places, the spec defines different behaviors of RequestDispatcher when it was obtained using getNamedDispatcher(String) and noticeably for handling parameters. So, our interpretation was that a RequestDispatcher obtained using getNamedDispatcher() was different than one obtained using getRequestDispatcher(): it has a different behavior and a wider scope. This was the reason behind our implementation and also the fact that our previous container (before migrating to Tomcat) seemed to work this way (as far as I remember). I'd really like to hear if simply eliminating the caching of this object fixes your problem. Does it? So far so good, the new code has been running fine for 2 hours while it was failing usually within 30 minutes. Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Gael, Marziou, Gael wrote: I don't think this is a bug in TC's implementation. This is a relatively subtle mistake that seems both easy enough to make and easy to fix. Maybe Tomcat should enforce this like for instance when we get an exception trying to write into a response that was already committed. It would have saved weeks of effort on our side. Can you suggest a fix? I'm not sure how this kind of thing could be safely veto'd... for instance, it might actually be appropriate for a RequestDispatcher to be re-used in some context (say, repeating a request twice) or even to do so with different threads, as long as the access is done strictly serially. We did implement this 2 years ago, so details have vanished but it was mainly based on our spec interpretation and also I admit on premature optimization. No problem. I'm pretty sure we're all guilty of that some of the time. In several places, the spec defines different behaviors of RequestDispatcher when it was obtained using getNamedDispatcher(String) and noticeably for handling parameters. So, our interpretation was that a RequestDispatcher obtained using getNamedDispatcher() was different than one obtained using getRequestDispatcher(): it has a different behavior and a wider scope. Agreed... the spirit of the named dispatcher as opposed to a standard dispatcher obtained for a certain URI seems to be that it could be used universally. This was the reason behind our implementation and also the fact that our previous container (before migrating to Tomcat) seemed to work this way (as far as I remember). Perhaps TC should consider returning an object of a different type when using getNamedDispatcher -- one that does not keep any request-related information at all. I'd really like to hear if simply eliminating the caching of this object fixes your problem. Does it? So far so good, the new code has been running fine for 2 hours while it was failing usually within 30 minutes. Fantastic! I'm glad we were able to help, even if it did take a while. -chris signature.asc Description: OpenPGP digital signature
RE: Multi processor issue
By looking at the code of ApplicationDispatcher, I would tend to think that the answer is no because of fields like wrapRequest and requestURI (there are other fields but they are unused) however the javadoc does not mention it. I forgot to include the question but some of you may have guessed it ;-) The question was: can we call forward() on same ApplicationDispatcher object from 2 concurrent threads? Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
BINGO :-) Please check the servlet spec 2.5, SRV 8.2 To use a request dispatcher, a servlet calls either the include method or forward method of the RequestDispatcher interface. The parameters to these methods can be either the request and response arguments that were passed in via the service method of the javax.servlet interface, or instances of subclasses of the request an response wrapper classes that were introduced for version 2.3 of the specification. In the latter case, the wrapper instances must wrap the request or response objects that the container passed into the service method. The Container Provider should ensure that the dispatch of the request to a target servlet occurs in the same thread of the same JVM as the original request. Please note that the last paragraph. I assume if the servlet container works properly, your method simply can't work. Can you remove Dispatcher caching and check whether the issue is gone? regards Leon On 12/20/06, Marziou, Gael [EMAIL PROTECTED] wrote: By looking at the code of ApplicationDispatcher, I would tend to think that the answer is no because of fields like wrapRequest and requestURI (there are other fields but they are unused) however the javadoc does not mention it. I forgot to include the question but some of you may have guessed it ;-) The question was: can we call forward() on same ApplicationDispatcher object from 2 concurrent threads? Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Leon Rosenberg [mailto:[EMAIL PROTECTED] Subject: Re: Multi processor issue The Container Provider should ensure that the dispatch of the request to a target servlet occurs in the same thread of the same JVM as the original request. I'm not sure the above is really pertinent to the issue, which I think can be boiled down to this: Should a RequestDispatcher object be treated as if it were in the scope of a Request or instead in the scope of a Servlet? The spec provides no definitive statement that I can find, but there are a couple of hints that a RequestDispatcher should be considered unique to each request. In particular: SRV.8.1.1 Query Strings in Request Dispatcher Paths: The ServletContext and ServletRequest methods that create RequestDispatcher objects using path information allow the optional attachment of query string information to the path. Parameters specified in the query string used to create the RequestDispatcher take precedence over other parameters of the same name passed to the included servlet. The parameters associated with a RequestDispatcher are scoped to apply only for the duration of the include or forward call. SRV.8.4.1 Query String: The request dispatching mechanism is responsible for aggregating query string parameters when forwarding or including requests. SRV.15.2.8 ServletContext, getRequestDispatcher(String): The resource can be dynamic or static. Based on the above, I think the app's reuse of RequestDispatcher objects is indeed erroneous. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Caldarale, Charles R wrote: From: Leon Rosenberg [mailto:[EMAIL PROTECTED] Subject: Re: Multi processor issue The Container Provider should ensure that the dispatch of the request to a target servlet occurs in the same thread of the same JVM as the original request. I'm not sure the above is really pertinent to the issue, which I think can be boiled down to this: Should a RequestDispatcher object be treated as if it were in the scope of a Request or instead in the scope of a Servlet? The spec provides no definitive statement that I can find, but there are a couple of hints that a RequestDispatcher should be considered unique to each request. In particular: What about RequestDispatcher ServletContext.getNamedDispatcher(String) Since this method is at the Context level, any RequestDispatcher obtained effectively has context wide scope. It is this method that the OP is using and, given the non-threadsafe nature of o.a.c.core.ApplicationDispatcher, this looks like a Tomcat bug to me. I haven't had a chance to see how hard it might be to fix yet. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Mark Thomas [mailto:[EMAIL PROTECTED] Subject: Re: Multi processor issue What about RequestDispatcher ServletContext.getNamedDispatcher(String) Since this method is at the Context level, any RequestDispatcher obtained effectively has context wide scope. It is this method that the OP is using and, given the non-threadsafe nature of o.a.c.core.ApplicationDispatcher, this looks like a Tomcat bug to me. I haven't had a chance to see how hard it might be to fix yet. I think some Servlet spec clarification would be really helpful here. Unless the spec intends for there to be different flavors of RequestDispatcher, the apparent context-level scope seems to conflict with the request-specific comments I noted earlier. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Dong, I'm not sure I understand your suggestion, I attached below my bug
catcher in org.apache.catalina.connector.Request.
Do you mean I should add a delay before the while loop to increase the
length of the critical section?
What do you mean by yelding the CPU (sorry I'm French) ?
Gael
public Map getParameterMap() {
synchronized(parameterMap) {
Thread currentThread = Thread.currentThread();
if (pmapThread == null) {
// Store context of first caller thread
pmapThread = currentThread;
pmapStackTrace = currentThread.getStackTrace();
} else {
if (currentThread != pmapThread) {
String msg = Original thread = + pmapThread +
\nOriginal stack trace = +
stack2string(pmapStackTrace) +
\nCurrent thread = + currentThread;
throw new IllegalStateException(msg);
}
}
}
if (parameterMap.isLocked())
return parameterMap;
Enumeration enumeration = getParameterNames();
while (enumeration.hasMoreElements()) {
String name = enumeration.nextElement().toString();
String[] values = getParameterValues(name);
parameterMap.put(name, values);
}
parameterMap.setLocked(true);
return parameterMap;
}
-Original Message-
From: JiaDong Huang [mailto:[EMAIL PROTECTED]
Sent: Tuesday, December 12, 2006 10:42 AM
To: 'Tomcat Users List'
Subject: RE: Multi processor issue
Gael,
If you can build the Tomcat code/component, put in time delay code and
also yield the CPU, within the ParameterMap initialization procedure.
Dong
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
-Original Message- From: Marziou, Gael [mailto:[EMAIL PROTECTED] Sent: Tuesday, 12 December 2006 9:21 PM To: Tomcat Users List Subject: RE: Multi processor issue Do you mean I should add a delay before the while loop to increase the length of the critical section? Yes. Before and maybe within the loop. That should get the MT issue reproduced much easier. What do you mean by yelding the CPU (sorry I'm French) ? Yielding the CPU means to let your thread give up the CPU so as to give the others thread a change to run. The delay call Thread.sleep() should yield the CPU. Sorry, I should have said in my previous email: make some time delay - and the thread should yield the CUP. Dong - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dong, JiaDong Huang wrote: Yes. Before and maybe within the loop. That should get the MT issue reproduced much easier. This was a great idea, by the way. What do you mean by yelding the CPU (sorry I'm French) ? Yielding the CPU means to let your thread give up the CPU so as to give the others thread a change to run. The delay call Thread.sleep() should yield the CPU. Thread.yield() will also work. It's unclear which is better from the API, but I would imagine that a VM implementor would write the Thread.yield method such that any remaining CPU time is yielded immediately, whereas you would have to guess at the amount of sleep time to use (but I'm sure something like 500ms would be just fine, as CPU time slices are pretty small). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFfsBF9CaO5/Lv0PARAmKlAKCG7ABMw3INZLWyTwpV5wwc9X2OxACggk+z zlUGR1XVADFUDeRSloRIbq4= =lb4L -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: Multi processor issue Thread.yield() will also work. It's unclear which is better from the API, but I would imagine that a VM implementor would write the Thread.yield method such that any remaining CPU time is yielded immediately, whereas you would have to guess at the amount of sleep time to use (but I'm sure something like 500ms would be just fine, as CPU time slices are pretty small). Thread.yield() would be much, much more efficient than a sleep() call, both in terms of OS internal operations and response time back to the client. Neither has anything to do with the time slice. However, I don't think either one is really necessary in this exercise. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue I attached below my bug catcher in org.apache.catalina.connector.Request. Looks good to me. I'd be very interested in the results. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
I used sleep() as you can see in the modified version of getParameterMap() at the end of this message. I got the issue and here is the trace. What I call original thread is the one that was the first to call Requuest.getParameterMap() Correct me if I am wrong but it seems to me that the 2 threads are from Tomcat, not from our application. Gael java.lang.IllegalStateException: Original thread = Thread[http-80-3,5,main] Original stack trace = java.lang.Thread.dumpThreads() java.lang.Thread.getStackTrace() org.apache.catalina.connector.Request.getParameterMap()(Request.java:102 3) org.apache.catalina.connector.RequestFacade.getParameterMap()(RequestFac ade.java:411) org.apache.catalina.core.ApplicationHttpRequest.parseParameters()(Applic ationHttpRequest.java:727) org.apache.catalina.core.ApplicationHttpRequest.getParameter()(Applicati onHttpRequest.java:352) com.hp.swift.portal.state.HttpEventContextObject.getParameter()(HttpEven tContextObject.java:64) com.hp.cii.radix.user.web.home.Home2Context.doUpdateAlertSummary()(Home2 Context.java:709) com.hp.cii.radix.user.web.home.Home2Context.doDisplayAlertSummary()(Home 2Context.java:789) sun.reflect.NativeMethodAccessorImpl.invoke0() sun.reflect.NativeMethodAccessorImpl.invoke() sun.reflect.DelegatingMethodAccessorImpl.invoke() java.lang.reflect.Method.invoke() com.hp.swift.util.MethodInvoker.invoke()(MethodInvoker.java:52) com.hp.swift.state.core.StateContextAction.execute()(StateContextAction. java:51) com.hp.swift.state.core.Transition.executeAction()(Transition.java:124) com.hp.swift.state.core.StateMachineController.fireTransition()(StateMac hineController.java:1013) com.hp.swift.state.core.StateMachineController.doHandle()(StateMachineCo ntroller.java:788) com.hp.swift.state.core.StateMachineController.handle()(StateMachineCont roller.java:520) com.hp.swift.portal.servlet.SmvController.doHandleRequest()(SmvControlle r.java:393) com.hp.swift.portal.servlet.SmvController.doGet()(SmvController.java:235 ) javax.servlet.http.HttpServlet.service()(HttpServlet.java:689) javax.servlet.http.HttpServlet.service()(HttpServlet.java:802) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli cationFilterChain.java:252) org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi lterChain.java:173) org.apache.catalina.core.ApplicationDispatcher.invoke()(ApplicationDispa tcher.java:672) org.apache.catalina.core.ApplicationDispatcher.processRequest()(Applicat ionDispatcher.java:463) org.apache.catalina.core.ApplicationDispatcher.doForward()(ApplicationDi spatcher.java:359) org.apache.catalina.core.ApplicationDispatcher.forward()(ApplicationDisp atcher.java:301) com.hp.swift.proxy.ProxySessionController.forwardRequest()(ProxySessionC ontroller.java:57) com.hp.swift.portal.servlet.SessionController$1.run()(SessionController. java:164) com.hp.swift.security.manager.LightSecurityManager.doAsPrivileged()(Ligh tSecurityManager.java:272) com.hp.swift.security.manager.SecurityChecker.doAsPrivileged()(SecurityC hecker.java:187) com.hp.swift.portal.servlet.SessionController.forwardRequestAs()(Session Controller.java:159) com.hp.swift.portal.servlet.SessionController.doHttp()(SessionController .java:134) com.hp.swift.portal.servlet.SessionController.doGet()(SessionController. java:110) javax.servlet.http.HttpServlet.service()(HttpServlet.java:689) com.hp.swift.portal.init.FirstLevelInitableServlet.service()(FirstLevelI nitableServlet.java:154) javax.servlet.http.HttpServlet.service()(HttpServlet.java:802) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli cationFilterChain.java:252) org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi lterChain.java:173) com.hp.swift.portal.servlet.HttpTracingFilter.doFilter()(HttpTracingFilt er.java:94) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli cationFilterChain.java:202) org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi lterChain.java:173) com.hp.swift.portal.servlet.SecurityFilter.doFilter()(SecurityFilter.jav a:61) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli cationFilterChain.java:202) org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi lterChain.java:173) org.apache.catalina.core.StandardWrapperValve.invoke()(StandardWrapperVa lve.java:213) org.apache.catalina.core.StandardContextValve.invoke()(StandardContextVa lve.java:178) org.apache.catalina.core.StandardHostValve.invoke()(StandardHostValve.ja va:126)
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue Correct me if I am wrong but it seems to me that the 2 threads are from Tomcat, not from our application. They are, but they go through a lot of your application code prior to reaching getParameterMap(). I'm especially suspicious of this area: com.hp.swift.portal.servlet.SessionController.forwardRequestAs()(Session Controller.java:159) com.hp.swift.portal.servlet.SessionController.doHttp()(SessionController .java:134) com.hp.swift.portal.servlet.SessionController.doGet()(SessionController. java:110) Anything dealing with a session must be careful not to expose a Request object to the session scope, since there's the risk another request for the same session may get confused with the first - I suspect that's what's happening here. You should take a careful look at all the com.hp code in the stack trace (not just the above three places) and verify that Request objects are not getting mixed up. Might want to add some trace code in various places to insure that a given thread is procesing the same Request object at all points. Could also try turning off APR (remove or rename tcnative-1.dll) and see if that has an effect. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Anything dealing with a session must be careful not to expose a Request object to the session scope, since there's the risk another request for the same session may get confused with the first - I suspect that's what's happening here. Will do. You should take a careful look at all the com.hp code in the stack trace (not just the above three places) and verify that Request objects are not getting mixed up. Might want to add some trace code in various places to insure that a given thread is procesing the same Request object at all points. Yes, it makes sense. Could also try turning off APR (remove or rename tcnative-1.dll) and see if that has an effect. I already did at the beginning of our investigation as I suspected the JNI code but it had no effect. Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue I already did at the beginning of our investigation as I suspected the JNI code but it had no effect. That's what I thought, but wasn't sure, since at least one of the stack traces you sent (8 December) had no APR entries in it, but this one did. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Great to see the bottom of the MT issue has been revealed/understood. I believe Gael may be able to fix his app now. Hope it can be an easy fix... Not sure if the app really cares which thread should do the parameterMap initialization. If it does not, then it would be a good idea for Tomcat to refine/enhance the parameterMap initialization procedure to be MT safe. Apparently, the current lock facility (isLocked() and setLocked()) does not protect MT case properly, while initializing. Is it worthwhile and appropriate to have certain MT synchronization facility like isInitialized() and setInitialized() around the code? In theory, the second thread can wait for isInitialized() there. Not sure if it will cause any concern to other apps, like having other racing condition. Really depend on the parameterMap usage/operation. MT issue fix can never be simple. Chuck understands parameterMap usage very well. Maybe he can make further comments/recommendation to the Tomcat dev team. Dong -Original Message- From: Marziou, Gael [mailto:[EMAIL PROTECTED] Sent: Wednesday, 13 December 2006 3:24 AM To: Tomcat Users List Subject: RE: Multi processor issue Anything dealing with a session must be careful not to expose a Request object to the session scope, since there's the risk another request for the same session may get confused with the first - I suspect that's what's happening here. Will do. You should take a careful look at all the com.hp code in the stack trace (not just the above three places) and verify that Request objects are not getting mixed up. Might want to add some trace code in various places to insure that a given thread is procesing the same Request object at all points. Yes, it makes sense. Could also try turning off APR (remove or rename tcnative-1.dll) and see if that has an effect. I already did at the beginning of our investigation as I suspected the JNI code but it had no effect. Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: JiaDong Huang [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue Not sure if the app really cares which thread should do the parameterMap initialization. You're missing the entire point of this exercise: no Request object should EVER be processed by more than one thread. No synchronization changes are needed within Tomcat. The most likely cause of the problem is code in the app itself exposing a Request object at the session or context level (e.g., storing it as a session attribute), thereby causing a thread working on a second, separate request to grab the wrong Request object. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Jiadong/Gael Can you download the code to a sandbox and implement your synchronized algorithm in setLocked to test out the intended result? Or is the code not available? M- --- This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. --- Le présent message électronique (y compris les pièces qui y sont annexées, le cas échéant) s'adresse au destinataire indiqué et peut contenir des renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le destinataire de ce document, nous vous signalons qu'il est strictement interdit de le diffuser, de le distribuer ou de le reproduire. - Original Message - From: JiaDong Huang [EMAIL PROTECTED] To: 'Tomcat Users List' users@tomcat.apache.org Sent: Tuesday, December 12, 2006 7:24 PM Subject: RE: Multi processor issue Great to see the bottom of the MT issue has been revealed/understood. I believe Gael may be able to fix his app now. Hope it can be an easy fix... Not sure if the app really cares which thread should do the parameterMap initialization. If it does not, then it would be a good idea for Tomcat to refine/enhance the parameterMap initialization procedure to be MT safe. Apparently, the current lock facility (isLocked() and setLocked()) does not protect MT case properly, while initializing. Is it worthwhile and appropriate to have certain MT synchronization facility like isInitialized() and setInitialized() around the code? In theory, the second thread can wait for isInitialized() there. Not sure if it will cause any concern to other apps, like having other racing condition. Really depend on the parameterMap usage/operation. MT issue fix can never be simple. Chuck understands parameterMap usage very well. Maybe he can make further comments/recommendation to the Tomcat dev team. Dong -Original Message- From: Marziou, Gael [mailto:[EMAIL PROTECTED] Sent: Wednesday, 13 December 2006 3:24 AM To: Tomcat Users List Subject: RE: Multi processor issue Anything dealing with a session must be careful not to expose a Request object to the session scope, since there's the risk another request for the same session may get confused with the first - I suspect that's what's happening here. Will do. You should take a careful look at all the com.hp code in the stack trace (not just the above three places) and verify that Request objects are not getting mixed up. Might want to add some trace code in various places to insure that a given thread is procesing the same Request object at all points. Yes, it makes sense. Could also try turning off APR (remove or rename tcnative-1.dll) and see if that has an effect. I already did at the beginning of our investigation as I suspected the JNI code but it had no effect. Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Chuck, I am very clear about the whole thing. Please try to understand my point properly too. Your point can be right and valid and would be very significant to help Gael. Gael should be able to get to the point where his app might do wrong, or miss leading Tomcat threads. What I have been saying is that we should reconsider very carefully whether there should be any enhancement in Tomcat. From the Tomcat code you dug out, there is something not quite perfect, in term of MT safe while initializing. No argument. One of your posted messages did state that it is valid for two threads to access/operate on the ParameterMap once it has been established. So, it may be pretty hard and confusing what the app developer should be doing at the upper level. Whether it is appropriate to enhance Tomcat or not, and when, I don't think I am in the right position to justify. I guess once Gael has clarified his issue. He may be in a better position to make significant comments, and see if you can agree. Dong -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Wednesday, 13 December 2006 11:34 AM To: Tomcat Users List Subject: RE: Multi processor issue From: JiaDong Huang [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue Not sure if the app really cares which thread should do the parameterMap initialization. You're missing the entire point of this exercise: no Request object should EVER be processed by more than one thread. No synchronization changes are needed within Tomcat. The most likely cause of the problem is code in the app itself exposing a Request object at the session or context level (e.g., storing it as a session attribute), thereby causing a thread working on a second, separate request to grab the wrong Request object. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: JiaDong Huang [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue What I have been saying is that we should reconsider very carefully whether there should be any enhancement in Tomcat. From the Tomcat code you dug out, there is something not quite perfect, in term of MT safe while initializing. Absolutely wrong. There is nothing incorrect about the code in either the Request class or the ParameterMap class; it fulfills every aspect of its specification. Changing the default behavior of the Request class to be synchronized on its get/set methods would be completely inane. One of your posted messages did state that it is valid for two threads to access/operate on the ParameterMap once it has been established. Yes - with a big caveat. Any application doing so must provide its own synchronization to insure the integrity of the Request object (and embedded ParameterMap). In this case, the application has NO intention of using two threads on one Request - it simply has a bug in it. If we were to apply your synchronize always approach to software development, we'd need a lot better performance out of the hardware than Moore's law expects in order to keep up. Nor would it solve very many problems, including this one. Think about it - what would have happened if the Request object did have synchronized methods? The underlying flaw in the app that is causing one thread to lose track of its own Request and latch onto another would have gone undetected - except for some client somewhere not getting the expected response, but instead seeing someone else's output show up. Simple synchronization does not address the real problem here. So, it may be pretty hard and confusing what the app developer should be doing at the upper level. It's not really, IF you take time to read the specs - but few do. Admittedly, there are a lot of things that should be studied, and project deadlines rarely take that into account. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Appreciate your reply analyzing my guessing cases. So it is just a case of simple MT issue - multiple threads have accessed/operated on a class/object that is not MT safe. I thought they had done certain code review and did not find any MT safe issue. If an effective stress test had been done on single CPU environment, the issue should have been revealed too. Dong -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Monday, 11 December 2006 3:30 PM To: Tomcat Users List Subject: RE: Multi processor issue From: JiaDong Huang [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue It is a MT issue, but MP specific. Strictly speaking, it's not MP-specific. If run long enough, the problem will appear on a single CPU. Its occurrence would require the first thread to exhaust its CPU time slice at a critical point, and have the second thread be dispatched immediately. (In most operating systems, when a thread uses up its time slice, it's placed at the end of the dispatch queue for its priority, thus allowing other threads of the same or higher priority to run ahead of it. Note that the OS-defined priority in this case often has little to do with the Java-specified priority for a thread.) That said, the situation is certainly much, much more likely to occur on an MP system. The Tomcat code you had dug out (having problem and throwing the exception) has been designed as single threaded. But in MP environment, multiple threads get in and cause issue like this. That means somewhere in Tomcat or JVM, the synchronization facility has already been broken While that would be true in general, it's not the case here. Objects of the Request class are never intended to be used concurrently by multiple threads, so there is no synchronization defined or implemented for them. No second thread - whatever its origin - should be operating on the same Request object that another thread is handling. As far as I understand, Tomcat connector may use JNI The APR and AJP connects are implemented with JNI, the standard HTTP and HTTPS ones are pure Java. not sure the threads we are talking about had run through the code implemented with JNI. They did not, but it's not really relevant, given the Tomcat requirement of only one thread manipulating a Request object at a time. Threads running through JNI may be re-marshaled between OS spaces - switching OS rings or VM. They can change ring (generically, the thread privilege level), but not VMs. If you're suggesting that one thread may change its process association, that is theoretically possible on some OS implementations (anybody remember Multics?), but it is certainly not happening here, and, in any event, is completely irrelevant. (Any such change from one process to another must be extremely tightly controlled for security purposes, and is a capability simply not available to typical user threads.) In another word, while switching between rings, the lock associated with the objects may have been lost, etc. That's erroneous, even if it were relevant to this situation, which it isn't. There's no synchronization issue involved, since the Request object is not intended to be manipulated concurrently by multiple threads. This problem is almost certainly an application problem, either starting an additional thread somewhere or exposing the Request object improperly (e.g., storing a reference to it in the session) so that another thread erroneously starts working with this Request instead of the one its supposed to be working on. Also, the JVM or OS API may need other synchronization facility underneath while switching rings. These are only my guess anyway. No code in the JVM depends on or expects to change privilege level (other than the privileges controlled entirely within the JVM), due to its platform independence. I think you may be trying to apply some rather esoteric capabilities of certain operating systems, but they really have no bearing on this situation. It could be the JVM's synchronization facility does not work properly in MP, for Tomcat. Or, Tomcat could be enhanced to prevent this sort synchronization issue from happening. You're ignoring one basic fact: this problem has been reported on only *one* system running *one* particular application. If there were a general synchronization problem in Tomcat, the JVM, or the OS, it would certainly show up on some of the thousands (millions?) of other multi-processor systems running Tomcat. As I stated earlier, we have zero problems running Tomcat on our 32x servers (of various architectures). There is a very small chance that something in the hardware on this particular system has gone bad (e.g., lock signal not being propagated on the bus), but such a problem would undoubtedly show up in more places than this one spot in Tomcat. After reviewing the messages, it is tomcat5.exe has been modified. That means it may be tomcat5.exe should
RE: Multi processor issue
From: JiaDong Huang [mailto:[EMAIL PROTECTED] I thought they had done certain code review and did not find any MT safe issue. They had (and I naively took them at their word), and they didn't find any such issue. However, as Chuck points out so effectively, we didn't find any issues isn't the same thing as there are no issues in the code. Unfortunately, code reviews merely increase the chance of revealing a defect - they don't increase it to 100%. Nor does any other technique, otherwise the safety-critical folks would be using it exclusively. - Peter -- So far as we know, we have never had an undiscovered error. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Thanks for all replies, you have convinced me that we should investigate further. I propose to implement a bug catcher in Tomcat ParameterMap by storing the thread that called the constructor amd then check its invariance when calling other methods (e.g. isLocked) of this class, if the invariance is violated we will throw an exception. It seems to me that with this data, we should be able to determine which code is accessing the map and so we should know whether it's a bug in the application or not. Any other suggestion? Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: JiaDong Huang [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue So it is just a case of simple MT issue - multiple threads have accessed/operated on a class/object that is not MT safe. I wouldn't label the situation as simple - threading issues frequently are not. If an effective stress test had been done on single CPU environment, the issue should have been revealed too. In theory yes, but it could well take years to trigger the problem on a single CPU, unless artificial pauses are inserted to widen the timing windows. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue I propose to implement a bug catcher in Tomcat ParameterMap by storing the thread that called the constructor amd then check its invariance when calling other methods (e.g. isLocked) of this class, if the invariance is violated we will throw an exception. From a purely theoretical perspective, it is valid for two threads to call getParameterMap() (and therefore isLocked()), once the ParameterMap has been established. Probably not a concern in the real world. Since the only call to ParameterMap.isLocked() comes from Request.getParameterMap(), I'd put the saving of the calling Thread object in a synchronized block there, along with a check to insure no other thread had been there. In addition, I would also save the array returned by getStackTrace() (assuming you're running on SE 5) to provide some context for the call. This would also help to widen the window and hopefully cause the problem to occur sooner. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: JiaDong Huang [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue It is a MT issue, but MP specific. Strictly speaking, it's not MP-specific. If run long enough, the problem will appear on a single CPU. Its occurrence would require the first thread to exhaust its CPU time slice at a critical point, and have the second thread be dispatched immediately. (In most operating systems, when a thread uses up its time slice, it's placed at the end of the dispatch queue for its priority, thus allowing other threads of the same or higher priority to run ahead of it. Note that the OS-defined priority in this case often has little to do with the Java-specified priority for a thread.) That said, the situation is certainly much, much more likely to occur on an MP system. The Tomcat code you had dug out (having problem and throwing the exception) has been designed as single threaded. But in MP environment, multiple threads get in and cause issue like this. That means somewhere in Tomcat or JVM, the synchronization facility has already been broken While that would be true in general, it's not the case here. Objects of the Request class are never intended to be used concurrently by multiple threads, so there is no synchronization defined or implemented for them. No second thread - whatever its origin - should be operating on the same Request object that another thread is handling. As far as I understand, Tomcat connector may use JNI The APR and AJP connects are implemented with JNI, the standard HTTP and HTTPS ones are pure Java. not sure the threads we are talking about had run through the code implemented with JNI. They did not, but it's not really relevant, given the Tomcat requirement of only one thread manipulating a Request object at a time. Threads running through JNI may be re-marshaled between OS spaces - switching OS rings or VM. They can change ring (generically, the thread privilege level), but not VMs. If you're suggesting that one thread may change its process association, that is theoretically possible on some OS implementations (anybody remember Multics?), but it is certainly not happening here, and, in any event, is completely irrelevant. (Any such change from one process to another must be extremely tightly controlled for security purposes, and is a capability simply not available to typical user threads.) In another word, while switching between rings, the lock associated with the objects may have been lost, etc. That's erroneous, even if it were relevant to this situation, which it isn't. There's no synchronization issue involved, since the Request object is not intended to be manipulated concurrently by multiple threads. This problem is almost certainly an application problem, either starting an additional thread somewhere or exposing the Request object improperly (e.g., storing a reference to it in the session) so that another thread erroneously starts working with this Request instead of the one its supposed to be working on. Also, the JVM or OS API may need other synchronization facility underneath while switching rings. These are only my guess anyway. No code in the JVM depends on or expects to change privilege level (other than the privileges controlled entirely within the JVM), due to its platform independence. I think you may be trying to apply some rather esoteric capabilities of certain operating systems, but they really have no bearing on this situation. It could be the JVM's synchronization facility does not work properly in MP, for Tomcat. Or, Tomcat could be enhanced to prevent this sort synchronization issue from happening. You're ignoring one basic fact: this problem has been reported on only *one* system running *one* particular application. If there were a general synchronization problem in Tomcat, the JVM, or the OS, it would certainly show up on some of the thousands (millions?) of other multi-processor systems running Tomcat. As I stated earlier, we have zero problems running Tomcat on our 32x servers (of various architectures). There is a very small chance that something in the hardware on this particular system has gone bad (e.g., lock signal not being propagated on the bus), but such a problem would undoubtedly show up in more places than this one spot in Tomcat. After reviewing the messages, it is tomcat5.exe has been modified. That means it may be tomcat5.exe should be considered to enhance, in the area of thread synchronization/scheduling. Completely off base. The tomcat5.exe program is simply another way to launch a standard JVM; it does almost nothing other than apply the configured JVM options. It is merely the starting point for the process, so it's the easiest spot at which to impose affinity rules that will apply to all threads of the process. To have the thread scheduler single threaded with single CPU would be safer/better, and a reasonable thing to do. Perhaps safer (not guaranteed
Re: Multi processor issue
the affinity 'trick' is a replacement dll for the 80 runtime stock libraries Well, the one we used does not work this way. In fact, we started by changing the affinity of Tomcat service's live process using Windows taskmgr or Windows System Resource Manager. Then, as problem disappeared, we looked for a permanent way to set this affinity and the tool that we used does it by settings the affinity bitmask in the executable header, so it does not require any special DLL. I compared the 2 executables and they differ only one byte. http://forums.amd.com/lofiversion/index.php/t65211.html -- Gaël - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] So, I don't believe it's a Tomcat, I would rather think to an H/W, or JRE or OS issue. Fascinating - partly because I may hit the same issue soon and partly to log it on the list for the future, could you tell me 32-bit or 64-bit OS, service pack levels, and what the hardware is? I don't want to trip over the same problem, but nor do I want to give the hardware too wide a berth! - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: Multi processor issue We have recently switched a stable web application from JRun 4 to Tomcat 5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08, running some testing for months, but when we moved to production environment we immediately experienced errors IllegalStateException: No modifications are allowed to a locked ParameterMap. We made code reviews of our application code and Tomcat's code and did not find any thread safety issues. Look again. So, I don't believe it's a Tomcat, I would rather think to an H/W, or JRE or OS issue. Welcome to the world of multi-processors (which I've been working with since the 1960s.) The problem is almost undoubtedly in your application - we run Tomcat on 16x and 32x systems without any such issues. Within Tomcat, request objects are normally processed by a single thread at a time, yet you note that two threads are accessing this particular request - where did the second one come from? Is your app starting extra threads? (Perhaps unknowingly, by using some third-party package.) Since requests are typically handled by a single thread, there is little or no synchronization done during manipulation of references associated with them, such as the parameter map. If your app is using multiple threads to handle a request, it needs to provide its own synchronization when calling methods of the request that manipulate data associated with the request - even when just reading, since the other thread may be changing it concurrently. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
Good Morning Gael- which VM are you using for this test? Thanks, Martin -- --- This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. - Original Message - From: Marziou, Gael [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org Sent: Friday, December 08, 2006 9:22 AM Subject: Multi processor issue Hi, As a preamble, let me say that I am not complaining about Tomcat (as the problem root cause is uncertain) and I don't expect much assistance as we are happy with our workaround. However, our experience could help others or someone may have other ideas. We have recently switched a stable web application from JRun 4 to Tomcat 5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08, running some testing for months, but when we moved to production environment we immediately experienced errors IllegalStateException: No modifications are allowed to a locked ParameterMap. Stack trace analysis concluded to RequestFacade objects shared between 2 threads. We made code reviews of our application code and Tomcat's code and did not find any thread safety issues. So, we checked all differences between staging and production, we thought that the major difference was that the staging server was a single CPU machine while the production servers had 2 physical processors and 4 logical ones. So, we constrained the Tomcat service to run on only one processor by setting its processor affinity mask to 1 (run on CPU 0) and after several weeks of monitoring we concluded that it fixed the problem. So, we used a small utility (search for setaffinity.zip) to patch the affinity mask in the Tomcat service executable (tomcat5.exe). Later on, we set up a second production environment also on bi-processor servers but we did not experience the same issue. The hardware is not exactly the same as first production setup and the OS is also different (Windows 2000 on 2nd env, Windows 2003 on 1st env having the issue). So, I don't believe it's a Tomcat, I would rather think to an H/W, or JRE or OS issue. In any case, it's a rather complex issue to investigate and we haven't enough time for this as we are happy with performances running on one single CPU. Thanks, Gael Marziou Hewlett Packard - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Martin Gainty [mailto:[EMAIL PROTECTED] Subject: Re: Multi processor issue which VM are you using for this test? Try actually reading the message: We have recently switched a stable web application from JRun 4 to Tomcat 5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08 - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Thanks Chuck for your reply, I only cited part of your message where I could follow up. Within Tomcat, request objects are normally processed by a single thread at a time, yet you note that two threads are accessing this particular request - where did the second one come from? Both were web threads and error did occur at the very beginning of request processing when analyzing the request or when logging it in a filter. Is your app starting extra threads? (Perhaps unknowingly, by using some third-party package.) No. We checked this under profiler and all our threads are built by our thread factories and do not manipulate any request objects only our own business objects. Since requests are typically handled by a single thread, there is little or no synchronization done during manipulation of references associated with them, such as the parameter map. That's also our understanding. Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gael, Within Tomcat, request objects are normally processed by a single thread at a time, yet you note that two threads are accessing this particular request - where did the second one come from? Both were web threads and error did occur at the very beginning of request processing when analyzing the request or when logging it in a filter. Can you post the stack trace for that IllegalStateException? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFeZcR9CaO5/Lv0PARAvnrAKC34v6kL7obecqyXY/nZoujVI63SACfRo7t bOxjE3GJithcAiczrsIjWH4= =TG0h -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
This is our code and there's no threading here. -Original Message- From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] Sent: Friday, December 08, 2006 6:31 PM To: Tomcat Users List Subject: RE: Multi processor issue From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue com.hp.swift.portal.state.HttpEventContextObject.getParameter( HttpEventContextObject.java:64) This looks suspicious - perhaps this portal code is the source of the second thread. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
ApplicationHttpRequest.parseParameters isn't synchronized... thats probably the source of your evil. leon On 12/8/06, Leon Rosenberg [EMAIL PROTECTED] wrote: looks like Request.getParameterMap() is called twice. The first call locks the request, the second fails. what exactly does your com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC ontextObject.java:64) ? Could it be that two threads passes through this codepiece concurrently and both calls comes through to the Request object? That would be a clear synchronization issue and pretty typical for tomcat :-) regards Leon On 12/8/06, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue com.hp.swift.portal.state.HttpEventContextObject.getParameter( HttpEventContextObject.java:64) This looks suspicious - perhaps this portal code is the source of the second thread. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
First, to clarify in my message, I did put 2 stack traces of 2 independent errors. Tomcat does not synchronize before locking the map but that should not be an issue because the request is guaranteed to be processed by only one thread. HttpEventContextObject.getParameter() does just extract the parameters of the request to store them in what we cal an event, I suppose it could be compared to a Struts action. So, we only read the Request parameters, nothing fancy. Gael -Original Message- From: Leon Rosenberg [mailto:[EMAIL PROTECTED] Sent: Friday, December 08, 2006 6:38 PM To: Tomcat Users List Subject: Re: Multi processor issue looks like Request.getParameterMap() is called twice. The first call locks the request, the second fails. what exactly does your com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC ontextObject.java:64) ? Could it be that two threads passes through this codepiece concurrently and both calls comes through to the Request object? That would be a clear synchronization issue and pretty typical for tomcat :-) regards Leon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Leon, Leon Rosenberg wrote: ApplicationHttpRequest.parseParameters isn't synchronized... thats probably the source of your evil. That should not really be an issue... the real problem is that it appears that he has two threads accessing this code at the same time when he is only expecting one. Sure, synchronization might fix the problem, but it doesn't really get to the root of why it's happening in the first place. Gael, It probably won't end up showing anything, but could you catch the IllegalStateException and add a full thread dump in the exception handler? Then you might see what other threads are using that particular request object. I say it probably won't help because by the time the thread dump is done, your threads are probably out of that code anyway. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFeajl9CaO5/Lv0PARAtpQAJ4jwfiHJRw4InJIaHAPqzARZ3qvAQCeLI4C V6u7ZA4iDNRdga+XWnnZDNk= =ny0B -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue This is our code and there's no threading here. Does the full stack trace show where the threads originated? You appear to have given us only a partial of each error. Is the request object stored in any session or global structure? Could another thread have found it there and picked it up erroneously? Do you have any listeners configured that might be getting in the game? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue Tomcat does not synchronize before locking the map but that should not be an issue because the request is guaranteed to be processed by only one thread. But the evidence seems to show two threads are manipulating the same request, otherwise you wouldn't get the error on the ParameterMap. Does a full thread dump show any threads that aren't part of the set started by Tomcat? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
On 12/8/06, Caldarale, Charles R [EMAIL PROTECTED] wrote: From: Marziou, Gael [mailto:[EMAIL PROTECTED] Subject: RE: Multi processor issue Tomcat does not synchronize before locking the map but that should not be an issue because the request is guaranteed to be processed by only one thread. But the evidence seems to show two threads are manipulating the same request, otherwise you wouldn't get the error on the ParameterMap. Does a full thread dump show any threads that aren't part of the set started by Tomcat? either that, or a Request is reused before recycle() has been executed completely. I fear we have to know what your software on top does exactly :-) Alas, you could do a test and check whether the method getParameter is actually ever called by two threads during one lifecycle (patching the ParameterMap class, storing last accessing thread, and checking whether another thread is already stored on access) Theoretically Cael is right with the above, but only theoretically, since we don't know how thread safe Gael's ontop application is, and even than, it wouldn't be the first synchronization issue we found in tomcat. And Gael, the CPU is the most valuable resource in the production environment, are you really ready to give up your scaleability but simply not investigating further? Leon - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
Does the full stack trace show where the threads originated? You appear to have given us only a partial of each error. OK, see below. Is the request object stored in any session or global structure? No. Could another thread have found it there and picked it up erroneously? No. Do you have any listeners configured that might be getting in the game? No. java.lang.IllegalStateException: No modifications are allowed to a locked ParameterMap at org.apache.catalina.util.ParameterMap.put(ParameterMap.java:165) at org.apache.catalina.connector.Request.getParameterMap(Request.java:1024) at org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacad e.java:411) at org.apache.catalina.core.ApplicationHttpRequest.parseParameters(Applicat ionHttpRequest.java:727) at org.apache.catalina.core.ApplicationHttpRequest.getParameter(Application HttpRequest.java:352) at com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC ontextObject.java:64) at com.hp.cii.radix.user.web.home.Home2Context.doUpdateSolutionSummary(Home 2Context.java:833) at com.hp.cii.radix.user.web.home.Home2Context.doDisplaySolutionSummary(Hom e2Context.java:919) at sun.reflect.GeneratedMethodAccessor593.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.hp.swift.util.MethodInvoker.invoke(MethodInvoker.java:52) at com.hp.swift.state.core.StateContextAction.execute(StateContextAction.ja va:51) at com.hp.swift.state.core.Transition.executeAction(Transition.java:124) at com.hp.swift.state.core.StateMachineController.fireTransition(StateMachi neController.java:1013) at com.hp.swift.state.core.StateMachineController.doHandle(StateMachineCont roller.java:788) at com.hp.swift.state.core.StateMachineController.handle(StateMachineContro ller.java:520) at com.hp.swift.portal.servlet.SmvController.doHandleRequest(SmvController. java:393) at com.hp.swift.portal.servlet.SmvController.doGet(SmvController.java:235) at com.hp.swift.portal.servlet.SmvController.doPost(SmvController.java:215) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc her.java:672) at org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio nDispatcher.java:463) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp atcher.java:359) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat cher.java:301) at com.hp.swift.proxy.ProxySessionController.forwardRequest(ProxySessionCon troller.java:57) at com.hp.swift.portal.servlet.SessionController$1.run(SessionController.ja va:159) at com.hp.swift.security.manager.LightSecurityManager.doAsPrivileged(LightS ecurityManager.java:272) at com.hp.swift.security.manager.SecurityChecker.doAsPrivileged(SecurityChe cker.java:187) at com.hp.swift.portal.servlet.SessionController.forwardRequestAs(SessionCo ntroller.java:155) at com.hp.swift.portal.servlet.SessionController.doHttp(SessionController.j ava:132) at com.hp.swift.portal.servlet.SessionController.doPost(SessionController.j ava:102) at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) at com.hp.swift.portal.init.FirstLevelInitableServlet.service(FirstLevelIni tableServlet.java:154) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at com.hp.swift.portal.servlet.HttpTracingFilter.doFilter(HttpTracingFilter .java:89) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at com.hp.swift.portal.servlet.SecurityFilter.doFilter(SecurityFilter.java: 61) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv e.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv e.java:178) at
RE: Multi processor issue
But the evidence seems to show two threads are manipulating the same request, otherwise you wouldn't get the error on the ParameterMap. Does a full thread dump show any threads that aren't part of the set started by Tomcat? either that, or a Request is reused before recycle() has been executed completely. We assumed this as well but after instrumenting more our code, we concluded it was not possible because the corruption seemed to occur before the end of the 2 requests processing and so before recycle() was called. I got some logs which sort of showed this but I can't find them right now, I have stopped working on this issue 3 weeks ago. Alas, you could do a test and check whether the method getParameter is actually ever called by two threads during one lifecycle (patching the ParameterMap class, storing last accessing thread, and checking whether another thread is already stored on access) We did it but for some reason the log statements did not get logged in our usual log file. Then after we tried the affinity trick which made the symptom to disappear and got no chance to try again. And Gael, the CPU is the most valuable resource in the production environment, are you really ready to give up your scaleability but simply not investigating further? Well, this application is load balanced on 4 servers and the load is currently pretty low on each of them even when running on a single CPU. We did add more logs to our application and we did also instrument Tomcat but with no luck and as we were unable to reproduce it in a dev environment, it's difficult to ask for short downtimes to restart a server in production. On top of that, my team is being moved to another project... Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
the affinity 'trick' is a replacement dll for the 80 runtime stock libraries The individual most familiar with interfacing to the runtime is mladen turk I would make all attempts to ping him on the solution As a comparison between the 2 dlls Here are the binary characteristics of the stock msvcr80.dll that ships with retail windows packages Section contains the following exports for MSVCR80.dll 0 characteristics 40ECD724 time date stamp Thu Jul 08 01:09:56 2004 0.00 version 1 ordinal base 1367 number of functions 1367 number of names ordinal hint RVA name /*and the replacement msvcr80.dll provided in setaffinity*/ Section contains the following exports for MSVCR80.dll 0 characteristics 4333A455 time date stamp Fri Sep 23 02:44:37 2005 0.00 version 1 ordinal base 1459 number of functions 1459 number of names ordinal hint RVA name so the new msvcr80.dll from setaffinity has 92 new functions which allow it to work reliably with MP calls HTH, Martin-- --- This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. --- Le présent message électronique (y compris les pièces qui y sont annexées, le cas échéant) s'adresse au destinataire indiqué et peut contenir des renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le destinataire de ce document, nous vous signalons qu'il est strictement interdit de le diffuser, de le distribuer ou de le reproduire. But the evidence seems to show two threads are manipulating the same request, otherwise you wouldn't get the error on the ParameterMap. Does a full thread dump show any threads that aren't part of the set started by Tomcat? either that, or a Request is reused before recycle() has been executed completely. We assumed this as well but after instrumenting more our code, we concluded it was not possible because the corruption seemed to occur before the end of the 2 requests processing and so before recycle() was called. I got some logs which sort of showed this but I can't find them right now, I have stopped working on this issue 3 weeks ago. Alas, you could do a test and check whether the method getParameter is actually ever called by two threads during one lifecycle (patching the ParameterMap class, storing last accessing thread, and checking whether another thread is already stored on access) We did it but for some reason the log statements did not get logged in our usual log file. Then after we tried the affinity trick which made the symptom to disappear and got no chance to try again. And Gael, the CPU is the most valuable resource in the production environment, are you really ready to give up your scaleability but simply not investigating further? Well, this application is load balanced on 4 servers and the load is currently pretty low on each of them even when running on a single CPU. We did add more logs to our application and we did also instrument Tomcat but with no luck and as we were unable to reproduce it in a dev environment, it's difficult to ask for short downtimes to restart a server in production. On top of that, my team is being moved to another project... Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From the description of the symptom - months' testing experience, we could pretty much confirm that it is a MP issue, not a MT issue. Chasing the stack dump of Java may not get to any significant point. If you try to fix the MP specific issue like this at the high level like Tomcat/JAVA app code, you will get to a situation like: fix one issue, another issue will come up eventually. The producer of MSVCR80.dll may be interested in knowing this. This may be Win32 OS or JVM specific. Appreciate your findings and solution! Dong -Original Message- From: Martin Gainty [mailto:[EMAIL PROTECTED] Sent: Saturday, 9 December 2006 6:18 AM To: Tomcat Users List Subject: Re: Multi processor issue the affinity 'trick' is a replacement dll for the 80 runtime stock libraries The individual most familiar with interfacing to the runtime is mladen turk I would make all attempts to ping him on the solution As a comparison between the 2 dlls Here are the binary characteristics of the stock msvcr80.dll that ships with retail windows packages Section contains the following exports for MSVCR80.dll 0 characteristics 40ECD724 time date stamp Thu Jul 08 01:09:56 2004 0.00 version 1 ordinal base 1367 number of functions 1367 number of names ordinal hint RVA name /*and the replacement msvcr80.dll provided in setaffinity*/ Section contains the following exports for MSVCR80.dll 0 characteristics 4333A455 time date stamp Fri Sep 23 02:44:37 2005 0.00 version 1 ordinal base 1459 number of functions 1459 number of names ordinal hint RVA name so the new msvcr80.dll from setaffinity has 92 new functions which allow it to work reliably with MP calls HTH, Martin-- --- This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. --- Le présent message électronique (y compris les pièces qui y sont annexées, le cas échéant) s'adresse au destinataire indiqué et peut contenir des renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le destinataire de ce document, nous vous signalons qu'il est strictement interdit de le diffuser, de le distribuer ou de le reproduire. But the evidence seems to show two threads are manipulating the same request, otherwise you wouldn't get the error on the ParameterMap. Does a full thread dump show any threads that aren't part of the set started by Tomcat? either that, or a Request is reused before recycle() has been executed completely. We assumed this as well but after instrumenting more our code, we concluded it was not possible because the corruption seemed to occur before the end of the 2 requests processing and so before recycle() was called. I got some logs which sort of showed this but I can't find them right now, I have stopped working on this issue 3 weeks ago. Alas, you could do a test and check whether the method getParameter is actually ever called by two threads during one lifecycle (patching the ParameterMap class, storing last accessing thread, and checking whether another thread is already stored on access) We did it but for some reason the log statements did not get logged in our usual log file. Then after we tried the affinity trick which made the symptom to disappear and got no chance to try again. And Gael, the CPU is the most valuable resource in the production environment, are you really ready to give up your scaleability but simply not investigating further? Well, this application is load balanced on 4 servers and the load is currently pretty low on each of them even when running on a single CPU. We did add more logs to our application and we did also instrument Tomcat but with no luck and as we were unable to reproduce it in a dev environment, it's difficult to ask for short downtimes to restart a server in production. On top of that, my team is being moved to another project... Gael - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Multi processor issue
After the request parameters are parsed, Tomcat makes it's parameter map read-only (i.e. locked). I suspect that this is the source of your problems. Having said that, if you really do have two threads accessing a Request instance, then you really really need to re-design your application ;-). Marziou, Gael [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Hi, As a preamble, let me say that I am not complaining about Tomcat (as the problem root cause is uncertain) and I don't expect much assistance as we are happy with our workaround. However, our experience could help others or someone may have other ideas. We have recently switched a stable web application from JRun 4 to Tomcat 5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08, running some testing for months, but when we moved to production environment we immediately experienced errors IllegalStateException: No modifications are allowed to a locked ParameterMap. Stack trace analysis concluded to RequestFacade objects shared between 2 threads. We made code reviews of our application code and Tomcat's code and did not find any thread safety issues. So, we checked all differences between staging and production, we thought that the major difference was that the staging server was a single CPU machine while the production servers had 2 physical processors and 4 logical ones. So, we constrained the Tomcat service to run on only one processor by setting its processor affinity mask to 1 (run on CPU 0) and after several weeks of monitoring we concluded that it fixed the problem. So, we used a small utility (search for setaffinity.zip) to patch the affinity mask in the Tomcat service executable (tomcat5.exe). Later on, we set up a second production environment also on bi-processor servers but we did not experience the same issue. The hardware is not exactly the same as first production setup and the OS is also different (Windows 2000 on 2nd env, Windows 2003 on 1st env having the issue). So, I don't believe it's a Tomcat, I would rather think to an H/W, or JRE or OS issue. In any case, it's a rather complex issue to investigate and we haven't enough time for this as we are happy with performances running on one single CPU. Thanks, Gael Marziou Hewlett Packard - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Multi processor issue
From: news [mailto:[EMAIL PROTECTED] On Behalf Of Bill Barker
Subject: Re: Multi processor issue
After the request parameters are parsed, Tomcat makes it's
parameter map read-only (i.e. locked).
This is true, but it's not a simple case of someone calling put() after
locking the ParameterMap. Here's the code of interest
(org.apache.catalina.connector.Request):
1015public Map getParameterMap() {
1017if (parameterMap.isLocked())
1018return parameterMap;
1020Enumeration enumeration = getParameterNames();
1021while (enumeration.hasMoreElements()) {
1022String name = enumeration.nextElement().toString();
1023String[] values = getParameterValues(name);
1024parameterMap.put(name, values);
1025}
1027parameterMap.setLocked(true);
1029return parameterMap;
1031}
The stack trace shows the failed call to put() being made from line
1024. In order to get this far, the test for isLocked() at line 1017
must have returned false. In other words, the locked state of the
ParameterMap associated with this request changed between lines 1017 and
1024. Since locked isn't set until line 1027 (and that appears to be
the only place in Tomcat that does so), some thread other than the one
in the stack trace must have changed the state by coming through this
code simultaneously. Restricting Tomcat to a single CPU almost
completely precludes that from ever happening. (But if left running
long enough - possibly years - the situation could still occur with a
single CPU, when one thread's time slice expired between lines 1020 and
1025, and whatever second thread is also erroneously processing the same
Request object starts running. Contrary to what another poster stated,
this is definitely a multi-thread issue, not a multi-processor one.)
If hardware or OS infrastructure were the cause, I would expect that to
manifest itself in more ways than just this one particular situation in
Tomcat. If it were my system, I'd be adding some trap code to mark a
Request object as taken when a thread starts processing it, and verify
that a request isn't already taken when a thread starts processing it.
Unfortunate that the OP's company seems to have lost interest in fixing
what appears to be a bona fide problem in their application.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
