Re: Multi processor issue

2007-01-17 Thread Gael Marziou

Thanks Mark.

Gael


Mark Thomas wrote:
  
In 5.5.21 onwards, the RD will be thread safe so the OP's code should

work without modification.

Mark

  


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-30 Thread Mark Thomas
Mark Thomas wrote:
> I plan to look at modifying the current RD code over the holiday.

In 5.5.21 onwards, the RD will be thread safe so the OP's code should
work without modification.

Mark

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-22 Thread Christopher Schultz
Gael,

Marziou, Gael wrote:
>> Can you suggest a fix? I'm not sure how this kind of thing 
>> could be safely veto'd... for instance, it might actually be 
>> appropriate for a RequestDispatcher to be re-used in some 
>> context (say, repeating a request twice) or even to do so 
>> with different threads, as long as the access is done 
>> strictly serially.
> 
> Do you mean submitting an enhancement request with some proposed code?

Well, I'm sure the the developers would appreciate that, but meant just
a suggestion of how it might be done -- just in words, not necessarily
any code.

Often, problems like this only affect a small number of people, and
"fixing" them would break other people's code -- there's always someone
who found a way to use it in an unexpected way.

-chris



signature.asc
Description: OpenPGP digital signature


Re: Multi processor issue

2006-12-22 Thread Mark Thomas
Caldarale, Charles R wrote:
> Since query information is normally unique to a request, I think the
> above could be construed that the RequestDispatcher is cognizant of the
> query string.  If so, each RequestDispatcher would be associated with a
> specific request.

I agree that the RD is cognizant of the query string but I don't see
that this prevents the RD from being re-used. The RD merges its query
string (specified on creation) with the query string of the request
passed to the include/forward method. No reason it can't do this
several times.

> SRV.15.2.8 ServletContext, getRequestDispatcher(String):
> "The resource can be dynamic or static."
> 
> Since dynamic resources can come and go and their existence can be
> dependent on specific requests, there is an implication that a
> RequestDispatcher is associated with the request it has been created
> for.

I see where you are coming form here but this strikes me as an
application issue.

> "Send lawyers, guns, and money."  (That's not in the spec, but perhaps
> should be.)

He he. Indeed!

What does it for me is:
a) that you can obtain an RD from the context;
b) SRV.8.1 that states:
"The method uses the path to look up a servlet ... wraps it with a
RequestDispatcher object, and returns the resulting object."

In my mind this aligns RDs more with servlets than requests.

I plan to look at modifying the current RD code over the holiday.

Mark

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-22 Thread Marziou, Gael

> Can you suggest a fix? I'm not sure how this kind of thing 
> could be safely veto'd... for instance, it might actually be 
> appropriate for a RequestDispatcher to be re-used in some 
> context (say, repeating a request twice) or even to do so 
> with different threads, as long as the access is done 
> strictly serially.

Do you mean submittingan enhancement request with some proposed code?

> Fantastic! I'm glad we were able to help, even if it did take a while.

Yes and I want to thank all the people from the mailing list for their
good advices and also for giving me additional motivation to investigate
further.

Gael 

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-21 Thread Christopher Schultz
Gael,

Marziou, Gael wrote:
>> I don't think this is a bug in TC's implementation. This is a 
>> relatively subtle mistake that seems both easy enough to make 
>> and easy to fix.
> 
> Maybe Tomcat should enforce this like for instance when we get an
> exception trying to write into a response that was already committed.
> It would have saved weeks of effort on our side.

Can you suggest a fix? I'm not sure how this kind of thing could be
safely veto'd... for instance, it might actually be appropriate for a
RequestDispatcher to be re-used in some context (say, repeating a
request twice) or even to do so with different threads, as long as the
access is done strictly serially.

> We did implement this 2 years ago, so details have vanished but it was
> mainly based on our spec interpretation and also I admit on premature
> optimization.

No problem. I'm pretty sure we're all guilty of that some of the time.

> In several places, the spec defines different behaviors of
> RequestDispatcher when it was obtained using getNamedDispatcher(String)
> and noticeably for handling parameters.
> So, our interpretation was that a RequestDispatcher obtained using
> getNamedDispatcher() was different than one obtained using
> getRequestDispatcher(): it has a different behavior and a wider scope.

Agreed... the "spirit" of the "named dispatcher" as opposed to a
standard dispatcher obtained for a certain URI seems to be that it could
be used universally.

> This was the reason behind our implementation and also the fact that our
> previous container (before migrating to Tomcat) seemed to work this way
> (as far as I remember).

Perhaps TC should consider returning an object of a different type when
using getNamedDispatcher -- one that does not keep any request-related
information at all.

>> I'd really like to hear if simply eliminating the caching of 
>> this object fixes your problem. Does it?
> 
> So far so good, the new code has been running fine for 2 hours while it
> was failing usually within 30 minutes.

Fantastic! I'm glad we were able to help, even if it did take a while.

-chris



signature.asc
Description: OpenPGP digital signature


RE: Multi processor issue

2006-12-21 Thread Marziou, Gael
> I don't think this is a bug in TC's implementation. This is a 
> relatively subtle mistake that seems both easy enough to make 
> and easy to fix.

Maybe Tomcat should enforce this like for instance when we get an
exception trying to write into a response that was already committed.
It would have saved weeks of effort on our side.

> Gael, what was the reasoning behind caching that 
> RequestDispatcher? Was it simply because you would have been 
> calling the same method with the same parameters repeatedly, 
> and figured that you'd get some performance edge by saving 
> the resulting object? Were you observing any performance 
> degradation at any point? Or was this a case of premature 
> optimization? ;)

We did implement this 2 years ago, so details have vanished but it was
mainly based on our spec interpretation and also I admit on premature
optimization.

In several places, the spec defines different behaviors of
RequestDispatcher when it was obtained using getNamedDispatcher(String)
and noticeably for handling parameters.
So, our interpretation was that a RequestDispatcher obtained using
getNamedDispatcher() was different than one obtained using
getRequestDispatcher(): it has a different behavior and a wider scope.
This was the reason behind our implementation and also the fact that our
previous container (before migrating to Tomcat) seemed to work this way
(as far as I remember).

> I'd really like to hear if simply eliminating the caching of 
> this object fixes your problem. Does it?

So far so good, the new code has been running fine for 2 hours while it
was failing usually within 30 minutes.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-21 Thread Christopher Schultz
Chuck,

Caldarale, Charles R wrote:
> "Parameters specified in the query string used to create the
> RequestDispatcher take precedence over other parameters of the same name
> passed to the included servlet. The parameters associated with a
> RequestDispatcher are scoped to apply only for the duration of the
> include or forward call."

That part right there pretty much closes the book on this one for me: if
you pass (URL) parameters into the methods that return RequestDispatcher
objects, then you get a customized RequestDispatcher. There's no reason
to think that "no parameters" is any different than "some parameters".

I don't think this is a bug in TC's implementation. This is a relatively
subtle mistake that seems both easy enough to make and easy to fix.

Gael, what was the reasoning behind caching that RequestDispatcher? Was
it simply because you would have been calling the same method with the
same parameters repeatedly, and figured that you'd get some performance
edge by saving the resulting object? Were you observing any performance
degradation at any point? Or was this a case of premature optimization? ;)

I'd really like to hear if simply eliminating the caching of this object
fixes your problem. Does it?

-chris




signature.asc
Description: OpenPGP digital signature


RE: Multi processor issue

2006-12-21 Thread Caldarale, Charles R
> From: Mark Thomas [mailto:[EMAIL PROTECTED] 
> Subject: Re: Multi processor issue
> 
> I have re-read the sections of the spec you quoted several times and I
> don't see anything that suggests to me that the RequestDispatcher has
> only request level scope that would suggest the OPs use is invalid.
> Could you elaborate?

As I said yesterday, the statements in the spec are hints and by no
means definitive.  They are certainly subject to interpretation.  The
quoted items are from the spec, my comments follow each.

SRV.8.1.1 Query Strings in Request Dispatcher Paths:
"The ServletContext and ServletRequest methods that create
RequestDispatcher objects using path information allow the optional
attachment of query string information to the path."

"Parameters specified in the query string used to create the
RequestDispatcher take precedence over other parameters of the same name
passed to the included servlet. The parameters associated with a
RequestDispatcher are scoped to apply only for the duration of the
include or forward call."

Since query information is normally unique to a request, I think the
above could be construed that the RequestDispatcher is cognizant of the
query string.  If so, each RequestDispatcher would be associated with a
specific request.

SRV.8.4.1 Query String:
"The request dispatching mechanism is responsible for aggregating query
string parameters when forwarding or including requests."

Again, I think the above can be viewed as tying a RequestDispatcher
object to a specific request.

SRV.15.2.8 ServletContext, getRequestDispatcher(String):
"The resource can be dynamic or static."

Since dynamic resources can come and go and their existence can be
dependent on specific requests, there is an implication that a
RequestDispatcher is associated with the request it has been created
for.

"Send lawyers, guns, and money."  (That's not in the spec, but perhaps
should be.)

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-21 Thread Marziou, Gael
> Mark Thomas said:
> 
> I have re-read the sections of the spec you quoted several 
> times and I don't see anything that suggests to me that the 
> RequestDispatcher has only request level scope that would 
> suggest the OPs use is invalid.

That's the key point, also I am under the impression that the
ApplicationDispatcher could be re-factored to be thread safe by avoiding
having request scope fields and using only local variables in forward()
method.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-21 Thread Mark Thomas
Caldarale, Charles R wrote:
>> From: Mark Thomas [mailto:[EMAIL PROTECTED] 
>> Subject: Re: Multi processor issue
>>
>> What about RequestDispatcher ServletContext.getNamedDispatcher(String)
>>
>> Since this method is at the Context level, any RequestDispatcher
>> obtained effectively has context wide scope. It is this method that
>> the OP is using and, given the non-threadsafe nature of
>> o.a.c.core.ApplicationDispatcher, this looks like a Tomcat bug to me.
>> I haven't had a chance to see how hard it might be to fix yet.
> 
> I think some Servlet spec clarification would be really helpful here.
> Unless the spec intends for there to be different flavors of
> RequestDispatcher, the apparent context-level scope seems to conflict
> with the request-specific comments I noted earlier.

I have re-read the sections of the spec you quoted several times and I
don't see anything that suggests to me that the RequestDispatcher has
only request level scope that would suggest the OPs use is invalid.
Could you elaborate?

Mark

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-20 Thread Caldarale, Charles R
> From: Mark Thomas [mailto:[EMAIL PROTECTED] 
> Subject: Re: Multi processor issue
> 
> What about RequestDispatcher ServletContext.getNamedDispatcher(String)
> 
> Since this method is at the Context level, any RequestDispatcher
> obtained effectively has context wide scope. It is this method that
> the OP is using and, given the non-threadsafe nature of
> o.a.c.core.ApplicationDispatcher, this looks like a Tomcat bug to me.
> I haven't had a chance to see how hard it might be to fix yet.

I think some Servlet spec clarification would be really helpful here.
Unless the spec intends for there to be different flavors of
RequestDispatcher, the apparent context-level scope seems to conflict
with the request-specific comments I noted earlier.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-20 Thread Mark Thomas
Caldarale, Charles R wrote:
>> From: Leon Rosenberg [mailto:[EMAIL PROTECTED] 
>> Subject: Re: Multi processor issue
>>
>> "The Container Provider should ensure that the dispatch of the 
>> request to a target servlet occurs in the same thread of the 
>> same JVM as the original request."
> 
> I'm not sure the above is really pertinent to the issue, which I think
> can be boiled down to this:
> 
> Should a RequestDispatcher object be treated as if it were in the scope
> of a Request or instead in the scope of a Servlet?  The spec provides no
> definitive statement that I can find, but there are a couple of hints
> that a RequestDispatcher should be considered unique to each request.
> In particular:

What about RequestDispatcher ServletContext.getNamedDispatcher(String)

Since this method is at the Context level, any RequestDispatcher
obtained effectively has context wide scope. It is this method that
the OP is using and, given the non-threadsafe nature of
o.a.c.core.ApplicationDispatcher, this looks like a Tomcat bug to me.
I haven't had a chance to see how hard it might be to fix yet.

Mark

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-20 Thread Mark Thomas
Caldarale, Charles R wrote:
>> From: Mark Thomas [mailto:[EMAIL PROTECTED] 
>> Subject: Re: Multi processor issue
>>
>> This requirement will be enforceable in 5.5.21 onwards. It was
>> implemented in response to
>> http://issues.apache.org/bugzilla/show_bug.cgi?id=34956
> 
> Does the change in 5.5.21 address the OP's real problem?  (I.e., his app
> cached and reused the same RequestDispatcher object for multiple
> concurrent requests.)  It's difficult for me to see how Tomcat could
> protect against that abuse without significant changes to the guts of
> ApplicationDispatcher.

Ignore my last post. I was reading too fast and looked at the wrong
paragraph.

Mark


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-20 Thread Caldarale, Charles R
> From: Mark Thomas [mailto:[EMAIL PROTECTED] 
> Subject: Re: Multi processor issue
> 
> This requirement will be enforceable in 5.5.21 onwards. It was
> implemented in response to
> http://issues.apache.org/bugzilla/show_bug.cgi?id=34956

Does the change in 5.5.21 address the OP's real problem?  (I.e., his app
cached and reused the same RequestDispatcher object for multiple
concurrent requests.)  It's difficult for me to see how Tomcat could
protect against that abuse without significant changes to the guts of
ApplicationDispatcher.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-20 Thread Mark Thomas
Leon Rosenberg wrote:
> BINGO :-)
> 
> Please check the servlet spec 2.5, SRV 8.2
>

> The Container Provider should ensure that the dispatch of the request to a
> target servlet occurs in the same thread of the same JVM as the
> original request."
> 
> Please note that the last paragraph. I assume if the servlet container
> works properly, your method simply can't work. Can you remove
> Dispatcher caching and check whether the issue is gone?

This requirement will be enforceable in 5.5.21 onwards. It was
implemented in response to
http://issues.apache.org/bugzilla/show_bug.cgi?id=34956


Mark

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-20 Thread Caldarale, Charles R
> From: Leon Rosenberg [mailto:[EMAIL PROTECTED] 
> Subject: Re: Multi processor issue
> 
> "The Container Provider should ensure that the dispatch of the 
> request to a target servlet occurs in the same thread of the 
> same JVM as the original request."

I'm not sure the above is really pertinent to the issue, which I think
can be boiled down to this:

Should a RequestDispatcher object be treated as if it were in the scope
of a Request or instead in the scope of a Servlet?  The spec provides no
definitive statement that I can find, but there are a couple of hints
that a RequestDispatcher should be considered unique to each request.
In particular:

SRV.8.1.1 Query Strings in Request Dispatcher Paths:
"The ServletContext and ServletRequest methods that create
RequestDispatcher objects using path information allow the optional
attachment of query string information to the path."

"Parameters specified in the query string used to create the
RequestDispatcher take precedence over other parameters of the same name
passed to the included servlet. The parameters associated with a
RequestDispatcher are scoped to apply only for the duration of the
include or forward call."

SRV.8.4.1 Query String:
"The request dispatching mechanism is responsible for aggregating query
string parameters when forwarding or including requests."

SRV.15.2.8 ServletContext, getRequestDispatcher(String):
"The resource can be dynamic or static."

Based on the above, I think the app's reuse of RequestDispatcher objects
is indeed erroneous.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-20 Thread Leon Rosenberg

BINGO :-)

Please check the servlet spec 2.5, SRV 8.2

"To use a request dispatcher, a servlet calls either the include
method or forward
method of the RequestDispatcher interface. The parameters to these methods can
be either the request and response arguments that were passed in via the service
method of the javax.servlet interface, or instances of subclasses of
the request an
response wrapper classes that were introduced for version 2.3 of the
specification. In the latter case, the wrapper instances must wrap the
request or response objects that the container passed into the service
method.
The Container Provider should ensure that the dispatch of the request to a
target servlet occurs in the same thread of the same JVM as the
original request."

Please note that the last paragraph. I assume if the servlet container
works properly, your method simply can't work. Can you remove
Dispatcher caching and check whether the issue is gone?

regards
Leon


On 12/20/06, Marziou, Gael <[EMAIL PROTECTED]> wrote:

> By looking at the code of ApplicationDispatcher, I would tend to think
that
> the answer is no because of fields like wrapRequest and requestURI
(there
> are other fields but they are unused) however the javadoc does not
mention it.

I forgot to include the question but some of you may have guessed it ;-)
The question was: can we call forward() on same ApplicationDispatcher
object from 2 concurrent threads?

Gael




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-20 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Gael,

Marziou, Gael wrote:
> So, we see that the problems occurs between
> ProxySessionController.forwardRequest() and the first line of
> PassThroughController.doHttp(). In fact we can go down to
> PassThroughController.doGet() as it is just directly calling doHttp()
> without any change.
> 
> At this point, I must explain what does
> ProxySessionController.forwardRequest(): it does select a
> RequestDispatcher (a servlet) based on the request path info and then
> calls forward() on it.

Is it possible to post the code for this method, or even the entire
class? I realize that it might be considered sensitive intellectual
property, so I understand if that's not possible. If completely public
distribution is not possible, but more limited distribution /is/
possible, you might consider sending that code directly to the folks who
have been involved in this thread (and if they request it). For
instance, I would be glad to look at your code.

> An important point here is that we re-use the same RequestDispatcher got
> once by ServletContext.getNamedDispatcher() over and over.
> In Tomcat, the class implementing RequestDispatcher is
> org.apache.catalina.core.ApplicationDispatcher
> 
> So, it means that the 2 concurrent threads will call forward() on same
> ApplicationDispatcher object.
> 
> By looking at the code of ApplicationDispatcher, I would tend to think
> that the answer is no because of fields like wrapRequest and requestURI
> (there are other fields but they are unused) however the javadoc does
> not mention it.
> 
> Looking at org.apache.catalina.core.ApplicationContextFacade and
> org.apache.catalina.core.ApplicationContext, it seems to me that
> getNamedDispatcher() always return a fresh RequestDispatcher object, so
> it seems that it is intended to be used only once per request.

If you call getNamedDispatcher each time, does your problem go away? I
would imagine that if the same dispatcher was being used all the time
(and that was a problem), that you would get errors all over the place...

- -chris

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFiW+A9CaO5/Lv0PARAtnAAKC8ACrPTRD+RXAw8rfG02ol6lxA7QCePCQO
Wvv/f+ERkGd/z+hzaHLi714=
=+Rkg
-END PGP SIGNATURE-

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-20 Thread Marziou, Gael
> By looking at the code of ApplicationDispatcher, I would tend to think
that 
> the answer is no because of fields like wrapRequest and requestURI
(there
> are other fields but they are unused) however the javadoc does not
mention it.

I forgot to include the question but some of you may have guessed it ;-)
The question was: can we call forward() on same ApplicationDispatcher
object from 2 concurrent threads?

Gael 




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-20 Thread Marziou, Gael
Some fresh news and hopefully I have narrowed the scope enough to get
the root cause.

First, the previous traces that I posted last week were wrong because my
bug catcher was buggy: I forgot to reset the fields in recycle().

So, I fixed this and saved few other fields in
org.apache.catalina.connector.Request in getParameterMap(), we got
another trace and found that we needed also to log the association of
thread and request at different steps in our application in methods that
were found in the stack traces.

So, we ran again the bug catcher and this time, it failed with the 3rd
symptom which did not involve getParameterMap() but still was due to a
request object being shared between 2 http threads.

Hopefully, the mailing list supports attachments as I have attached the
log file.
I edited the log file to remove statements from other sessions using
other threads and also user data (e.g. email address).

So, here is my analysis of what happened.

Line 1: We get a request for "red.gif" on thread http-80-5

Line 2: We get a request for "Green.gif" on thread http-80-4

Then you can follow up different threads either by using thread name or
by using log4j NDC which consists in jsessionid/request# (e.g.
D1D4F7EE6827BC3B569FE2F3BE3C0F15/4).

Line 9 and 10: We can see that each thread has still its own
RequestFacade object with a different URI still consistent with lines 1
and 2. The log statement is done at ProxySessionController.java:52

Line 11 and 12: here is the problem the 2 threads share the same
ApplicationHttpRequest object and same request URI.

So now let's take a look at the stack trace by concatenating the stack
of the cause and its nesting exception.


67  at
com.hp.swift.proxy.passthrough.PassThroughController.doHttp(PassThroughC
ontroller.java:100)
68  at
com.hp.swift.proxy.passthrough.PassThroughController.doGet(PassThroughCo
ntroller.java:77)
69  at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
70  at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
71  at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:252) 

18  at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:275)
19  at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
20  at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
her.java:672)
21  at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio
nDispatcher.java:463)
22  at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
atcher.java:359)
23  at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
cher.java:301)
24  at
com.hp.swift.proxy.ProxySessionController.forwardRequest(ProxySessionCon
troller.java:60)


So, we see that the problems occurs between
ProxySessionController.forwardRequest() and the first line of
PassThroughController.doHttp(). In fact we can go down to
PassThroughController.doGet() as it is just directly calling doHttp()
without any change.

At this point, I must explain what does
ProxySessionController.forwardRequest(): it does select a
RequestDispatcher (a servlet) based on the request path info and then
calls forward() on it.
An important point here is that we re-use the same RequestDispatcher got
once by ServletContext.getNamedDispatcher() over and over.
In Tomcat, the class implementing RequestDispatcher is
org.apache.catalina.core.ApplicationDispatcher

So, it means that the 2 concurrent threads will call forward() on same
ApplicationDispatcher object.

By looking at the code of ApplicationDispatcher, I would tend to think
that the answer is no because of fields like wrapRequest and requestURI
(there are other fields but they are unused) however the javadoc does
not mention it.

Looking at org.apache.catalina.core.ApplicationContextFacade and
org.apache.catalina.core.ApplicationContext, it seems to me that
getNamedDispatcher() always return a fresh RequestDispatcher object, so
it seems that it is intended to be used only once per request.

Thanks for your help,

Gael 



 1  2006-12-20 07:23:04,257: INFO  
com.hp.swift.portal.servlet.HttpTracingFilter - Starting GET 
/radix/bin/infoboard/images/red.gif HTTP/1.1 {http-80-5 | [EMAIL PROTECTED] 
 2  2006-12-20 07:23:04,257: INFO  
com.hp.swift.portal.servlet.HttpTracingFilter - Starting GET 
/radix/bin/infoboard/images/Green.gif HTTP/1.1 {http-80-4 | [EMAIL PROTECTED] 
 3  2006-12-20 07:23:04,257: INFO  com.hp.swift.portal.servlet.request - 
(D1D4F7EE6827BC3B569FE2F3BE3C0F15/4) Starting GET 
/radix/bin/infoboard/images/red.gif HTTP/1.1
 4  2006-12-20 07:23:04,257: INFO  com.hp.swift.portal.servlet.request - 
(D1D4F7EE6827BC3B569FE2F3BE3C0F15/5) Starting GET 
/radix/bin/infoboard/images/Green.gif HTTP/1.1
 5  2006-12-20 07:23:04,257: INFO  log.appli - 
(D1D4F7EE6827BC3B569FE2F3BE3C0F15/4) SessionCont

RE: Multi processor issue

2006-12-13 Thread Peter Crowther
> From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] 
> It's not really, IF you take time to read the specs - but few do.
> Admittedly, there are a lot of things that should be studied, and
> project deadlines rarely take that into account.

And budgets, and hence project staffing (and skill) levels.  "What can
we get away with paying?" is entirely too common a question.  The answer
of "If you pay less now in development, you'll pay more later due to
poor quality" is, unfortunately, not what the sales team or the client
want to hear.

- Peter

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-13 Thread JiaDong Huang
Great analysis and elaboration! Although I am not the one that wants
everything "synchronize always". What I really expect is to get the thing
done to the best possible. MT issue fix is always various, and not
simple/easy to get to the best.

We should appreciate Chuck's knowledge of the Request class and the
ParameterMap class - that has guided us to do the right thing at the app
level. Fixing the MT issue at the low level may not always be a good idea.

I think we have done a very hard enough twist. We can pretty much close this
case for Tomcat now - Tomcat has done the right thing throwing the exception
effectively if the app does not synchronize thing properly, or storing thing
in the wrong place. Tomcat should not have any further fix/synchronization
for such a buggy case.

Hope Gael can fix his app happily...

Dong

-Original Message-
From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, 13 December 2006 3:36 PM
To: Tomcat Users List
Subject: RE: Multi processor issue

> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> What I have been saying is that we should reconsider very 
> carefully whether there should be any enhancement in Tomcat.
> From the Tomcat code you dug out, there is something not
> quite perfect, in term of MT safe while initializing.

Absolutely wrong.  There is nothing incorrect about the code in either
the Request class or the ParameterMap class; it fulfills every aspect of
its specification.  Changing the default behavior of the Request class
to be synchronized on its get/set methods would be completely inane.

> One of your posted messages did state that it is valid for 
> two threads to access/operate on the ParameterMap once it
> has been established.

Yes - with a big caveat.  Any application doing so must provide its own
synchronization to insure the integrity of the Request object (and
embedded ParameterMap).  In this case, the application has NO intention
of using two threads on one Request - it simply has a bug in it.

If we were to apply your "synchronize always" approach to software
development, we'd need a lot better performance out of the hardware than
Moore's law expects in order to keep up.  Nor would it solve very many
problems, including this one.

Think about it - what would have happened if the Request object did have
synchronized methods?  The underlying flaw in the app that is causing
one thread to lose track of its own Request and latch onto another would
have gone undetected - except for some client somewhere not getting the
expected response, but instead seeing someone else's output show up.
Simple synchronization does not address the real problem here.

> So, it may be pretty hard and confusing what the app developer
> should be doing at the upper level.

It's not really, IF you take time to read the specs - but few do.
Admittedly, there are a lot of things that should be studied, and
project deadlines rarely take that into account.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Caldarale, Charles R
> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> What I have been saying is that we should reconsider very 
> carefully whether there should be any enhancement in Tomcat.
> From the Tomcat code you dug out, there is something not
> quite perfect, in term of MT safe while initializing.

Absolutely wrong.  There is nothing incorrect about the code in either
the Request class or the ParameterMap class; it fulfills every aspect of
its specification.  Changing the default behavior of the Request class
to be synchronized on its get/set methods would be completely inane.

> One of your posted messages did state that it is valid for 
> two threads to access/operate on the ParameterMap once it
> has been established.

Yes - with a big caveat.  Any application doing so must provide its own
synchronization to insure the integrity of the Request object (and
embedded ParameterMap).  In this case, the application has NO intention
of using two threads on one Request - it simply has a bug in it.

If we were to apply your "synchronize always" approach to software
development, we'd need a lot better performance out of the hardware than
Moore's law expects in order to keep up.  Nor would it solve very many
problems, including this one.

Think about it - what would have happened if the Request object did have
synchronized methods?  The underlying flaw in the app that is causing
one thread to lose track of its own Request and latch onto another would
have gone undetected - except for some client somewhere not getting the
expected response, but instead seeing someone else's output show up.
Simple synchronization does not address the real problem here.

> So, it may be pretty hard and confusing what the app developer
> should be doing at the upper level.

It's not really, IF you take time to read the specs - but few do.
Admittedly, there are a lot of things that should be studied, and
project deadlines rarely take that into account.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread JiaDong Huang
Chuck,

I am very clear about the whole thing. Please try to understand my point
properly too.

Your point can be right and valid and would be very significant to help
Gael. Gael should be able to get to the point where his app might do wrong,
or miss leading Tomcat threads.

What I have been saying is that we should reconsider very carefully whether
there should be any enhancement in Tomcat. From the Tomcat code you dug out,
there is something not quite perfect, in term of MT safe while initializing.
No argument.

One of your posted messages did state that it is valid for two threads to
access/operate on the ParameterMap once it has been established. So, it may
be pretty hard and confusing what the app developer should be doing at the
upper level. Whether it is appropriate to enhance Tomcat or not, and when, I
don't think I am in the right position to justify. I guess once Gael has
clarified his issue. He may be in a better position to make significant
comments, and see if you can agree.

Dong

-Original Message-
From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, 13 December 2006 11:34 AM
To: Tomcat Users List
Subject: RE: Multi processor issue

> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> Not sure if the app really cares which thread should do the 
> parameterMap initialization.

You're missing the entire point of this exercise: no Request object
should EVER be processed by more than one thread.  No synchronization
changes are needed within Tomcat.  The most likely cause of the problem
is code in the app itself exposing a Request object at the session or
context level (e.g., storing it as a session attribute), thereby causing
a thread working on a second, separate request to grab the wrong Request
object.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-12 Thread Martin Gainty
Jiadong/Gael

Can you download the code to a sandbox and implement your synchronized 
algorithm in setLocked to test out the intended result?
Or is the code not available?

M-
--- 
This e-mail message (including attachments, if any) is intended for the use of 
the individual or entity to which it is addressed and may contain information 
that is privileged, proprietary , confidential and exempt from disclosure. If 
you are not the intended recipient, you are notified that any dissemination, 
distribution or copying of this communication is strictly prohibited.
--- 
Le présent message électronique (y compris les pièces qui y sont annexées, le 
cas échéant) s'adresse au destinataire indiqué et peut contenir des 
renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le 
destinataire de ce document, nous vous signalons qu'il est strictement interdit 
de le diffuser, de le distribuer ou de le reproduire.
- Original Message - 
From: "JiaDong Huang" <[EMAIL PROTECTED]>
To: "'Tomcat Users List'" 
Sent: Tuesday, December 12, 2006 7:24 PM
Subject: RE: Multi processor issue


> Great to see the bottom of the MT issue has been revealed/understood. I
> believe Gael may be able to fix his app now. Hope it can be an easy fix...
> 
> Not sure if the app really cares which thread should do the parameterMap
> initialization. If it does not, then it would be a good idea for Tomcat to
> refine/enhance the parameterMap initialization procedure to be MT safe.
> 
> Apparently, the current lock facility (isLocked() and setLocked()) does not
> protect MT case properly, while initializing.
> 
> Is it worthwhile and appropriate to have certain MT synchronization facility
> like isInitialized() and setInitialized() around the code? In theory, the
> second thread can wait for isInitialized() there. Not sure if it will cause
> any concern to other apps, like having other racing condition. Really depend
> on the parameterMap usage/operation. MT issue fix can never be simple.
> 
> Chuck understands parameterMap usage very well. Maybe he can make further
> comments/recommendation to the Tomcat dev team.
> 
> Dong
> 
> -Original Message-
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Sent: Wednesday, 13 December 2006 3:24 AM
> To: Tomcat Users List
> Subject: RE: Multi processor issue
> 
> 
>> Anything dealing with a session must be careful not to expose a
>> Request object to the session scope, since there's the risk another
>> request for the same session may get confused with the first - I
>> suspect that's what's happening here. 
> 
> Will do.
> 
>> You should take a careful look at all the com.hp code in the stack
>> trace (not just the above three places) and verify that Request
>> objects are not getting mixed up.  Might want to add some trace code
>> in various places to insure that a given thread is procesing the same
>> Request object at all points. 
> 
> Yes, it makes sense.
> 
>> Could also try turning off APR (remove or rename tcnative-1.dll) and
>> see if that has an effect. 
> 
> I already did at the beginning of our investigation as I suspected the
> JNI code but it had no effect.
> 
> Gael
> 
> -
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 
> 
> 
> -
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
> 
>

RE: Multi processor issue

2006-12-12 Thread Caldarale, Charles R
> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> Not sure if the app really cares which thread should do the 
> parameterMap initialization.

You're missing the entire point of this exercise: no Request object
should EVER be processed by more than one thread.  No synchronization
changes are needed within Tomcat.  The most likely cause of the problem
is code in the app itself exposing a Request object at the session or
context level (e.g., storing it as a session attribute), thereby causing
a thread working on a second, separate request to grab the wrong Request
object.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread JiaDong Huang
Great to see the bottom of the MT issue has been revealed/understood. I
believe Gael may be able to fix his app now. Hope it can be an easy fix...

Not sure if the app really cares which thread should do the parameterMap
initialization. If it does not, then it would be a good idea for Tomcat to
refine/enhance the parameterMap initialization procedure to be MT safe.

Apparently, the current lock facility (isLocked() and setLocked()) does not
protect MT case properly, while initializing.

Is it worthwhile and appropriate to have certain MT synchronization facility
like isInitialized() and setInitialized() around the code? In theory, the
second thread can wait for isInitialized() there. Not sure if it will cause
any concern to other apps, like having other racing condition. Really depend
on the parameterMap usage/operation. MT issue fix can never be simple.

Chuck understands parameterMap usage very well. Maybe he can make further
comments/recommendation to the Tomcat dev team.

Dong

-Original Message-
From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, 13 December 2006 3:24 AM
To: Tomcat Users List
Subject: RE: Multi processor issue


> Anything dealing with a session must be careful not to expose a
> Request object to the session scope, since there's the risk another
> request for the same session may get confused with the first - I
> suspect that's what's happening here. 

Will do.

> You should take a careful look at all the com.hp code in the stack
> trace (not just the above three places) and verify that Request
> objects are not getting mixed up.  Might want to add some trace code
> in various places to insure that a given thread is procesing the same
> Request object at all points. 

Yes, it makes sense.

> Could also try turning off APR (remove or rename tcnative-1.dll) and
> see if that has an effect. 

I already did at the beginning of our investigation as I suspected the
JNI code but it had no effect.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> I already did at the beginning of our investigation as I suspected the
> JNI code but it had no effect.

That's what I thought, but wasn't sure, since at least one of the stack
traces you sent (8 December) had no APR entries in it, but this one did.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Marziou, Gael

> Anything dealing with a session must be careful not to expose a
> Request object to the session scope, since there's the risk another
> request for the same session may get confused with the first - I
> suspect that's what's happening here. 

Will do.

> You should take a careful look at all the com.hp code in the stack
> trace (not just the above three places) and verify that Request
> objects are not getting mixed up.  Might want to add some trace code
> in various places to insure that a given thread is procesing the same
> Request object at all points. 

Yes, it makes sense.

> Could also try turning off APR (remove or rename tcnative-1.dll) and
> see if that has an effect. 

I already did at the beginning of our investigation as I suspected the
JNI code but it had no effect.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> Correct me if I am wrong but it seems to me that the 2 
> threads are from Tomcat, not from our application.

They are, but they go through a lot of your application code prior to
reaching getParameterMap().

I'm especially suspicious of this area:

com.hp.swift.portal.servlet.SessionController.forwardRequestAs()(Session
Controller.java:159)
com.hp.swift.portal.servlet.SessionController.doHttp()(SessionController
.java:134)
com.hp.swift.portal.servlet.SessionController.doGet()(SessionController.
java:110)

Anything dealing with a session must be careful not to expose a Request
object to the session scope, since there's the risk another request for
the same session may get confused with the first - I suspect that's
what's happening here.

You should take a careful look at all the com.hp code in the stack trace
(not just the above three places) and verify that Request objects are
not getting mixed up.  Might want to add some trace code in various
places to insure that a given thread is procesing the same Request
object at all points.

Could also try turning off APR (remove or rename tcnative-1.dll) and see
if that has an effect.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Marziou, Gael
I used sleep() as you can see in the modified version of
getParameterMap() at the end of this message.

I got the issue and here is the trace.

What I call "original thread" is the one that was the first to call
Requuest.getParameterMap()

Correct me if I am wrong but it seems to me that the 2 threads are from
Tomcat, not from our application.

Gael


java.lang.IllegalStateException: 

Original thread = Thread[http-80-3,5,main]
Original stack trace =  java.lang.Thread.dumpThreads()
java.lang.Thread.getStackTrace()

org.apache.catalina.connector.Request.getParameterMap()(Request.java:102
3)

org.apache.catalina.connector.RequestFacade.getParameterMap()(RequestFac
ade.java:411)

org.apache.catalina.core.ApplicationHttpRequest.parseParameters()(Applic
ationHttpRequest.java:727)

org.apache.catalina.core.ApplicationHttpRequest.getParameter()(Applicati
onHttpRequest.java:352)

com.hp.swift.portal.state.HttpEventContextObject.getParameter()(HttpEven
tContextObject.java:64)

com.hp.cii.radix.user.web.home.Home2Context.doUpdateAlertSummary()(Home2
Context.java:709)

com.hp.cii.radix.user.web.home.Home2Context.doDisplayAlertSummary()(Home
2Context.java:789)
sun.reflect.NativeMethodAccessorImpl.invoke0()
sun.reflect.NativeMethodAccessorImpl.invoke()
sun.reflect.DelegatingMethodAccessorImpl.invoke()
java.lang.reflect.Method.invoke()
com.hp.swift.util.MethodInvoker.invoke()(MethodInvoker.java:52)

com.hp.swift.state.core.StateContextAction.execute()(StateContextAction.
java:51)

com.hp.swift.state.core.Transition.executeAction()(Transition.java:124)

com.hp.swift.state.core.StateMachineController.fireTransition()(StateMac
hineController.java:1013)

com.hp.swift.state.core.StateMachineController.doHandle()(StateMachineCo
ntroller.java:788)

com.hp.swift.state.core.StateMachineController.handle()(StateMachineCont
roller.java:520)

com.hp.swift.portal.servlet.SmvController.doHandleRequest()(SmvControlle
r.java:393)

com.hp.swift.portal.servlet.SmvController.doGet()(SmvController.java:235
)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:802)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:252)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

org.apache.catalina.core.ApplicationDispatcher.invoke()(ApplicationDispa
tcher.java:672)

org.apache.catalina.core.ApplicationDispatcher.processRequest()(Applicat
ionDispatcher.java:463)

org.apache.catalina.core.ApplicationDispatcher.doForward()(ApplicationDi
spatcher.java:359)

org.apache.catalina.core.ApplicationDispatcher.forward()(ApplicationDisp
atcher.java:301)

com.hp.swift.proxy.ProxySessionController.forwardRequest()(ProxySessionC
ontroller.java:57)

com.hp.swift.portal.servlet.SessionController$1.run()(SessionController.
java:164)

com.hp.swift.security.manager.LightSecurityManager.doAsPrivileged()(Ligh
tSecurityManager.java:272)

com.hp.swift.security.manager.SecurityChecker.doAsPrivileged()(SecurityC
hecker.java:187)

com.hp.swift.portal.servlet.SessionController.forwardRequestAs()(Session
Controller.java:159)

com.hp.swift.portal.servlet.SessionController.doHttp()(SessionController
.java:134)

com.hp.swift.portal.servlet.SessionController.doGet()(SessionController.
java:110)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:689)

com.hp.swift.portal.init.FirstLevelInitableServlet.service()(FirstLevelI
nitableServlet.java:154)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:802)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:252)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

com.hp.swift.portal.servlet.HttpTracingFilter.doFilter()(HttpTracingFilt
er.java:94)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:202)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

com.hp.swift.portal.servlet.SecurityFilter.doFilter()(SecurityFilter.jav
a:61)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:202)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

org.apache.catalina.core.StandardWrapperValve.invoke()(StandardWrapperVa
lve.java:213)

org.apache.catalina.core.StandardContextValve.invoke()(StandardContextVa
lve.java:178)

org.apache.catalina.core.StandardHostValve.invoke()(StandardHostValve.ja
va:126)

org.apache.catalina.valv

RE: Multi processor issue

2006-12-12 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> I attached below my bug catcher in 
> org.apache.catalina.connector.Request.

Looks good to me.  I'd be very interested in the results.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Caldarale, Charles R
> From: Christopher Schultz [mailto:[EMAIL PROTECTED] 
> Subject: Re: Multi processor issue
> 
> Thread.yield() will also work. It's unclear which is better from the
> API, but I would imagine that a VM implementor would write the
> Thread.yield method such that any remaining CPU time is yielded
> immediately, whereas you would have to guess at the amount of "sleep"
> time to use (but I'm sure something like 500ms would be just fine, as
> CPU time slices are pretty small).

Thread.yield() would be much, much more efficient than a sleep() call,
both in terms of OS internal operations and response time back to the
client.  Neither has anything to do with the time slice.  However, I
don't think either one is really necessary in this exercise.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-12 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Dong,

JiaDong Huang wrote:
> Yes. Before and maybe within the loop. That should get the MT issue
> reproduced much easier.

This was a great idea, by the way.

>> What do you mean by yelding the CPU (sorry I'm French) ?
> 
> Yielding the CPU means to let your thread give up the CPU so as to give the
> others thread a change to run. The delay call Thread.sleep() should yield
> the CPU.

Thread.yield() will also work. It's unclear which is better from the
API, but I would imagine that a VM implementor would write the
Thread.yield method such that any remaining CPU time is yielded
immediately, whereas you would have to guess at the amount of "sleep"
time to use (but I'm sure something like 500ms would be just fine, as
CPU time slices are pretty small).

- -chris

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFfsBF9CaO5/Lv0PARAmKlAKCG7ABMw3INZLWyTwpV5wwc9X2OxACggk+z
zlUGR1XVADFUDeRSloRIbq4=
=lb4L
-END PGP SIGNATURE-

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread JiaDong Huang
-Original Message-
From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, 12 December 2006 9:21 PM
To: Tomcat Users List
Subject: RE: Multi processor issue

> Do you mean I should add a delay before the while loop to increase the
> length of the critical section?

Yes. Before and maybe within the loop. That should get the MT issue
reproduced much easier.

> What do you mean by yelding the CPU (sorry I'm French) ?

Yielding the CPU means to let your thread give up the CPU so as to give the
others thread a change to run. The delay call Thread.sleep() should yield
the CPU.

Sorry, I should have said in my previous email: "make some time delay - and
the thread should yield the CUP".

Dong


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Marziou, Gael
Dong, I'm not sure I understand your suggestion, I attached below my bug
catcher in org.apache.catalina.connector.Request.

Do you mean I should add a delay before the while loop to increase the
length of the critical section?

What do you mean by yelding the CPU (sorry I'm French) ?

Gael

public Map getParameterMap() {

synchronized(parameterMap) {
Thread currentThread = Thread.currentThread();
if (pmapThread == null) {
// Store context of first caller thread
pmapThread = currentThread;
pmapStackTrace = currentThread.getStackTrace();
} else {
if (currentThread != pmapThread) {
String msg = "Original thread = " + pmapThread +
"\nOriginal stack trace = " +
stack2string(pmapStackTrace) +
"\nCurrent thread = " + currentThread;
throw new IllegalStateException(msg);
}
}
}

if (parameterMap.isLocked())
return parameterMap;

Enumeration enumeration = getParameterNames();
while (enumeration.hasMoreElements()) {
String name = enumeration.nextElement().toString();
String[] values = getParameterValues(name);
parameterMap.put(name, values);
}

parameterMap.setLocked(true);

return parameterMap;

} 

-Original Message-
From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, December 12, 2006 10:42 AM
To: 'Tomcat Users List'
Subject: RE: Multi processor issue

Gael,

If you can build the Tomcat code/component, put in time delay code and
also yield the CPU, within the ParameterMap initialization procedure.

Dong

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread JiaDong Huang
Gael,

If you can build the Tomcat code/component, put in time delay code and also
yield the CPU, within the ParameterMap initialization procedure.

Dong

-Original Message-
From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, 12 December 2006 12:36 AM
To: Tomcat Users List
Subject: RE: Multi processor issue

Thanks for all replies, you have convinced me that we should investigate
further.

I propose to implement a bug catcher in Tomcat ParameterMap by storing
the thread that called the constructor amd then check its invariance
when calling other methods (e.g. isLocked) of this class, if the
invariance is violated we will throw an exception.

It seems to me that with this data, we should be able to determine which
code is accessing the map and so we should know whether it's a bug in
the application or not.

Any other suggestion?


Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-11 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Dong,

JiaDong Huang wrote:
> Let's step back and see the whole picture, more widely. Let's presume
> the symptom as: the company's Web developers have done the same test
> with MT on single CPU and nothing wrong found. And the issue only
> happens within the MP environment. That should be defined/considered
> as MP specific - what I meant by MP issue but not MT issue.

Just because a result is more easily obtained in particular environment
doesn't mean that the environment is the cause. This is a correlation,
not a causality. You are likely to have to try much harder to reproduce
this problem on single CPU hardware. The same can be said about any
threading problem, ever.

> The Tomcat code you had dug out (having problem and throwing the
> exception) has been designed as single threaded. But in MP
> environment, multiple threads get in and cause issue like this. That
> means somewhere in Tomcat or JVM, the synchronization facility has
> already been broken, before running the code shown.

Dude, what are you talking about? The Tomcat team "designed" this to be
single-threaded, yes, but that means that they hadn't intended for
someone to use that object by multiple threads. The number of processors
is irrelevant... even if you have 100 CPUs, the same single thread
cannot run twice at the same time on the same object... you need another
thread. Thus, a threading issue.

> The root is where the synchronization facility was broken.

There is no synchronization at all. It's not broken... it's missing.
Because the Tomcat team designed it as single-threaded.

> As far as I understand, Tomcat connector may use JNI, not sure the
> threads we are talking about had run through the code implemented
> with JNI.

Irrelevant.

> To dig out this root is not simple. Ideally, we should have the issue
>  reproduced and check what sort of thread synchronization problem it
> may be - to see if my above guess is the case.

Another big problem is that the team having the problem no longer cares
to investigate it.

> Or, Tomcat could be enhanced to prevent this sort synchronization 
> issue from happening.

It could, but you'll have a hard time getting anyone on the Tomcat team
to add the word "synchronized" to any part of the TC code.

> To have the thread scheduler single threaded with single CPU would be
>  safer/better, and a reasonable thing to do.

This is a terrible idea. One person has a threading issue somewhere and
so the entire Tomcat user community now needs to discard the ability to
use multiple processors? Foolish.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFfXcv9CaO5/Lv0PARAp7bAJ446ue9rSvIRO4I5ydbryVC4QrHQACdG5hN
Jtd8IeYxMlI3I334fPDHsPs=
=9Jdd
-END PGP SIGNATURE-

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-11 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> I propose to implement a bug catcher in Tomcat ParameterMap by storing
> the thread that called the constructor amd then check its invariance
> when calling other methods (e.g. isLocked) of this class, if the
> invariance is violated we will throw an exception.

>From a purely theoretical perspective, it is valid for two threads to
call getParameterMap() (and therefore isLocked()), once the ParameterMap
has been established.  Probably not a concern in the real world.

Since the only call to ParameterMap.isLocked() comes from
Request.getParameterMap(), I'd put the saving of the calling Thread
object in a synchronized block there, along with a check to insure no
other thread had been there.  In addition, I would also save the array
returned by getStackTrace() (assuming you're running on SE 5) to provide
some context for the call.  This would also help to widen the window and
hopefully cause the problem to occur sooner.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-11 Thread Caldarale, Charles R
> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> So it is just a case of simple MT issue - multiple threads have
> accessed/operated on a class/object that is not MT safe.


I wouldn't label the situation as "simple" - threading issues frequently
are not.

> If an effective stress test had been done on single CPU 
> environment, the issue should have been revealed too.

In theory yes, but it could well take years to trigger the problem on a
single CPU, unless artificial pauses are inserted to widen the timing
windows.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-11 Thread Marziou, Gael
Thanks for all replies, you have convinced me that we should investigate
further.

I propose to implement a bug catcher in Tomcat ParameterMap by storing
the thread that called the constructor amd then check its invariance
when calling other methods (e.g. isLocked) of this class, if the
invariance is violated we will throw an exception.

It seems to me that with this data, we should be able to determine which
code is accessing the map and so we should know whether it's a bug in
the application or not.

Any other suggestion?


Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-11 Thread Peter Crowther
> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> I thought they had
> done certain code review and did not find any MT safe issue.

They had (and I naively took them at their word), and they didn't find
any such issue.  However, as Chuck points out so effectively, "we didn't
find any issues" isn't the same thing as "there are no issues in the
code".  Unfortunately, code reviews merely increase the chance of
revealing a defect - they don't increase it to 100%.  Nor does any other
technique, otherwise the safety-critical folks would be using it
exclusively.

- Peter
--
"So far as we know, we have never had an undiscovered error."

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-11 Thread JiaDong Huang
Appreciate your reply analyzing my guessing cases.

So it is just a case of simple MT issue - multiple threads have
accessed/operated on a class/object that is not MT safe. I thought they had
done certain code review and did not find any MT safe issue.

If an effective stress test had been done on single CPU environment, the
issue should have been revealed too.

Dong

-Original Message-
From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] 
Sent: Monday, 11 December 2006 3:30 PM
To: Tomcat Users List
Subject: RE: Multi processor issue

> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> It is a MT issue, but MP specific.

Strictly speaking, it's not MP-specific.  If run long enough, the
problem will appear on a single CPU.  Its occurrence would require the
first thread to exhaust its CPU time slice at a critical point, and have
the second thread be dispatched immediately.  (In most operating
systems, when a thread uses up its time slice, it's placed at the end of
the dispatch queue for its priority, thus allowing other threads of the
same or higher priority to run ahead of it.  Note that the OS-defined
priority in this case often has little to do with the Java-specified
priority for a thread.)  That said, the situation is certainly much,
much more likely to occur on an MP system.

> The Tomcat code you had dug out (having problem and throwing 
> the exception) has been designed as single threaded. But in MP
> environment, multiple threads get in and cause issue like this.
> That means somewhere in Tomcat or JVM, the synchronization
> facility has already been broken

While that would be true in general, it's not the case here.  Objects of
the Request class are never intended to be used concurrently by multiple
threads, so there is no synchronization defined or implemented for them.
No second thread - whatever its origin - should be operating on the same
Request object that another thread is handling.

> As far as I understand, Tomcat connector may use JNI

The APR and AJP connects are implemented with JNI, the standard HTTP and
HTTPS ones are pure Java.

> not sure the threads we are talking about had run through the
> code implemented with JNI.

They did not, but it's not really relevant, given the Tomcat requirement
of only one thread manipulating a Request object at a time.

> Threads running through JNI may be re-marshaled between OS spaces - 
> switching OS rings or VM.

They can change ring (generically, the thread privilege level), but not
VMs.  If you're suggesting that one thread may change its process
association, that is theoretically possible on some OS implementations
(anybody remember Multics?), but it is certainly not happening here,
and, in any event, is completely irrelevant.  (Any such change from one
process to another must be extremely tightly controlled for security
purposes, and is a capability simply not available to typical user
threads.)

> In another word, while switching between rings, the lock associated
> with the objects may have been lost, etc.

That's erroneous, even if it were relevant to this situation, which it
isn't.  There's no synchronization issue involved, since the Request
object is not intended to be manipulated concurrently by multiple
threads.  This problem is almost certainly an application problem,
either starting an additional thread somewhere or exposing the Request
object improperly (e.g., storing a reference to it in the session) so
that another thread erroneously starts working with this Request instead
of the one its supposed to be working on.

> Also, the JVM or OS API may need other synchronization 
> facility underneath while switching rings. These are
> only my guess anyway.

No code in the JVM depends on or expects to change privilege level
(other than the privileges controlled entirely within the JVM), due to
its platform independence.  I think you may be trying to apply some
rather esoteric capabilities of certain operating systems, but they
really have no bearing on this situation.

> It could be the JVM's synchronization facility does not work 
> properly in MP, for Tomcat. Or, Tomcat could be enhanced to
> prevent this sort synchronization issue from happening.

You're ignoring one basic fact: this problem has been reported on only
*one* system running *one* particular application.  If there were a
general synchronization problem in Tomcat, the JVM, or the OS, it would
certainly show up on some of the thousands (millions?) of other
multi-processor systems running Tomcat.  As I stated earlier, we have
zero problems running Tomcat on our 32x servers (of various
architectures).  There is a very small chance that something in the
hardware on this particular system has gone bad (e.g., lock signal not
being propagated on the bus), but such a problem would undoubtedly show
up in more places

RE: Multi processor issue

2006-12-10 Thread Caldarale, Charles R
> From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> It is a MT issue, but MP specific.

Strictly speaking, it's not MP-specific.  If run long enough, the
problem will appear on a single CPU.  Its occurrence would require the
first thread to exhaust its CPU time slice at a critical point, and have
the second thread be dispatched immediately.  (In most operating
systems, when a thread uses up its time slice, it's placed at the end of
the dispatch queue for its priority, thus allowing other threads of the
same or higher priority to run ahead of it.  Note that the OS-defined
priority in this case often has little to do with the Java-specified
priority for a thread.)  That said, the situation is certainly much,
much more likely to occur on an MP system.

> The Tomcat code you had dug out (having problem and throwing 
> the exception) has been designed as single threaded. But in MP
> environment, multiple threads get in and cause issue like this.
> That means somewhere in Tomcat or JVM, the synchronization
> facility has already been broken

While that would be true in general, it's not the case here.  Objects of
the Request class are never intended to be used concurrently by multiple
threads, so there is no synchronization defined or implemented for them.
No second thread - whatever its origin - should be operating on the same
Request object that another thread is handling.

> As far as I understand, Tomcat connector may use JNI

The APR and AJP connects are implemented with JNI, the standard HTTP and
HTTPS ones are pure Java.

> not sure the threads we are talking about had run through the
> code implemented with JNI.

They did not, but it's not really relevant, given the Tomcat requirement
of only one thread manipulating a Request object at a time.

> Threads running through JNI may be re-marshaled between OS spaces - 
> switching OS rings or VM.

They can change ring (generically, the thread privilege level), but not
VMs.  If you're suggesting that one thread may change its process
association, that is theoretically possible on some OS implementations
(anybody remember Multics?), but it is certainly not happening here,
and, in any event, is completely irrelevant.  (Any such change from one
process to another must be extremely tightly controlled for security
purposes, and is a capability simply not available to typical user
threads.)

> In another word, while switching between rings, the lock associated
> with the objects may have been lost, etc.

That's erroneous, even if it were relevant to this situation, which it
isn't.  There's no synchronization issue involved, since the Request
object is not intended to be manipulated concurrently by multiple
threads.  This problem is almost certainly an application problem,
either starting an additional thread somewhere or exposing the Request
object improperly (e.g., storing a reference to it in the session) so
that another thread erroneously starts working with this Request instead
of the one its supposed to be working on.

> Also, the JVM or OS API may need other synchronization 
> facility underneath while switching rings. These are
> only my guess anyway.

No code in the JVM depends on or expects to change privilege level
(other than the privileges controlled entirely within the JVM), due to
its platform independence.  I think you may be trying to apply some
rather esoteric capabilities of certain operating systems, but they
really have no bearing on this situation.

> It could be the JVM's synchronization facility does not work 
> properly in MP, for Tomcat. Or, Tomcat could be enhanced to
> prevent this sort synchronization issue from happening.

You're ignoring one basic fact: this problem has been reported on only
*one* system running *one* particular application.  If there were a
general synchronization problem in Tomcat, the JVM, or the OS, it would
certainly show up on some of the thousands (millions?) of other
multi-processor systems running Tomcat.  As I stated earlier, we have
zero problems running Tomcat on our 32x servers (of various
architectures).  There is a very small chance that something in the
hardware on this particular system has gone bad (e.g., lock signal not
being propagated on the bus), but such a problem would undoubtedly show
up in more places than this one spot in Tomcat.

> After reviewing the messages, it is tomcat5.exe has been modified.
> That means it may be tomcat5.exe should be considered to enhance,
> in the area of thread synchronization/scheduling.

Completely off base.  The tomcat5.exe program is simply another way to
launch a standard JVM; it does almost nothing other than apply the
configured JVM options.  It is merely the starting point for the
process, so it's the easiest spot at which to impose affinity rules that
will apply to all threads of the proces

Re: Multi processor issue

2006-12-09 Thread Gael Marziou

the affinity 'trick' is a replacement dll for the 80 runtime stock libraries



Well, the one we used does not work this way.

In fact, we started by changing the affinity of Tomcat service's live 
process using Windows taskmgr or Windows System Resource Manager.


Then, as problem disappeared, we looked for a permanent way to set this 
affinity and the tool that we used does it by settings the affinity 
bitmask in the executable header, so it does not require any special 
DLL. I compared the 2 executables and they differ only one byte.


http://forums.amd.com/lofiversion/index.php/t65211.html

-- Gaël


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Caldarale, Charles R
> From: news [mailto:[EMAIL PROTECTED] On Behalf Of Bill Barker
> Subject: Re: Multi processor issue
> 
> After the request parameters are parsed, Tomcat makes it's 
> parameter map read-only (i.e. "locked").

This is true, but it's not a simple case of someone calling put() after
locking the ParameterMap.  Here's the code of interest
(org.apache.catalina.connector.Request):

1015public Map getParameterMap() {
1017if (parameterMap.isLocked())
1018return parameterMap;
1020Enumeration enumeration = getParameterNames();
1021while (enumeration.hasMoreElements()) {
1022String name = enumeration.nextElement().toString();
1023String[] values = getParameterValues(name);
1024parameterMap.put(name, values);
1025}
1027parameterMap.setLocked(true);
1029return parameterMap;
1031}

The stack trace shows the failed call to put() being made from line
1024.  In order to get this far, the test for isLocked() at line 1017
must have returned false.  In other words, the locked state of the
ParameterMap associated with this request changed between lines 1017 and
1024.  Since locked isn't set until line 1027 (and that appears to be
the only place in Tomcat that does so), some thread other than the one
in the stack trace must have changed the state by coming through this
code simultaneously.  Restricting Tomcat to a single CPU almost
completely precludes that from ever happening.  (But if left running
long enough - possibly years - the situation could still occur with a
single CPU, when one thread's time slice expired between lines 1020 and
1025, and whatever second thread is also erroneously processing the same
Request object starts running.  Contrary to what another poster stated,
this is definitely a multi-thread issue, not a multi-processor one.)

If hardware or OS infrastructure were the cause, I would expect that to
manifest itself in more ways than just this one particular situation in
Tomcat.  If it were my system, I'd be adding some trap code to mark a
Request object as taken when a thread starts processing it, and verify
that a request isn't already taken when a thread starts processing it.
Unfortunate that the OP's company seems to have lost interest in fixing
what appears to be a bona fide problem in their application.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-08 Thread Bill Barker
After the request parameters are parsed, Tomcat makes it's parameter map 
read-only (i.e. "locked").  I suspect that this is the source of your 
problems.  Having said that, if you really do have two threads accessing a 
Request instance, then you really really need to re-design your application 
;-).

"Marziou, Gael" <[EMAIL PROTECTED]> wrote in message 
news:[EMAIL PROTECTED]
Hi,

As a preamble, let me say that I am not complaining about Tomcat (as the
problem root cause is uncertain) and I don't expect much assistance as
we are happy with our workaround.
However, our experience could help others or someone may have other
ideas.

We have recently switched a stable web application from JRun 4 to Tomcat
5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08, running some testing for
months, but when we moved to production environment we immediately
experienced errors IllegalStateException: No modifications are allowed
to a locked ParameterMap.

Stack trace analysis concluded to RequestFacade objects shared between 2
threads.

We made code reviews of our application code and Tomcat's code and did
not find any thread safety issues.

So, we checked all differences between staging and production, we
thought that the major difference was that the staging server was a
single CPU machine while the production servers had 2 physical
processors and 4 logical ones.

So, we constrained the Tomcat service to run on only one processor by
setting its processor affinity mask to 1 (run on CPU 0) and after
several weeks of monitoring we concluded that it "fixed" the problem.

So, we used a small utility (search for setaffinity.zip) to patch the
affinity mask in the Tomcat service executable (tomcat5.exe).

Later on, we set up a second production environment also on bi-processor
servers but we did not experience the same issue.
The hardware is not exactly the same as first production setup and the
OS is also different (Windows 2000 on 2nd env, Windows 2003 on 1st env
having the issue).

So, I don't believe it's a Tomcat, I would rather think to an H/W, or
JRE or OS issue.
In any case, it's a rather complex issue to investigate and we haven't
enough time for this as we are happy with performances running on one
single CPU.

Thanks,

Gael Marziou
Hewlett Packard

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]





-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread JiaDong Huang
>From the description of the symptom - months' testing experience, we could
pretty much confirm that it is a MP issue, not a MT issue. Chasing the stack
dump of Java may not get to any significant point. If you try to fix the MP
specific issue like this at the high level like Tomcat/JAVA app code, you
will get to a situation like: fix one issue, another issue will come up
eventually.

The producer of MSVCR80.dll may be interested in knowing this. This may be
Win32 OS or JVM specific.

Appreciate your findings and solution!

Dong

-Original Message-
From: Martin Gainty [mailto:[EMAIL PROTECTED] 
Sent: Saturday, 9 December 2006 6:18 AM
To: Tomcat Users List
Subject: Re: Multi processor issue

the affinity 'trick' is a replacement dll for the 80 runtime stock libraries

The individual most familiar with interfacing to the runtime is mladen turk
I would make all attempts to ping him on the solution

As a comparison between the 2 dlls 
Here are the binary characteristics of the stock msvcr80.dll that ships with
retail windows packages

  Section contains the following exports for MSVCR80.dll

   0 characteristics
40ECD724 time date stamp Thu Jul 08 01:09:56 2004
0.00 version
   1 ordinal base
1367 number of functions
1367 number of names

ordinal hint RVA  name

/*and the replacement msvcr80.dll provided in setaffinity*/

  Section contains the following exports for MSVCR80.dll

   0 characteristics
4333A455 time date stamp Fri Sep 23 02:44:37 2005
0.00 version
   1 ordinal base
1459 number of functions
1459 number of names

ordinal hint RVA  name

so the new msvcr80.dll from setaffinity has 92 new functions which allow it
to work reliably with MP calls

HTH,
Martin--
--- 
This e-mail message (including attachments, if any) is intended for the use
of the individual or entity to which it is addressed and may contain
information that is privileged, proprietary , confidential and exempt from
disclosure. If you are not the intended recipient, you are notified that any
dissemination, distribution or copying of this communication is strictly
prohibited.
--- 
Le présent message électronique (y compris les pièces qui y sont annexées,
le cas échéant) s'adresse au destinataire indiqué et peut contenir des
renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le
destinataire de ce document, nous vous signalons qu'il est strictement
interdit de le diffuser, de le distribuer ou de le reproduire.



> But the evidence seems to show two threads are manipulating the same 
> request, otherwise you wouldn't get the error on the ParameterMap.  
> Does a full thread dump show any threads that aren't part of the set 
> started by Tomcat?

> either that, or a Request is reused before recycle() has been executed
completely.

We assumed this as well but after instrumenting more our code, we
concluded it was not possible because the corruption seemed to occur
before the end of the 2 requests processing and so before recycle() was
called.

I got some logs which sort of showed this but I can't find them right
now, I have stopped working on this issue 3 weeks ago.

> Alas, you could do a test and check whether the method getParameter is

> actually ever called by two threads during one lifecycle (patching the
> ParameterMap class, storing last accessing thread, and checking
whether
> another thread is already stored on access)

We did it but for some reason the log statements did not get logged in
our usual log file.
Then after we tried the affinity trick which made the symptom to
disappear and got no chance to try again.

> And Gael, the CPU is the most valuable resource in the production
environment, 
> are you really ready to give up your scaleability but simply not
investigating 
> further?

Well, this application is load balanced on 4 servers and the load is
currently pretty low on each of them even when running on a single CPU.
We did add more logs to our application and we did also instrument
Tomcat but with no luck and as we were unable to reproduce it in a dev
environment, it's difficult to ask for short downtimes to restart a
server in production.
On top of that, my team is being moved to another project...

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-08 Thread Martin Gainty
the affinity 'trick' is a replacement dll for the 80 runtime stock libraries 
The individual most familiar with interfacing to the runtime is mladen turk
I would make all attempts to ping him on the solution

As a comparison between the 2 dlls 
Here are the binary characteristics of the stock msvcr80.dll that ships with 
retail windows packages

  Section contains the following exports for MSVCR80.dll

   0 characteristics
40ECD724 time date stamp Thu Jul 08 01:09:56 2004
0.00 version
   1 ordinal base
1367 number of functions
1367 number of names

ordinal hint RVA  name

/*and the replacement msvcr80.dll provided in setaffinity*/

  Section contains the following exports for MSVCR80.dll

   0 characteristics
4333A455 time date stamp Fri Sep 23 02:44:37 2005
0.00 version
   1 ordinal base
1459 number of functions
1459 number of names

ordinal hint RVA  name

so the new msvcr80.dll from setaffinity has 92 new functions which allow it to 
work reliably with MP calls

HTH,
Martin--
--- 
This e-mail message (including attachments, if any) is intended for the use of 
the individual or entity to which it is addressed and may contain information 
that is privileged, proprietary , confidential and exempt from disclosure. If 
you are not the intended recipient, you are notified that any dissemination, 
distribution or copying of this communication is strictly prohibited.
--- 
Le présent message électronique (y compris les pièces qui y sont annexées, le 
cas échéant) s'adresse au destinataire indiqué et peut contenir des 
renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le 
destinataire de ce document, nous vous signalons qu'il est strictement interdit 
de le diffuser, de le distribuer ou de le reproduire.



> But the evidence seems to show two threads are manipulating the same 
> request, otherwise you wouldn't get the error on the ParameterMap.  
> Does a full thread dump show any threads that aren't part of the set 
> started by Tomcat?

> either that, or a Request is reused before recycle() has been executed
completely.

We assumed this as well but after instrumenting more our code, we
concluded it was not possible because the corruption seemed to occur
before the end of the 2 requests processing and so before recycle() was
called.

I got some logs which sort of showed this but I can't find them right
now, I have stopped working on this issue 3 weeks ago.

> Alas, you could do a test and check whether the method getParameter is

> actually ever called by two threads during one lifecycle (patching the
> ParameterMap class, storing last accessing thread, and checking
whether
> another thread is already stored on access)

We did it but for some reason the log statements did not get logged in
our usual log file.
Then after we tried the affinity trick which made the symptom to
disappear and got no chance to try again.

> And Gael, the CPU is the most valuable resource in the production
environment, 
> are you really ready to give up your scaleability but simply not
investigating 
> further?

Well, this application is load balanced on 4 servers and the load is
currently pretty low on each of them even when running on a single CPU.
We did add more logs to our application and we did also instrument
Tomcat but with no luck and as we were unable to reproduce it in a dev
environment, it's difficult to ask for short downtimes to restart a
server in production.
On top of that, my team is being moved to another project...

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael

> But the evidence seems to show two threads are manipulating the same 
> request, otherwise you wouldn't get the error on the ParameterMap.  
> Does a full thread dump show any threads that aren't part of the set 
> started by Tomcat?

> either that, or a Request is reused before recycle() has been executed
completely.

We assumed this as well but after instrumenting more our code, we
concluded it was not possible because the corruption seemed to occur
before the end of the 2 requests processing and so before recycle() was
called.

I got some logs which sort of showed this but I can't find them right
now, I have stopped working on this issue 3 weeks ago.

> Alas, you could do a test and check whether the method getParameter is

> actually ever called by two threads during one lifecycle (patching the
> ParameterMap class, storing last accessing thread, and checking
whether
> another thread is already stored on access)

We did it but for some reason the log statements did not get logged in
our usual log file.
Then after we tried the affinity trick which made the symptom to
disappear and got no chance to try again.

> And Gael, the CPU is the most valuable resource in the production
environment, 
> are you really ready to give up your scaleability but simply not
investigating 
> further?

Well, this application is load balanced on 4 servers and the load is
currently pretty low on each of them even when running on a single CPU.
We did add more logs to our application and we did also instrument
Tomcat but with no luck and as we were unable to reproduce it in a dev
environment, it's difficult to ask for short downtimes to restart a
server in production.
On top of that, my team is being moved to another project...

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael

> Does the full stack trace show where the threads originated?  
> You appear to have given us only a partial of each error.

OK, see below.

> Is the request object stored in any session or global structure?

No.

> Could another thread have found it there and picked it up erroneously?


No.

> Do you have any listeners configured that might be getting in the
game?

No.


java.lang.IllegalStateException: No modifications are allowed to a
locked ParameterMap
at
org.apache.catalina.util.ParameterMap.put(ParameterMap.java:165)
at
org.apache.catalina.connector.Request.getParameterMap(Request.java:1024)
at
org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacad
e.java:411)
at
org.apache.catalina.core.ApplicationHttpRequest.parseParameters(Applicat
ionHttpRequest.java:727)
at
org.apache.catalina.core.ApplicationHttpRequest.getParameter(Application
HttpRequest.java:352)
at
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)
at
com.hp.cii.radix.user.web.home.Home2Context.doUpdateSolutionSummary(Home
2Context.java:833)
at
com.hp.cii.radix.user.web.home.Home2Context.doDisplaySolutionSummary(Hom
e2Context.java:919)
at sun.reflect.GeneratedMethodAccessor593.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.hp.swift.util.MethodInvoker.invoke(MethodInvoker.java:52)
at
com.hp.swift.state.core.StateContextAction.execute(StateContextAction.ja
va:51)
at
com.hp.swift.state.core.Transition.executeAction(Transition.java:124)
at
com.hp.swift.state.core.StateMachineController.fireTransition(StateMachi
neController.java:1013)
at
com.hp.swift.state.core.StateMachineController.doHandle(StateMachineCont
roller.java:788)
at
com.hp.swift.state.core.StateMachineController.handle(StateMachineContro
ller.java:520)
at
com.hp.swift.portal.servlet.SmvController.doHandleRequest(SmvController.
java:393)
at
com.hp.swift.portal.servlet.SmvController.doGet(SmvController.java:235)
at
com.hp.swift.portal.servlet.SmvController.doPost(SmvController.java:215)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
her.java:672)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio
nDispatcher.java:463)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
atcher.java:359)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
cher.java:301)
at
com.hp.swift.proxy.ProxySessionController.forwardRequest(ProxySessionCon
troller.java:57)
at
com.hp.swift.portal.servlet.SessionController$1.run(SessionController.ja
va:159)
at
com.hp.swift.security.manager.LightSecurityManager.doAsPrivileged(LightS
ecurityManager.java:272)
at
com.hp.swift.security.manager.SecurityChecker.doAsPrivileged(SecurityChe
cker.java:187)
at
com.hp.swift.portal.servlet.SessionController.forwardRequestAs(SessionCo
ntroller.java:155)
at
com.hp.swift.portal.servlet.SessionController.doHttp(SessionController.j
ava:132)
at
com.hp.swift.portal.servlet.SessionController.doPost(SessionController.j
ava:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at
com.hp.swift.portal.init.FirstLevelInitableServlet.service(FirstLevelIni
tableServlet.java:154)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
com.hp.swift.portal.servlet.HttpTracingFilter.doFilter(HttpTracingFilter
.java:89)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
com.hp.swift.portal.servlet.SecurityFilter.doFilter(SecurityFilter.java:
61)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:178)
at
org.apac

Re: Multi processor issue

2006-12-08 Thread Leon Rosenberg

On 12/8/06, Caldarale, Charles R <[EMAIL PROTECTED]> wrote:

> From: Marziou, Gael [mailto:[EMAIL PROTECTED]
> Subject: RE: Multi processor issue
>
> Tomcat does not synchronize before locking the map but that should not
> be an issue because the request is guaranteed to be processed by only
> one thread.

But the evidence seems to show two threads are manipulating the same
request, otherwise you wouldn't get the error on the ParameterMap.  Does
a full thread dump show any threads that aren't part of the set started
by Tomcat?


either that, or a Request is reused before recycle() has been executed
completely.
I fear we have to know what your software on top does exactly :-)

Alas, you could do a test and check whether the method getParameter is
actually ever called by two threads during one lifecycle (patching the
ParameterMap class, storing last accessing thread, and checking
whether another thread is already stored on access)

Theoretically Cael is right with the above, but only theoretically,
since we don't know how thread safe Gael's "ontop" application is, and
even than, it wouldn't be the first synchronization issue we found in
tomcat.
And Gael, the CPU is the most valuable resource in the production
environment, are you really ready to give up your scaleability but
simply not investigating further?

Leon



 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> Tomcat does not synchronize before locking the map but that should not
> be an issue because the request is guaranteed to be processed by only
> one thread.

But the evidence seems to show two threads are manipulating the same
request, otherwise you wouldn't get the error on the ParameterMap.  Does
a full thread dump show any threads that aren't part of the set started
by Tomcat?

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> This is our code and there's no threading here. 

Does the full stack trace show where the threads originated?  You appear
to have given us only a partial of each error.

Is the request object stored in any session or global structure?  Could
another thread have found it there and picked it up erroneously?  Do you
have any listeners configured that might be getting in the game?

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-08 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Leon,

Leon Rosenberg wrote:
> ApplicationHttpRequest.parseParameters isn't synchronized...
> thats probably the source of your evil.

That should not really be an issue... the real problem is that it
appears that he has two threads accessing this code at the same time
when he is only expecting one.

Sure, synchronization might "fix" the problem, but it doesn't really get
to the root of why it's happening in the first place.

Gael,
It probably won't end up showing anything, but could you catch the
IllegalStateException and add a full thread dump in the exception
handler? Then you might see what other threads are using that particular
request object. I say it probably won't help because by the time the
thread dump is done, your threads are probably out of that code anyway.

- -chris

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFeajl9CaO5/Lv0PARAtpQAJ4jwfiHJRw4InJIaHAPqzARZ3qvAQCeLI4C
V6u7ZA4iDNRdga+XWnnZDNk=
=ny0B
-END PGP SIGNATURE-

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
First, to clarify in my message, I did put 2 stack traces of 2
independent errors.

Tomcat does not synchronize before locking the map but that should not
be an issue because the request is guaranteed to be processed by only
one thread.

HttpEventContextObject.getParameter() does just extract the parameters
of the request to store them in what we cal an event, I suppose it could
be compared to a Struts action.
So, we only read the Request parameters, nothing fancy.


Gael

-Original Message-
From: Leon Rosenberg [mailto:[EMAIL PROTECTED] 
Sent: Friday, December 08, 2006 6:38 PM
To: Tomcat Users List
Subject: Re: Multi processor issue

looks like
Request.getParameterMap()
is called twice. The first call locks the request, the second fails.


what exactly does your
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)  ?

Could it be that two threads passes through this codepiece concurrently
and both calls comes through to the Request object? That would be a
clear synchronization issue and pretty typical for tomcat
:-)

regards
Leon

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-08 Thread Leon Rosenberg

ApplicationHttpRequest.parseParameters isn't synchronized...
thats probably the source of your evil.
leon

On 12/8/06, Leon Rosenberg <[EMAIL PROTECTED]> wrote:

looks like
Request.getParameterMap()
is called twice. The first call locks the request, the second fails.


what exactly does your
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)  ?

Could it be that two threads passes through this codepiece
concurrently and both calls comes through to the Request object? That
would be a clear synchronization issue and pretty typical for tomcat
:-)

regards
Leon


On 12/8/06, Caldarale, Charles R <[EMAIL PROTECTED]> wrote:
> > From: Marziou, Gael [mailto:[EMAIL PROTECTED]
> > Subject: RE: Multi processor issue
> >
> > com.hp.swift.portal.state.HttpEventContextObject.getParameter(
> > HttpEventContextObject.java:64)
>
> This looks suspicious - perhaps this portal code is the source of the
> second thread.
>
>  - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail
> and its attachments from all computers.
>
> -
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-08 Thread Leon Rosenberg

looks like
Request.getParameterMap()
is called twice. The first call locks the request, the second fails.


what exactly does your
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)  ?

Could it be that two threads passes through this codepiece
concurrently and both calls comes through to the Request object? That
would be a clear synchronization issue and pretty typical for tomcat
:-)

regards
Leon


On 12/8/06, Caldarale, Charles R <[EMAIL PROTECTED]> wrote:

> From: Marziou, Gael [mailto:[EMAIL PROTECTED]
> Subject: RE: Multi processor issue
>
> com.hp.swift.portal.state.HttpEventContextObject.getParameter(
> HttpEventContextObject.java:64)

This looks suspicious - perhaps this portal code is the source of the
second thread.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
This is our code and there's no threading here. 

-Original Message-
From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] 
Sent: Friday, December 08, 2006 6:31 PM
To: Tomcat Users List
Subject: RE: Multi processor issue

> From: Marziou, Gael [mailto:[EMAIL PROTECTED]
> Subject: RE: Multi processor issue
> 
> com.hp.swift.portal.state.HttpEventContextObject.getParameter(
> HttpEventContextObject.java:64)

This looks suspicious - perhaps this portal code is the source of the
second thread.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe,
e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: RE: Multi processor issue
> 
> com.hp.swift.portal.state.HttpEventContextObject.getParameter(
> HttpEventContextObject.java:64)

This looks suspicious - perhaps this portal code is the source of the
second thread.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
> Can you post the stack trace for that IllegalStateException?

Sure but that's only one of the symptoms we got.
I don't remember if the stack traces below were taken on 5.5.18 or
5.5.20 after we upgraded but as far as I know this part of the code was
not changed.


java.lang.IllegalStateException: No modifications are allowed to a
locked ParameterMap
at
org.apache.catalina.util.ParameterMap.put(ParameterMap.java:165)
at
org.apache.catalina.connector.Request.getParameterMap(Request.java:1024)
at
org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacad
e.java:411)
at
org.apache.catalina.core.ApplicationHttpRequest.parseParameters(Applicat
ionHttpRequest.java:727)
at
org.apache.catalina.core.ApplicationHttpRequest.getParameterMap(Applicat
ionHttpRequest.java:373)
at com.hp.swift.proxy.Proxy.fillParameters(Proxy.java:951)

And another one:

java.lang.IllegalStateException: No modifications are allowed to a
locked ParameterMap
at
org.apache.catalina.util.ParameterMap.put(ParameterMap.java:165)
at
org.apache.catalina.connector.Request.getParameterMap(Request.java:1024)
at
org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacad
e.java:411)
at
org.apache.catalina.core.ApplicationHttpRequest.parseParameters(Applicat
ionHttpRequest.java:727)
at
org.apache.catalina.core.ApplicationHttpRequest.getParameter(Application
HttpRequest.java:352)
at
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael

> could you tell me 32-bit or 64-bit OS, service pack levels, and what
the hardware is?  

Peter, I'm not sure it will help you, the only advice I can give you is
to test on same configuration as your future production environment. 
We knew it from the beginning but were not listened.

Windows 2003 Server Enterprise Edition SP1 (32 bits)
HP PROLIANT DL380 G3 4 x Intel Xeon 3.2 Ghz 

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-08 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Gael,

>> Within Tomcat, request objects are normally processed by a single
>> thread at a time, yet you note that two threads are accessing this
>> particular request - where did the second one come from?
> 
> Both were web threads and error did occur at the very beginning of
> request processing when analyzing the request or when logging it in a
> filter. 

Can you post the stack trace for that IllegalStateException?

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFeZcR9CaO5/Lv0PARAvnrAKC34v6kL7obecqyXY/nZoujVI63SACfRo7t
bOxjE3GJithcAiczrsIjWH4=
=TG0h
-END PGP SIGNATURE-

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
Thanks Chuck for your reply, I only cited part of your message where I
could follow up.
 
> Within Tomcat, request objects are normally processed by a single
thread 
> at a time, yet you note that two threads are accessing this particular

> request - where did the second one come from?  

Both were web threads and error did occur at the very beginning of
request processing when analyzing the request or when logging it in a
filter. 

> Is your app starting extra threads?  
> (Perhaps unknowingly, by using some third-party
> package.)

No. We checked this under profiler and all our threads are built by our
thread factories and do not manipulate any request objects only our own
business objects.

> Since requests are typically handled by a single thread, 
> there is little or no synchronization done during manipulation 
> of references associated with them, such as the parameter map.

That's also our understanding.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Caldarale, Charles R
> From: Martin Gainty [mailto:[EMAIL PROTECTED] 
> Subject: Re: Multi processor issue
> 
> which VM are you using for this test?

Try actually reading the message:

"We have recently switched a stable web application from JRun 4 to
Tomcat
5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08"

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Multi processor issue

2006-12-08 Thread Martin Gainty
Good Morning Gael-

which VM are you using for this test?

Thanks,
Martin --
--- 
This e-mail message (including attachments, if any) is intended for the use of 
the individual or entity to which it is addressed and may contain information 
that is privileged, proprietary , confidential and exempt from disclosure. If 
you are not the intended recipient, you are notified that any dissemination, 
distribution or copying of this communication is strictly prohibited.

- Original Message - 
From: "Marziou, Gael" <[EMAIL PROTECTED]>
To: "Tomcat Users List" 
Sent: Friday, December 08, 2006 9:22 AM
Subject: Multi processor issue


Hi,

As a preamble, let me say that I am not complaining about Tomcat (as the
problem root cause is uncertain) and I don't expect much assistance as
we are happy with our workaround.
However, our experience could help others or someone may have other
ideas.

We have recently switched a stable web application from JRun 4 to Tomcat
5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08, running some testing for
months, but when we moved to production environment we immediately
experienced errors IllegalStateException: No modifications are allowed
to a locked ParameterMap.

Stack trace analysis concluded to RequestFacade objects shared between 2
threads.

We made code reviews of our application code and Tomcat's code and did
not find any thread safety issues. 

So, we checked all differences between staging and production, we
thought that the major difference was that the staging server was a
single CPU machine while the production servers had 2 physical
processors and 4 logical ones.

So, we constrained the Tomcat service to run on only one processor by
setting its processor affinity mask to 1 (run on CPU 0) and after
several weeks of monitoring we concluded that it "fixed" the problem.

So, we used a small utility (search for setaffinity.zip) to patch the
affinity mask in the Tomcat service executable (tomcat5.exe).

Later on, we set up a second production environment also on bi-processor
servers but we did not experience the same issue.
The hardware is not exactly the same as first production setup and the
OS is also different (Windows 2000 on 2nd env, Windows 2003 on 1st env
having the issue).

So, I don't believe it's a Tomcat, I would rather think to an H/W, or
JRE or OS issue.
In any case, it's a rather complex issue to investigate and we haven't
enough time for this as we are happy with performances running on one
single CPU.

Thanks,

Gael Marziou
Hewlett Packard

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Caldarale, Charles R
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> Subject: Multi processor issue
> 
> We have recently switched a stable web application from JRun 
> 4 to Tomcat 5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08,
> running some testing for months, but when we moved to 
> production environment we immediately experienced errors
> IllegalStateException: No modifications are allowed
> to a locked ParameterMap.
> 
> We made code reviews of our application code and Tomcat's code and did
> not find any thread safety issues. 

Look again.

> So, I don't believe it's a Tomcat, I would rather think to an H/W, or
> JRE or OS issue.

Welcome to the world of multi-processors (which I've been working with
since the 1960s.)  The problem is almost undoubtedly in your application
- we run Tomcat on 16x and 32x systems without any such issues.

Within Tomcat, request objects are normally processed by a single thread
at a time, yet you note that two threads are accessing this particular
request - where did the second one come from?  Is your app starting
extra threads?  (Perhaps unknowingly, by using some third-party
package.)

Since requests are typically handled by a single thread, there is little
or no synchronization done during manipulation of references associated
with them, such as the parameter map.  If your app is using multiple
threads to handle a request, it needs to provide its own synchronization
when calling methods of the request that manipulate data associated with
the request - even when just reading, since the other thread may be
changing it concurrently.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Peter Crowther
> From: Marziou, Gael [mailto:[EMAIL PROTECTED] 
> So, I don't believe it's a Tomcat, I would rather think to an H/W, or
> JRE or OS issue.

Fascinating - partly because I may hit the same issue soon and partly to
log it on the list for the future, could you tell me 32-bit or 64-bit
OS, service pack levels, and what the hardware is?  I don't want to trip
over the same problem, but nor do I want to give the hardware too wide a
berth!

- Peter

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]