Re: Servlet Threads Changing Instance Data
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Jerry,
On 8/15/18 4:00 PM, Jerry Malcolm wrote:
> On 8/15/2018 1:50 PM, Olaf Kock wrote:
>> Jerry,
>>
>>
>> On 15.08.2018 18:14, Jerry Malcolm wrote:
>>> I have a mobile app that issues several http web service calls
>>> to initialize. I was making them sequentially with no issues.
>>> I then changed to give them all separate threads so they could
>>> load asynchronously. Then the bottom fell out. I started
>>> getting empty responses and some responses with results of
>>> three or four of the calls concatenated. I traced the problem
>>> from the app back through apache through mod_jk and found the
>>> culprit to be Tomcat.
>>>
>>> I'm a seasoned Tomcat developer for over 15 years. I've never
>>> seen anything like this. But it's really scary. What I found
>>> is that sometime during the execution of one servlet call, it's
>>> thread data is swapped to thread data of another servlet call.
>>> I know this sounds like Twilight Zone. But here is a log
>>> output. At the beginning of doGet(), I generated a random text
>>> string just to keep track of the thread data/:/
>>>
>>> Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread:
>>> ajp-nio-8009-exec-29 uid: ceycfqd --> Thread:
>>> ajp-nio-8009-exec-29 uid: ceycfqd <-- Thread:
>>> ajp-nio-8009-exec-24 uid: ceycfqd <--
>>>
>>> Note that when thread 24 starts I store the "rcl..." uid.
>>> Another call comes in and starts thread 29. By the time thread
>>> 24's servlet code is done, it now has thread 29's data. (The
>>> uid is just a quick variable for reference. The request
>>> object, response object, EVERYTHING is now thread 29's data).
>>>
>>> This explains why I'm getting empty responses and other
>>> response with the data for multiple requests concatenated
>>> together. The "rcl..." instance data has totally disappeared,
>>> and all of the server calls are now using the "cey..." instance
>>> data (i.e. response object).
>>>
>>> I figure this is some sort of timing/race condition that only
>>> occurs with a second call coming in while the first call is
>>> still in progress. I can go back to sending the mobile app
>>> calls serially and probably work around this. But this is a
>>> huge problem.
>>>
>>> Any ideas? (BTW... Tomcat 9.0.7)
>>>
>>
>> As we don't know which code generates this log output, it's hard
>> to judge what actually causes your problem. You say "thread data"
>> is being swapped, and the very first aspect that comes to my mind
>> is: Servlets are inherently multithreaded, and a common pattern
>> of bugs is if a servlet has a member variable that is used for
>> request processing: There typically is only one Servlet object
>> ever, thus they all share the same state, if the state is stored
>> in a member variable. This might be directly in the servlet or in
>> some other component or singleton somewhere.
>>
>> Any state and request processing must be done on the
>> request/response pair, and properly threadsafe in every other
>> part of your code.
>>
>> And most likely this is an issue that luckily shows up when
>> you're issuing a lot of parallel threads due to parallelizing one
>> client. It'd be a lot harder to reproduce if it were individual
>> users, who (very) occasionally see the wrong data. Consider
>> yourself lucky to have such a nice and reproducible issue.
>>
>> Olaf
>
> Olaf,
>
> I'm having a bit of trouble feeling "lucky" that I have a problem
> that is catastrophically blocking my development progress toward a
> deadline.
Uh hmm.
> I'm not sure what you mean by typically there is only one servlet
> object. There's one class. But a new instance is created on each
> request, right? And all instance variables should be scoped to
> that instance alone, right?
Nope. I think that's where everything you're observing is coming from,
then.
> It's fundamental to Java that a private local variable can't
> arbitrarily change to another value in the middle of executing a
> method other than by the local code changing it itself, right?
Since servlet instances are shared, so are all those private variable
references.
> Yet uid (and all the other variables including request and
> response) changed.. somewhere between the beginning and ending of
> doGet(). !! I'm not doing any singleton stuff. Simply
> request-->Servlet-->response.
So, that's a scary thing or me to hear: "and all the other variables
including request and response". Are you asking about the parameters
to the e.g. doGet() method, or are you saying that you also have
"private HttpServletRequest request" defined in your servlet and you
are somewhere doing this:
public void doGet(HttpServletRequest request, HttpServletResponse
response) {
this.request = request;
this.response = response;
... other stuff
}
?
> My servlet code is very basic. It is not multithreaded.
>
> public class MyServlet() { private String uid;// <
> PRIVATE access!!
Yes,
Re: Servlet Threads Changing Instance Data
Am 2018-08-15 um 22:00 schrieb Jerry Malcolm:
On 8/15/2018 1:50 PM, Olaf Kock wrote:
Jerry,
On 15.08.2018 18:14, Jerry Malcolm wrote:
I have a mobile app that issues several http web service calls to
initialize. I was making them sequentially with no issues. I then
changed to give them all separate threads so they could load
asynchronously. Then the bottom fell out. I started getting empty
responses and some responses with results of three or four of the
calls concatenated. I traced the problem from the app back through
apache through mod_jk and found the culprit to be Tomcat.
I'm a seasoned Tomcat developer for over 15 years. I've never seen
anything like this. But it's really scary. What I found is that
sometime during the execution of one servlet call, it's thread data
is swapped to thread data of another servlet call. I know this sounds
like Twilight Zone. But here is a log output. At the beginning of
doGet(), I generated a random text string just to keep track of the
thread data/:/
Thread: ajp-nio-8009-exec-24 uid: rclebgb -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd <--
Thread: ajp-nio-8009-exec-24 uid: ceycfqd <--
Note that when thread 24 starts I store the "rcl..." uid. Another
call comes in and starts thread 29. By the time thread 24's servlet
code is done, it now has thread 29's data. (The uid is just a quick
variable for reference. The request object, response object,
EVERYTHING is now thread 29's data).
This explains why I'm getting empty responses and other response with
the data for multiple requests concatenated together. The "rcl..."
instance data has totally disappeared, and all of the server calls
are now using the "cey..." instance data (i.e. response object).
I figure this is some sort of timing/race condition that only occurs
with a second call coming in while the first call is still in
progress. I can go back to sending the mobile app calls serially and
probably work around this. But this is a huge problem.
Any ideas? (BTW... Tomcat 9.0.7)
As we don't know which code generates this log output, it's hard to
judge what actually causes your problem. You say "thread data" is
being swapped, and the very first aspect that comes to my mind is:
Servlets are inherently multithreaded, and a common pattern of bugs is
if a servlet has a member variable that is used for request
processing: There typically is only one Servlet object ever, thus they
all share the same state, if the state is stored in a member variable.
This might be directly in the servlet or in some other component or
singleton somewhere.
Any state and request processing must be done on the request/response
pair, and properly threadsafe in every other part of your code.
And most likely this is an issue that luckily shows up when you're
issuing a lot of parallel threads due to parallelizing one client.
It'd be a lot harder to reproduce if it were individual users, who
(very) occasionally see the wrong data. Consider yourself lucky to
have such a nice and reproducible issue.
Olaf
Olaf,
I'm having a bit of trouble feeling "lucky" that I have a problem that
is catastrophically blocking my development progress toward a deadline.
I'm not sure what you mean by typically there is only one servlet
object. There's one class. But a new instance is created on each
request, right? And all instance variables should be scoped to that
instance alone, right? It's fundamental to Java that a private local
variable can't arbitrarily change to another value in the middle of
executing a method other than by the local code changing it itself,
right? Yet uid (and all the other variables including request and
response) changed.. somewhere between the beginning and ending of
doGet(). !! I'm not doing any singleton stuff. Simply
request-->Servlet-->response.
My servlet code is very basic. It is not multithreaded.
public class MyServlet() {
private String uid; // < PRIVATE access!!
Oh hell, this is so wrong. The servlet instance exists only once in the
webapp classloader. No one is creating a new instace on each an every
request. You *cannot* share a variable like that, it is not threadsafe.
This is your problem. You have to fix that. You also *must* review the
rest of your code. Here is your exact problem:
https://stackoverflow.com/a/2184147/696632. Trust Bauke Scholz!
Michael
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Servlet Threads Changing Instance Data
> From: Jerry Malcolm [mailto:techst...@malcolms.com] > Subject: Re: Servlet Threads Changing Instance Data > I'm not sure what you mean by typically there is only one servlet > object. There's one class. But a new instance is created on each > request, right? No - there's only one instance of each defined servlet. All requests for a particular servlet share that instance. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. smime.p7s Description: S/MIME cryptographic signature
Re: Servlet Threads Changing Instance Data
On 8/15/2018 1:50 PM, Olaf Kock wrote:
Jerry,
On 15.08.2018 18:14, Jerry Malcolm wrote:
I have a mobile app that issues several http web service calls to
initialize. I was making them sequentially with no issues. I then
changed to give them all separate threads so they could load
asynchronously. Then the bottom fell out. I started getting empty
responses and some responses with results of three or four of the
calls concatenated. I traced the problem from the app back through
apache through mod_jk and found the culprit to be Tomcat.
I'm a seasoned Tomcat developer for over 15 years. I've never seen
anything like this. But it's really scary. What I found is that
sometime during the execution of one servlet call, it's thread data
is swapped to thread data of another servlet call. I know this sounds
like Twilight Zone. But here is a log output. At the beginning of
doGet(), I generated a random text string just to keep track of the
thread data/:/
Thread: ajp-nio-8009-exec-24 uid: rclebgb -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd <--
Thread: ajp-nio-8009-exec-24 uid: ceycfqd <--
Note that when thread 24 starts I store the "rcl..." uid. Another
call comes in and starts thread 29. By the time thread 24's servlet
code is done, it now has thread 29's data. (The uid is just a quick
variable for reference. The request object, response object,
EVERYTHING is now thread 29's data).
This explains why I'm getting empty responses and other response with
the data for multiple requests concatenated together. The "rcl..."
instance data has totally disappeared, and all of the server calls
are now using the "cey..." instance data (i.e. response object).
I figure this is some sort of timing/race condition that only occurs
with a second call coming in while the first call is still in
progress. I can go back to sending the mobile app calls serially and
probably work around this. But this is a huge problem.
Any ideas? (BTW... Tomcat 9.0.7)
As we don't know which code generates this log output, it's hard to
judge what actually causes your problem. You say "thread data" is
being swapped, and the very first aspect that comes to my mind is:
Servlets are inherently multithreaded, and a common pattern of bugs is
if a servlet has a member variable that is used for request
processing: There typically is only one Servlet object ever, thus they
all share the same state, if the state is stored in a member variable.
This might be directly in the servlet or in some other component or
singleton somewhere.
Any state and request processing must be done on the request/response
pair, and properly threadsafe in every other part of your code.
And most likely this is an issue that luckily shows up when you're
issuing a lot of parallel threads due to parallelizing one client.
It'd be a lot harder to reproduce if it were individual users, who
(very) occasionally see the wrong data. Consider yourself lucky to
have such a nice and reproducible issue.
Olaf
Olaf,
I'm having a bit of trouble feeling "lucky" that I have a problem that
is catastrophically blocking my development progress toward a deadline.
I'm not sure what you mean by typically there is only one servlet
object. There's one class. But a new instance is created on each
request, right? And all instance variables should be scoped to that
instance alone, right? It's fundamental to Java that a private local
variable can't arbitrarily change to another value in the middle of
executing a method other than by the local code changing it itself,
right? Yet uid (and all the other variables including request and
response) changed.. somewhere between the beginning and ending of
doGet(). !! I'm not doing any singleton stuff. Simply
request-->Servlet-->response.
My servlet code is very basic. It is not multithreaded.
public class MyServlet() {
private String uid; // < PRIVATE access!!
doGet() {
uid = createUID();
System.out.println( "Thread: " + Thread.currentThread().getName() +
" uid: " + uid + " -->");
(no new threads, totally synchronous, simply get
a couple of vars from a JDBC call)
System.out.println( "Thread: " + Thread.currentThread().getName() +
" uid: " + uid + " <--");
}
That's it. I can think of nothing I could be doing that would cause the
PRIVATE uid varible to be different on the 2nd println from the 1st
println. And furthermore, the 2nd println has a uid from a DIFFERENT
request.
What could I possibly be doing in this code above that could cause this?
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat Clustering Support
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Scott, This list is typically for non-paid support requests, but there really isn't a great forum for finding Tomcat consultants... other than Google I suppose. So I'm marking this Off-Topic and replying below. On 8/15/18 3:43 PM, Scott Evans wrote: > Our system is on Apache Tomcat Version 8.0.47. OS is Windows Server > 2012 R2 Datacenter. > > We are looking for someone that may be interested in paid contract > work to assist with troubleshooting and resolving a Tomcat > clustering issue in our system. > > The system is composed of multiple Java PrimeFaces applications > running in a clustered Tomcat environment which is experiencing > occasional deadlocking issues from an unknown source requiring the > Nodes to be cycled in order to resolve. The issue is only > occurring in our Production environment and we've determined that > the issues are occurring at random with the replication threads. Is it feasible to disable clustering (session replication) completely in your environment to determine whether the replication itself is the problem, or perhaps the problem is elsewhere? Disabling clustering should only degrade your service if (a) you aren't using session-stickiness (which pretty much everyone should be using) and (b) you expect a lot of fail-over and (c) it is unacceptable to have a failed-over user lose their session information. > We would need someone to help investigate our configuration and > determine if there are any further changes that can be made to our > system to catch these deadlock issues before they occur (requiring > a Node cycle). Can you post your cluster configuration from conf/server.xml, minus any secrets that may be in there? If all configs are the same on each node except for e.g. IP lists, feel free to just post a single one and mention that the IPs are different. It would be good to know how many nodes are clustered and the replication strategy (e.g. delta versus back). The strategy will be obvious from the configuration. You'll need to tell us the size of the cluster... and perhaps the deployment model such as "all servers in the same DC" or "50/50 split between primary DC and secondary DC", etc. > Please let me know if you or someone you know may be interested or > if you have further questions I can help answer. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlt0hSUACgkQHPApP6U8 pFiGPg/+PdAG+pVDfZzPWQRCBfjbaSQaRwum7LDNtBY+w5xQjfuS1t3UmST678ft RGaMRP5Qm/TYuMBI9mzdXrrRIAFaRI/3QAuuyB0jdzKqn18/6fldoJKHpwlzm27x SE1r5R4tD/lihC1lFWfXEMzOosO2uk7iBZWUv532zKL7TJ0lrLNgVdD5eakX9iVE dMpnYMbK9CtWHzQZ0LOYcHlXrnAYcr3OqYxZomHCpYsRjHEndfS8gsWqY1t/+gA0 xl0/Vz4lWXYhjmC/PVyN21LL21PA7MNFuywFmt7Xw4sxzgtHXorkOdJcTaqsoz+3 qqItb91vEQ4MvqjW949NApgV1bTaH/juEx8Z99fe9AEAireBCQ6q3qRdvWVzYNqP +WE0Ghxv5400h1VPYiYoDkFrJUnlPeVortqn1OldEVqoAfQGFW1Gt9wgpEK7Snfx EVGYG8alAwuvd5sFR9Ge6FcY8NTp+9/awbCwFAPmCCW6dlwdRmYhCRa8tJ9K6MZn PeeTPFXqBny1HS3BGR5owDn87Mv97eDxTNDfBjAxJP4u4DcMmtvSvM1Uk5QkkI4z A+rivKxqNw60qR96njWJOyi6A4Nk1OpTSN541snEAegnNm7ad9SMSot349mk80a9 culdRnzLlMCNdv0GwYpoCZBESOckFRQ2G+XrMNqnaCpNVTfCUyo= =cPO2 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Clustering Support
On 15/08/18 20:43, Scott Evans wrote: > Hi, > > Our system is on Apache Tomcat Version 8.0.47. > OS is Windows Server 2012 R2 Datacenter. > > We are looking for someone that may be interested in paid contract work to > assist with troubleshooting and resolving a Tomcat clustering issue in our > system. > > The system is composed of multiple Java PrimeFaces applications running in > a clustered Tomcat environment which is experiencing occasional > deadlocking issues from an unknown source requiring the Nodes to be cycled > in order to resolve. The issue is only occurring in our Production > environment and we've determined that the issues are occurring at random > with the replication threads. > > We would need someone to help investigate our configuration and determine > if there are any further changes that can be made to our system to catch > these deadlock issues before they occur (requiring a Node cycle). > > Please let me know if you or someone you know may be interested or if you > have further questions I can help answer. If you can provide a thread dump of the deadlock when it occurs we can probably help you here for free. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Servlet Threads Changing Instance Data
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jerry, On 8/15/18 12:14 PM, Jerry Malcolm wrote: > I have a mobile app that issues several http web service calls to > initialize. I was making them sequentially with no issues. I > then changed to give them all separate threads so they could load > asynchronously. Then the bottom fell out. I started getting > empty responses and some responses with results of three or four of > the calls concatenated. I traced the problem from the app back > through apache through mod_jk and found the culprit to be Tomcat. > > I'm a seasoned Tomcat developer for over 15 years. I've never > seen anything like this. But it's really scary. What I found is > that sometime during the execution of one servlet call, it's thread > data is swapped to thread data of another servlet call. I know > this sounds like Twilight Zone. But here is a log output. At the > beginning of doGet(), I generated a random text string just to keep > track of the thread data/:/ > > Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread: > ajp-nio-8009-exec-29 uid: ceycfqd --> Thread: ajp-nio-8009-exec-29 > uid: ceycfqd <-- Thread: ajp-nio-8009-exec-24 uid: ceycfqd <-- > > Note that when thread 24 starts I store the "rcl..." uid. Another > call comes in and starts thread 29. By the time thread 24's > servlet code is done, it now has thread 29's data. (The uid is > just a quick variable for reference. The request object, response > object, EVERYTHING is now thread 29's data). > > This explains why I'm getting empty responses and other response > with the data for multiple requests concatenated together. The > "rcl..." instance data has totally disappeared, and all of the > server calls are now using the "cey..." instance data (i.e. > response object). > > I figure this is some sort of timing/race condition that only > occurs with a second call coming in while the first call is still > in progress. I can go back to sending the mobile app calls serially > and probably work around this. But this is a huge problem. > > Any ideas? (BTW... Tomcat 9.0.7) Some more specific questions thatn what Olaf has already asked: 1. Where is your "uid" being stored? 2. What does your code look like? Can you just post some of it? We really don't care what your application is doing, of course. Just the parts that generate+log the "uids" you are showing to us. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlt0g7gACgkQHPApP6U8 pFjxcQ/9GdFR302AfkCzCW2WDLqOYlcV/WqGHv2qIq3G5Eyv6b8fRHlqncg6QGbv JZOMT1DHvfWYnMZ/XOcVbF1AdRfpCx4TUPuuaw4YoeS4kqghGKM3L545+Uavm5Qi 8I1GR5NL87fpFExifWrjqVgDIWGXmsHHG0TPwyWq16ALJ8IQ0jStbGdft82AEZeF jUkbt2/+aDvKsGJhqROQ/Z+QnA+F9u6NW2vZTP2zhb2S9Max1L/MHySx3WLZv4aN 72sS592+KquSMOw1068Wj+BdsIN6ag6HOaNc/o3WCRivcLd/bQbZ4+v8UoegruRW DK57voXd28lMBZI0SOU4sCLJpRctdprA8+zIsEiHKx4ln6UD+SHg3C9MGHhED5vp zRI6dFTcuzoQ3f6/LTp3zNXqBsxTWPyF7C82CphRP0BOfQT526a4D0GkAO4BpdAU Xm9tEpy1jQEyLplOWRctrSdy/e1uaqlJ3F+3/dkZEn+zfV9hpnMQ76BS8Mrei+Ke aH8K5f6Ahak4iaey69nUKe8vVhGHex3uq8oo94Ud+mMVlSeWNFvcY8Kb7ByjAcUb z3SsqTsGRHcmdJGg3Ku3FNpLEi9HkdZaAhGxENBg1lOd0oKHOEE/fsvnB83tartc Aqjan2IdjHtil5vYRiQQqqmZ372KLIpxTfle1cj0PoJgd/P52DI= =tZjO -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat Clustering Support
Hi, Our system is on Apache Tomcat Version 8.0.47. OS is Windows Server 2012 R2 Datacenter. We are looking for someone that may be interested in paid contract work to assist with troubleshooting and resolving a Tomcat clustering issue in our system. The system is composed of multiple Java PrimeFaces applications running in a clustered Tomcat environment which is experiencing occasional deadlocking issues from an unknown source requiring the Nodes to be cycled in order to resolve. The issue is only occurring in our Production environment and we've determined that the issues are occurring at random with the replication threads. We would need someone to help investigate our configuration and determine if there are any further changes that can be made to our system to catch these deadlock issues before they occur (requiring a Node cycle). Please let me know if you or someone you know may be interested or if you have further questions I can help answer. Thanks, Scott - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Servlet Threads Changing Instance Data
Jerry, On 15.08.2018 18:14, Jerry Malcolm wrote: I have a mobile app that issues several http web service calls to initialize. I was making them sequentially with no issues. I then changed to give them all separate threads so they could load asynchronously. Then the bottom fell out. I started getting empty responses and some responses with results of three or four of the calls concatenated. I traced the problem from the app back through apache through mod_jk and found the culprit to be Tomcat. I'm a seasoned Tomcat developer for over 15 years. I've never seen anything like this. But it's really scary. What I found is that sometime during the execution of one servlet call, it's thread data is swapped to thread data of another servlet call. I know this sounds like Twilight Zone. But here is a log output. At the beginning of doGet(), I generated a random text string just to keep track of the thread data/:/ Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd <-- Thread: ajp-nio-8009-exec-24 uid: ceycfqd <-- Note that when thread 24 starts I store the "rcl..." uid. Another call comes in and starts thread 29. By the time thread 24's servlet code is done, it now has thread 29's data. (The uid is just a quick variable for reference. The request object, response object, EVERYTHING is now thread 29's data). This explains why I'm getting empty responses and other response with the data for multiple requests concatenated together. The "rcl..." instance data has totally disappeared, and all of the server calls are now using the "cey..." instance data (i.e. response object). I figure this is some sort of timing/race condition that only occurs with a second call coming in while the first call is still in progress. I can go back to sending the mobile app calls serially and probably work around this. But this is a huge problem. Any ideas? (BTW... Tomcat 9.0.7) As we don't know which code generates this log output, it's hard to judge what actually causes your problem. You say "thread data" is being swapped, and the very first aspect that comes to my mind is: Servlets are inherently multithreaded, and a common pattern of bugs is if a servlet has a member variable that is used for request processing: There typically is only one Servlet object ever, thus they all share the same state, if the state is stored in a member variable. This might be directly in the servlet or in some other component or singleton somewhere. Any state and request processing must be done on the request/response pair, and properly threadsafe in every other part of your code. And most likely this is an issue that luckily shows up when you're issuing a lot of parallel threads due to parallelizing one client. It'd be a lot harder to reproduce if it were individual users, who (very) occasionally see the wrong data. Consider yourself lucky to have such a nice and reproducible issue. Olaf - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Servlet Threads Changing Instance Data
I have a mobile app that issues several http web service calls to initialize. I was making them sequentially with no issues. I then changed to give them all separate threads so they could load asynchronously. Then the bottom fell out. I started getting empty responses and some responses with results of three or four of the calls concatenated. I traced the problem from the app back through apache through mod_jk and found the culprit to be Tomcat. I'm a seasoned Tomcat developer for over 15 years. I've never seen anything like this. But it's really scary. What I found is that sometime during the execution of one servlet call, it's thread data is swapped to thread data of another servlet call. I know this sounds like Twilight Zone. But here is a log output. At the beginning of doGet(), I generated a random text string just to keep track of the thread data/:/ Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd <-- Thread: ajp-nio-8009-exec-24 uid: ceycfqd <-- Note that when thread 24 starts I store the "rcl..." uid. Another call comes in and starts thread 29. By the time thread 24's servlet code is done, it now has thread 29's data. (The uid is just a quick variable for reference. The request object, response object, EVERYTHING is now thread 29's data). This explains why I'm getting empty responses and other response with the data for multiple requests concatenated together. The "rcl..." instance data has totally disappeared, and all of the server calls are now using the "cey..." instance data (i.e. response object). I figure this is some sort of timing/race condition that only occurs with a second call coming in while the first call is still in progress. I can go back to sending the mobile app calls serially and probably work around this. But this is a huge problem. Any ideas? (BTW... Tomcat 9.0.7) Jerry/ / - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Question about setting CATALINA_OPTS when starting Tomcat using a Windows Service in Tomcat 7.0.54
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Louis, On 8/15/18 7:58 AM, Louis Zipes wrote: > One correction. I meant to say that I'm using 8.5 (put my zero in > the wrong place) Glad you are upgrading. 8.5.x is mature enough at this point that there should be a remediation for any incompatibility you may find with previous versions of Tomcat. Moving to 9.0 may require some changes to the application because of stricter-adherence to various standards and the desire of the Tomcat team to finally move-on from backward-compatible hacks, etc. for those things. I would recommend that as soon as you get your application running properly under Tomcat 8.5.x you immediately begin testing it with Tomcat 9.0.x in order to future-proof said application. Any changes you make to fix issues with Tomcat 9.0.x /should/ be equally valid in Tomcat 8.5.x, but you'll also be ready for another upgrade in a year or so (or whatever your "major container upgrade" schedule turns out to be). FWIW, at $work we are only upgrading to 8.5.x starting this fall. > and yes, looking at the GUI for the Tomcat8w in the 'Start' and > 'Stop' tabs it does indeed say JVM and it runs with no 32 bit > error message like the packaged Tomcat 7.0.54 that came packaged > with my third party app. So the switch from "java" -> "jvm" is very likely the only relevant change, here. > Thanks again. There are a lot of articles on Stack Exchange that > would benefit from this additional information! Well, now you are an expert and you can enlighten everybody :) - -chris > > -Original Message- From: Christopher Schultz > [mailto:ch...@christopherschultz.net] Sent: Tuesday, August 14, > 2018 6:41 PM To: users@tomcat.apache.org Subject: Re: Question > about setting CATALINA_OPTS when starting Tomcat using a Windows > Service in Tomcat 7.0.54 > > - - - external message, proceed with caution - - - > > > Louis, > > On 8/14/18 3:28 PM, Louis Zipes wrote: >> Hi all, Just wanted to circle back. There was an early comment >> (maybe even in the first response to my question/problem) where >> someone mentioned that my set up (Running Tomcat as a Windows >> Service and putting the JMX parameters directly in the >> Tomcat7.0.54 GUI in the Java tab) SHOULD work and should startup >> and shutdown gracefully BUT that if it doesn't then try one of >> the of the later versions of Tomcat. > >> I did finally get a chance to try Tomcat 8.0.5X and it does seem >> to work with no other configuration changes. I can access JMX >> (JConsole) and start and shut down the Windows Service running >> Tomcat with no issues. Although, now my problem is that my >> third party application, that is running doesn't work, but that >> is not a problem for this mailing list. > >> So I think we can say that in the end the upgrade to a higher >> version resolved the issue. Thank you to all that contributed >> input! > > It's very possible that, if you used the service-installer, it > simply created a new service that uses the "jvm" launch-strategy. > > I'd be interested to see if that's the case. > > Before you spend a lot of time tracking-down the application > incompatabilities with 8.0.x, you might want to upgrade to 9.0.x > or 8.5.x and start there. Tomcat 8.0.x has reached EOL so it's > probably a waste of your time to test against it. > > Hope that helps, -chris > >> -Original Message- From: André Warnier (tomcat) >> [mailto:a...@ice-sa.com] Sent: Thursday, August 09, 2018 12:40 PM >> To: users@tomcat.apache.org Subject: Re: Question about setting >> CATALINA_OPTS when starting Tomcat using a Windows Service in >> Tomcat 7.0.54 > >> - - - external message, proceed with caution - - - > > >> Maybe it is time here to quote Arthur Clarke's 3rd law : "Any >> sufficiently advanced technology is indistinguishable from >> magic" (See : >> https://en.wikipedia.org/wiki/Clarke%27s_three_laws) > >> The process by which Tomcat is started and/or stopped - >> particularly under Windows and as a Service - is not very clear >> in the on-line documentation. Neither is it it very easy to write >> a comprehensive and accurate documentation, because the thing >> has gotten to a point where, for mere mortals, it is really >> quite complicated. (Have a look at bin/catalina.bat to get an >> idea). > >> So let me give you some overall pointers (some of them quite >> basic, I apologise), and maybe in there somewhere, you'll find >> wat you are missing to complete the picture and do what you want >> to do. > >> 1) Tomcat is a compiled java application, in java bytecode. To >> run this bytecode, you need a JVM. The JVM is machine-executable >> code, so to run tomcat, you run a JVM and tell it to run the >> tomcat bytecode. 2) the java JVM for Windows is not very good at >> running as a Windows Service (it does not handle the appropriate >> Windows "signals" etc.). To solve this, when you want to run >> tomcat as a Windows Service (or rather - see
RE: Question about setting CATALINA_OPTS when starting Tomcat using a Windows Service in Tomcat 7.0.54
One correction. I meant to say that I'm using 8.5 (put my zero in the wrong place) and yes, looking at the GUI for the Tomcat8w in the 'Start' and 'Stop' tabs it does indeed say JVM and it runs with no 32 bit error message like the packaged Tomcat 7.0.54 that came packaged with my third party app. Thanks again. There are a lot of articles on Stack Exchange that would benefit from this additional information! -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, August 14, 2018 6:41 PM To: users@tomcat.apache.org Subject: Re: Question about setting CATALINA_OPTS when starting Tomcat using a Windows Service in Tomcat 7.0.54 - - - external message, proceed with caution - - - -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Louis, On 8/14/18 3:28 PM, Louis Zipes wrote: > Hi all, Just wanted to circle back. There was an early comment > (maybe even in the first response to my question/problem) where > someone mentioned that my set up (Running Tomcat as a Windows > Service and putting the JMX parameters directly in the > Tomcat7.0.54 GUI in the Java tab) SHOULD work and should startup > and shutdown gracefully BUT that if it doesn't then try one of the > of the later versions of Tomcat. > > I did finally get a chance to try Tomcat 8.0.5X and it does seem to > work with no other configuration changes. I can access JMX > (JConsole) and start and shut down the Windows Service running > Tomcat with no issues. Although, now my problem is that my third > party application, that is running doesn't work, but that is not > a problem for this mailing list. > > So I think we can say that in the end the upgrade to a higher > version resolved the issue. Thank you to all that contributed > input! It's very possible that, if you used the service-installer, it simply created a new service that uses the "jvm" launch-strategy. I'd be interested to see if that's the case. Before you spend a lot of time tracking-down the application incompatabilities with 8.0.x, you might want to upgrade to 9.0.x or 8.5.x and start there. Tomcat 8.0.x has reached EOL so it's probably a waste of your time to test against it. Hope that helps, - -chris > -Original Message- From: André Warnier (tomcat) > [mailto:a...@ice-sa.com] Sent: Thursday, August 09, 2018 12:40 PM To: > users@tomcat.apache.org Subject: Re: Question about setting > CATALINA_OPTS when starting Tomcat using a Windows Service in > Tomcat 7.0.54 > > - - - external message, proceed with caution - - - > > > Maybe it is time here to quote Arthur Clarke's 3rd law : "Any > sufficiently advanced technology is indistinguishable from magic" > (See : https://en.wikipedia.org/wiki/Clarke%27s_three_laws) > > The process by which Tomcat is started and/or stopped - > particularly under Windows and as a Service - is not very clear in > the on-line documentation. Neither is it it very easy to write a > comprehensive and accurate documentation, because the thing has > gotten to a point where, for mere mortals, it is really quite > complicated. (Have a look at bin/catalina.bat to get an idea). > > So let me give you some overall pointers (some of them quite basic, > I apologise), and maybe in there somewhere, you'll find wat you are > missing to complete the picture and do what you want to do. > > 1) Tomcat is a compiled java application, in java bytecode. To run > this bytecode, you need a JVM. The JVM is machine-executable code, > so to run tomcat, you run a JVM and tell it to run the tomcat > bytecode. 2) the java JVM for Windows is not very good at running > as a Windows Service (it does not handle the appropriate Windows > "signals" etc.). To solve this, when you want to run tomcat as a > Windows Service (or rather - see above - run the JVM as a Windows > Service), you actually run a specialised "wrapper program" which > does work well as a Windows Service, and you ask this wrapper to > start the JVM which runs tomcat. To make matters a bit more > confusing (or maybe, for some, clearer), this generic "Windows > Service JVM wrapper" is renamed to "tomcatV.exe" (where V is the > tomcat version, so for tomcat 9, the program is called > tomcat9.exe). 3) the wrapper program, when it starts the JVM, has > to know which command-line switches it should pass to it. For the > Windows Service flavor of tomcat, these parameters are stored in a > number of special keys in the Windows Registry, and that is where > the wrapper picks them up, before starting the JVM. 4) To make it > easier to set and edit these JVM command-line parameters, tomcat > provides another Windows executable program - a specialised GUI > Registry Editor - which is also renamed according to the tomcat > version, as tomcatVw.exe (where V is the tomcat version, so for > tomcat 9 it would be tomcat9w.exe). > > 5) as a separate bit of knowledge, I would suppose that everyone > knows that on any given host, a given TCP listening port can only > be opened
