RE: SEVERE message from DeltaManager
Thankyou very much for your diagnosis here, Mark. I will investigate the proposed solution and let you know how it goes. Cheers, Matt. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Tuesday, 20 July 2010 3:07 AM To: Tomcat Users List Subject: Re: SEVERE message from DeltaManager On 16/07/2010 10:19, Mark Thomas wrote: On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Matt, Testing shows that it is caused by using async session replication. If you use synchronous replication that ensures messages are processed on the receiving nodes in the order they are sent. Asynchronous replication in conjunction with the fact the the receiving node uses a thread pool to process messages means that it is possible for messages to be processed out of sequence. If a session invalidate is processed before and update then you'll see this error. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
May be, but i do notthink so. This errors : SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang. IllegalStateException: removeAttribute: Session already invalidated may dependds on tomcat failure and i see in the Receiver configuration: Receiver ... maxThreads=6/ 2010/7/16 Mark Thomas ma...@apache.org On 16/07/2010 14:50, Luciano Fioriti wrote: I had done a stress test with 20 contemporary request to a web app using tomcat6 6bit. When maxThreads=200 tomcat gave response to only some request then went to 0 cpu time no more response, nothing on log file. This happenend only on contemporary requests (max threads reched i presume). Increasing maxThreads to 250 all request were satisfied. Which has absolutely nothing to do with the error Matt is reporting. There are various things that look odd in the statements above but this thread isn't the place to go into that. Mark Lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 14:19, Luciano Fioriti wrote: It may be caused by maxThreads parameter in http connector, try to increase On what basis are you reaching that conclusion? Mark lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
On 19/07/2010 10:50, Luciano Fioriti wrote: May be, but i do notthink so. Again, you are way off-base here. Hitting the max threads in the connector has nothing to do with errors in session replication. Mark This errors : SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang. IllegalStateException: removeAttribute: Session already invalidated may dependds on tomcat failure and i see in the Receiver configuration: Receiver ... maxThreads=6/ 2010/7/16 Mark Thomas ma...@apache.org On 16/07/2010 14:50, Luciano Fioriti wrote: I had done a stress test with 20 contemporary request to a web app using tomcat6 6bit. When maxThreads=200 tomcat gave response to only some request then went to 0 cpu time no more response, nothing on log file. This happenend only on contemporary requests (max threads reched i presume). Increasing maxThreads to 250 all request were satisfied. Which has absolutely nothing to do with the error Matt is reporting. There are various things that look odd in the statements above but this thread isn't the place to go into that. Mark Lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 14:19, Luciano Fioriti wrote: It may be caused by maxThreads parameter in http connector, try to increase On what basis are you reaching that conclusion? Mark lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
On 16/07/2010 10:19, Mark Thomas wrote: On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Matt, Testing shows that it is caused by using async session replication. If you use synchronous replication that ensures messages are processed on the receiving nodes in the order they are sent. Asynchronous replication in conjunction with the fact the the receiving node uses a thread pool to process messages means that it is possible for messages to be processed out of sequence. If a session invalidate is processed before and update then you'll see this error. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 7/19/2010 1:07 PM, Mark Thomas wrote: Testing shows that it is caused by using async session replication. If you use synchronous replication that ensures messages are processed on the receiving nodes in the order they are sent. Asynchronous replication in conjunction with the fact the the receiving node uses a thread pool to process messages means that it is possible for messages to be processed out of sequence. If a session invalidate is processed before and update then you'll see this error. I am by no means an expert on Tomcat clustering (nor any other container clustering for that matter), but it seems to me that a potential solution would be to use a replication queue something like this: Client event - server queue - event handled ^^ server thread poolserver replication thread pool If the queue were synchronized and a priority queue (based upon timestamp) or queueing were guaranteed on the basis of event reception, then the use of asynchronous versus synchronous replication should always result in deterministic replication behavior. I'm interested to know why synchronous replication solves this problem... is it a client event queuing problem when asynchronous replication is used? Or does either the client or the server serialize processing of replication events in some way only when synchronous replication is being used? Thanks, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxExXAACgkQ9CaO5/Lv0PDeegCgvZeiKaApROcZguagjOmEcDhB GbcAnjg48ECuBvfYBRau0hMjnZlyYYi3 =UnUy -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
On 19/07/2010 22:36, Christopher Schultz wrote: Mark, On 7/19/2010 1:07 PM, Mark Thomas wrote: Testing shows that it is caused by using async session replication. If you use synchronous replication that ensures messages are processed on the receiving nodes in the order they are sent. Asynchronous replication in conjunction with the fact the the receiving node uses a thread pool to process messages means that it is possible for messages to be processed out of sequence. If a session invalidate is processed before and update then you'll see this error. I am by no means an expert on Tomcat clustering (nor any other container clustering for that matter), but it seems to me that a potential solution would be to use a replication queue something like this: Client event - server queue - event handled ^^ server thread poolserver replication thread pool If the queue were synchronized and a priority queue (based upon timestamp) or queueing were guaranteed on the basis of event reception, then the use of asynchronous versus synchronous replication should always result in deterministic replication behavior. Not sure what you are proposing here. Async replication means that multiple messages for a single session may be sent from node A before node B has a chance to process them. The issue is that if multiple session update messages from node A are in node B's input queue and B processes that queue with multiple threads, it is possible - depending on thread prioritisation - for node B to process the messages out of order even if node B removes them from its queue in order. Fixing it would require limiting node B to processing a single message per session from node A at a time. (assuming messages were send and received in order). I think this comes down to a simple trade-off between performance (async) and deterministic (sync) behaviour. I think the constraints you would have to impose to make the async deterministic would remove most of the benefits. As an aside, if a UA sends multiple requests in parallel to a single node then you have little control of the order that events will occur in the session anyway. I'm interested to know why synchronous replication solves this problem... is it a client event queuing problem when asynchronous replication is used? Or does either the client or the server serialize processing of replication events in some way only when synchronous replication is being used? Synchronous replication essentially ensures that node B only ever has at most one message from node A for a single session and therefore has to process them in order. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark, On 7/19/2010 6:00 PM, Mark Thomas wrote: On 19/07/2010 22:36, Christopher Schultz wrote: Mark, On 7/19/2010 1:07 PM, Mark Thomas wrote: Testing shows that it is caused by using async session replication. If you use synchronous replication that ensures messages are processed on the receiving nodes in the order they are sent. Asynchronous replication in conjunction with the fact the the receiving node uses a thread pool to process messages means that it is possible for messages to be processed out of sequence. If a session invalidate is processed before and update then you'll see this error. I am by no means an expert on Tomcat clustering (nor any other container clustering for that matter), but it seems to me that a potential solution would be to use a replication queue something like this: Client event - server queue - event handled ^^ server thread poolserver replication thread pool If the queue were synchronized and a priority queue (based upon timestamp) or queueing were guaranteed on the basis of event reception, then the use of asynchronous versus synchronous replication should always result in deterministic replication behavior. Not sure what you are proposing here. Async replication means that multiple messages for a single session may be sent from node A before node B has a chance to process them. Duh. No amount of queuing can fix that. I obviously wasn't thinking very clearly... I was thinking about a server (meaning the receiving node) processing the events out-of-order after it had received them (or as part of a non-deterministic event-processing thread priority) rather than the client failing to send them in a timely manner. I think this comes down to a simple trade-off between performance (async) and deterministic (sync) behaviour. I think the constraints you would have to impose to make the async deterministic would remove most of the benefits. Agreed. Something that could be done is that this error message might be demoted from SEVERE to WARNING if async replication is being used. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxEzIwACgkQ9CaO5/Lv0PDjogCeNhn487GRHsNUG4ys1FA7Gsss VNAAnibzqtKWhVx7GSw1wNbqj9aMM4XK =q+n/ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
It may be caused by maxThreads parameter in http connector, try to increase lucio 2010/7/16 Mark Thomas ma...@apache.org On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
On 16/07/2010 14:19, Luciano Fioriti wrote: It may be caused by maxThreads parameter in http connector, try to increase On what basis are you reaching that conclusion? Mark lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
I had done a stress test with 20 contemporary request to a web app using tomcat6 6bit. When maxThreads=200 tomcat gave response to only some request then went to 0 cpu time no more response, nothing on log file. This happenend only on contemporary requests (max threads reched i presume). Increasing maxThreads to 250 all request were satisfied. Lucio 2010/7/16 Mark Thomas ma...@apache.org On 16/07/2010 14:19, Luciano Fioriti wrote: It may be caused by maxThreads parameter in http connector, try to increase On what basis are you reaching that conclusion? Mark lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
could you someone please help me? I need to know what is the end of life for Apache tomcat 5.5.28 and 5.5.29. I would appreciate anyone's help Thank you, -Nejet On Fri, Jul 16, 2010 at 9:19 AM, Luciano Fioriti lu.fior...@gmail.comwrote: It may be caused by maxThreads parameter in http connector, try to increase lucio 2010/7/16 Mark Thomas ma...@apache.org On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
nejet erradi wrote: could you someone please help me? I need to know what is the end of life for Apache tomcat 5.5.28 and 5.5.29. I would appreciate anyone's help Thank you, -Nejet 1) do not hijack threads 2) see http://tomcat.apache.org/whichversion.html - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SEVERE message from DeltaManager
On 16/07/2010 14:50, Luciano Fioriti wrote: I had done a stress test with 20 contemporary request to a web app using tomcat6 6bit. When maxThreads=200 tomcat gave response to only some request then went to 0 cpu time no more response, nothing on log file. This happenend only on contemporary requests (max threads reched i presume). Increasing maxThreads to 250 all request were satisfied. Which has absolutely nothing to do with the error Matt is reporting. There are various things that look odd in the statements above but this thread isn't the place to go into that. Mark Lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 14:19, Luciano Fioriti wrote: It may be caused by maxThreads parameter in http connector, try to increase On what basis are you reaching that conclusion? Mark lucio 2010/7/16 Mark Thomasma...@apache.org On 16/07/2010 06:49, Matt Peterson wrote: While load testing our clustered Tomcats, we are seeing the following stack trace in our catalina.out occasionally, but not regularly: Jul 16, 2010 3:34:49 PM org.apache.catalina.ha.session.DeltaManager messageReceived SEVERE: Manager [localhost#/urs]: Unable to receive message through TCP channel java.lang.IllegalStateException: removeAttribute: Session already invalidated snip/ Under what conditions would this occur? Could it be that a session diff is being transmitted, but the session it relates to has been invalidated by the time the diff is processed (via a user logout for example)? Or could it be that a timeout has been reached??? Someone at $work has been doing a load test with tc Server (which has identical code to Tomcat in this area) and seen the same issue. I know it isn't due to timeout since the sessions are only a few seconds old when it happens. My current guess is that the messages are not being processed in the same order as they are sent. I need to dig into this more to figure out if this is a configuration issue or a bug. I did wonder if switching to channel send options 6 would fix it. I'll get them to try that and see. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
