Re: Tomcat under load frontend reverse proxy timeouts
Hi Igor, On 03/27/2014 10:51 PM, Igor Cicimov wrote: On 27/03/2014 12:39 PM, Frederik Nosi frederik.n...@postecom.it wrote: Hi all, Having to deal with slow applications deployed under tomcat, with a reverse proxy in front, frequently i've noticed that even when the frontend timeouts and closes it's part of the TCP connection, the Tomcat thread processing the request goes on and on till it finishes. Is there a way to make the proccessing thread stop when the frontend connection get's closed? Thanks in advance, Frederik What kind of application is this? What is the reason that the thread takes longer than expected? Unfortunately for me is legacy apps, sometimes writen from companies that never exist anymore, but i have to somehow make stuff work. I cant modify them. Is it just the load on the app server or its waiting on the backend database maybe? Yes, nearly all apps talk with a backend DB, but for that part knowing our DBA stuff and havnig access to the database servers i'm confident it is not due to the load on our databases. Is mostly due to the number of queries per request from the application side. And another problematic point is that some of this applications dont even use connection pooling. And most of this apps make calls to webservices outside my control, without having the possibility to configure a timeout. Load is an issue too, usualy CPU load not memory, so not connected to the garbage collector. We've tried scaling horizontaly, and there's where i noticed the amplification effect of mod_jk, i posted another mail on this issue and a patch, with subject: Re: mod_jk - Failover behaviour and load + patch but the thread went nowhere FWICS. If load then for sure you can limit the tomcat connector threads and the accept queue to reduce it and add some more app servers. If backend connection then you can sync the client timeout and the db connection timeout. See above. Thanks Igor, Frederik - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat under load frontend reverse proxy timeouts
On 27/03/2014 01:39, Frederik Nosi wrote: Hi all, Having to deal with slow applications deployed under tomcat, with a reverse proxy in front, frequently i've noticed that even when the frontend timeouts and closes it's part of the TCP connection, the Tomcat thread processing the request goes on and on till it finishes. Is there a way to make the proccessing thread stop when the frontend connection get's closed? No. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat under load frontend reverse proxy timeouts
First thanks for your reply, On 03/27/2014 10:32 AM, Mark Thomas wrote: On 27/03/2014 01:39, Frederik Nosi wrote: Hi all, Having to deal with slow applications deployed under tomcat, with a reverse proxy in front, frequently i've noticed that even when the frontend timeouts and closes it's part of the TCP connection, the Tomcat thread processing the request goes on and on till it finishes. Is there a way to make the proccessing thread stop when the frontend connection get's closed? No. Any hints on how to deal with situations like this? Mark Frederik. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat under load frontend reverse proxy timeouts
Frederik Nosi wrote: First thanks for your reply, On 03/27/2014 10:32 AM, Mark Thomas wrote: On 27/03/2014 01:39, Frederik Nosi wrote: Hi all, Having to deal with slow applications deployed under tomcat, with a reverse proxy in front, frequently i've noticed that even when the frontend timeouts and closes it's part of the TCP connection, the Tomcat thread processing the request goes on and on till it finishes. Is there a way to make the proccessing thread stop when the frontend connection get's closed? No. Any hints on how to deal with situations like this? This is basically an issue similar to the one that is the subject of the other message thread Re: Is it possible to send a 'keep-alive' packet back to client session every x seconds?. AFAIK, the only portable way to detect this situation, is by forcing the web application to regularly send some output to the client. If the connection has been closed, it will then (*) get an exception, which it can handle. It implies that the webapp itself is not blocked waiting on some separate resource, and can regularly break out of its own main processing to send such output, resuming its main work if the result is ok. (*) with possibly some delay before everything gets flushed all the way to the client connection, and the disconnected state makes its way back up the chain. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat under load frontend reverse proxy timeouts
Hi Andre', On 03/27/2014 02:37 PM, André Warnier wrote: Frederik Nosi wrote: First thanks for your reply, On 03/27/2014 10:32 AM, Mark Thomas wrote: On 27/03/2014 01:39, Frederik Nosi wrote: Hi all, Having to deal with slow applications deployed under tomcat, with a reverse proxy in front, frequently i've noticed that even when the frontend timeouts and closes it's part of the TCP connection, the Tomcat thread processing the request goes on and on till it finishes. Is there a way to make the proccessing thread stop when the frontend connection get's closed? No. Any hints on how to deal with situations like this? This is basically an issue similar to the one that is the subject of the other message thread Re: Is it possible to send a 'keep-alive' packet back to client session every x seconds?. Not exactly, it's the opposite scenario, the load balancer and the firewall / NAT part is fine. Probably my english is a bit poor. What i wanted to acchieve is, without having the possibility to modify the webapp, when the uplink request (this can be mod_jk/apache or user's browser, whatever calls the application) timeouts to find a way to stop the tomcat / webapp proccessing thread. Though from Mark's reply i understand that this is not possible with tomcat. So, at this point, having to deal with such webapps, is there some common solution? My problem is that the failover mecchanism (mod_jk in my case) during peak usage amplifies the load on the application servers, it's a spiral of death :-) How do you guys deal with this situations? AFAIK, the only portable way to detect this situation, is by forcing the web application to regularly send some output to the client. If the connection has been closed, it will then (*) get an exception, which it can handle. in mod_jk cping/cpong, or the TCP stack (keepalive tcp). But is not my scenario unfortunately. Anyway, thanks for your reply! It implies that the webapp itself is not blocked waiting on some separate resource, and can regularly break out of its own main processing to send such output, resuming its main work if the result is ok. (*) with possibly some delay before everything gets flushed all the way to the client connection, and the disconnected state makes its way back up the chain. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Frederik - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat under load frontend reverse proxy timeouts
On 27/03/2014 12:39 PM, Frederik Nosi frederik.n...@postecom.it wrote: Hi all, Having to deal with slow applications deployed under tomcat, with a reverse proxy in front, frequently i've noticed that even when the frontend timeouts and closes it's part of the TCP connection, the Tomcat thread processing the request goes on and on till it finishes. Is there a way to make the proccessing thread stop when the frontend connection get's closed? Thanks in advance, Frederik What kind of application is this? What is the reason that the thread takes longer than expected? Is it just the load on the app server or its waiting on the backend database maybe? If load then for sure you can limit the tomcat connector threads and the accept queue to reduce it and add some more app servers. If backend connection then you can sync the client timeout and the db connection timeout.
