Re: Slow HTTP Rquest via Tomcat
Thone Soungpanya wrote: Hello All, Thank you all for responding to my issue. I will see if we can turn off autodeploy and see if this helps. Will let you know. Yes, that seems confusing if it is just stating "DEBUG Checking context[/ps_9.0_8.53.02_1] redeploy resource /usr/local/tomcat-instance5/webapps/ps_9.0_8.53.02_1.war". That statement seems as if it checked context, founded that something changed so it redeployed. But sounds like it did not per Mark's comment. Also, I checked the folders for conf files, etc. and no file timestamps have changed. It always had the same timestamp as the day we deployed the folder unless we directly changed the file data for some reason. Another suggestion maybe : as you see, the logfiles are currently quite confusing, in terms of identifying what /this/ current issue may be. That is in part because, by setting the log-level to "debug", we also get all these other (probably) unrelated messages in-between, for things that in this case are merely "clutter" which obscures the thing which we are really trying to figure out. On the other hand, I think I see that the application itself (which apparently you control) sends some messages of its own, at the "info" level. So my sggestion would be, for now, to reset the loglevel to "info", and maybe add some additional "info" loging messages in the application, to better trace just what is happening in the application. That is also because, so far, I don't think that your main issue (if I recall correctly, that was that "sometimes" it seems that the application takes very long to connect to the external server) is very visible in the logs. There should be a log message - a) when the application actually starts running - b) just before it makes that external connection to "PeopleSoft" - c) at the moment that connection is established - d) at the moment it sends something to the external server - e) and at the moment it returns from the call to that external server then we could see clearly (without tens of intermediate debug-level internal Tomcat messages) where the delays occur. It would also help very much if each of these log messages showed some kind of unique request-linked identifier, so that we can easily figure out if this is a single request, or several parallel ones. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Slow HTTP Rquest via Tomcat
Hello All, Thank you all for responding to my issue. I will see if we can turn off autodeploy and see if this helps. Will let you know. Yes, that seems confusing if it is just stating "DEBUG Checking context[/ps_9.0_8.53.02_1] redeploy resource /usr/local/tomcat-instance5/webapps/ps_9.0_8.53.02_1.war". That statement seems as if it checked context, founded that something changed so it redeployed. But sounds like it did not per Mark's comment. Also, I checked the folders for conf files, etc. and no file timestamps have changed. It always had the same timestamp as the day we deployed the folder unless we directly changed the file data for some reason. Thanks, Thone -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: Wednesday, January 14, 2015 12:52 PM To: Tomcat Users List Subject: Re: Slow HTTP Rquest via Tomcat On 14/01/2015 20:46, André Warnier wrote: > Mark Thomas wrote: >> On 14/01/2015 18:56, Christopher Schultz wrote: >> >>> Possibly. >>> >>> I re-read the log file, and there are no logs that actually say that >>> the context is reloading. It just says "checking context[/path] >>> reload resource" over and over again. I checked, and Tomcat does not >>> emit any log messages that look like that... this must come from somewhere >>> else. >> >> Nope. Look again at the debug logging in >> >> org.apache.cataline.startup.HostConfig.checkResources() >> >> >>> so I'm not sure what I'm looking at. What I can tell is that some >>> component is furiously checking "reload resources" during the time >>> period covered by that log file. >> >> Furiously? Look at the timestamps. This is taking milliseconds at most. >> This is normal behaviour. >> >>> I would check the /Tomcat/ log during the same period to see if >>> anything is happening, there. >> >> GC logging should rule that out as a factor. >> > > So a log message like : > > 2015-01-12 16:11:07,390 DEBUG Checking context[/ps_9.0_8.53.02_1] > redeploy resource > /usr/local/tomcat-instance5/webapps/ps_9.0_8.53.02_1.war > > only means that Tomcat is checking /if/ that resource has been > changed, not that it is actually redeploying it ? Correct. There are both redeploy resources and reload resources. Most web applications have several of both. If autoDeploy is enabled they are checked every ~15s by default. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 1/14/15 3:27 PM, Mark Thomas wrote: > On 14/01/2015 18:56, Christopher Schultz wrote: > >> Possibly. >> >> I re-read the log file, and there are no logs that actually say >> that the context is reloading. It just says "checking >> context[/path] reload resource" over and over again. I checked, >> and Tomcat does not emit any log messages that look like that... >> this must come from somewhere else. > > Nope. Look again at the debug logging in > > org.apache.cataline.startup.HostConfig.checkResources() Ah. Not all on one line. Stupid mistake while grepping. I even checked both .java and .properties files. >> so I'm not sure what I'm looking at. What I can tell is that >> some component is furiously checking "reload resources" during >> the time period covered by that log file. > > Furiously? Look at the timestamps. This is taking milliseconds at > most. This is normal behaviour. I saw the "Checking context" message many times for the "ps_9.0_8.53.11" context and thought it was happening over and over again. It seems there are many version of the "ps" webapp deployed simultaneously and I was assuming they were the same. Thus, furiously. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJUtth3AAoJEBzwKT+lPKRYafgP/2PX5GtW9hXnOjY4h4w5gVxu lk98RkyLLtX5nEeogeLjtDru3hRc2JsSX4v1ThMBPgk8g9qVxdf6yI6Tq6R1qiSN eiBcy21WYTRQtPbg3NxakmK8jEqx15X0hOZtI7zTTIW4ITwa13MpBd+WrFFEIvwy YkmPGRu3jI6s2HEwPjufLnpMAM8q6uF7QU41H9veD0VecDMKGrIMJDtLfc7jm+Op eh95GMpHxZSogLCBrtt5P2YmFwctq5cjE4FyFpMnRES7Klc2zUPLxhZNBGCwLZxN JueadGu1v4LQ5EZQ3NlY/YmA1lRFAfjKqvHE215vTY2nIeRhkr0gu3J/SH/tP9Xy 0dKz4Hb6cKbxmdjeJSsGOOu+PDiHpx6KQ4B2wWxhD2ATTxj4EYY3Cg3cRVcK1yng OLuz2C1xMfbqNVFFQgHPg92tJedMDLnd5MyaLWg1gyp532naw5fTEJa30udrdL4H gPvSiksLgNfCSTCWw743PajtMdcsmfs45eNtruelmLYD0yNjbXEKclzeSSHqW+xY gZ7rCMUWOlXKvi5+Gs8ScIre6wac692Dl7VXOVGKT/jNQBW/orgWsXTov8GOag+l GvzQX1CbkLuyxpiAL2c2ReoypaBdrc5+5v0ycWPUzQLE7Da4TII5sTdT/y85lOH1 AikKHDEgI70fUUQYky7f =/Sky -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
On 14/01/2015 20:46, André Warnier wrote: > Mark Thomas wrote: >> On 14/01/2015 18:56, Christopher Schultz wrote: >> >>> Possibly. >>> >>> I re-read the log file, and there are no logs that actually say that >>> the context is reloading. It just says "checking context[/path] reload >>> resource" over and over again. I checked, and Tomcat does not emit any >>> log messages that look like that... this must come from somewhere else. >> >> Nope. Look again at the debug logging in >> >> org.apache.cataline.startup.HostConfig.checkResources() >> >> >>> so I'm not sure what I'm looking at. What I can tell is that some >>> component is furiously checking "reload resources" during the time >>> period covered by that log file. >> >> Furiously? Look at the timestamps. This is taking milliseconds at most. >> This is normal behaviour. >> >>> I would check the /Tomcat/ log during the same period to see if >>> anything is happening, there. >> >> GC logging should rule that out as a factor. >> > > So a log message like : > > 2015-01-12 16:11:07,390 DEBUG Checking context[/ps_9.0_8.53.02_1] > redeploy resource /usr/local/tomcat-instance5/webapps/ps_9.0_8.53.02_1.war > > only means that Tomcat is checking /if/ that resource has been changed, > not that it is actually redeploying it ? Correct. There are both redeploy resources and reload resources. Most web applications have several of both. If autoDeploy is enabled they are checked every ~15s by default. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
Mark Thomas wrote: On 14/01/2015 18:56, Christopher Schultz wrote: Possibly. I re-read the log file, and there are no logs that actually say that the context is reloading. It just says "checking context[/path] reload resource" over and over again. I checked, and Tomcat does not emit any log messages that look like that... this must come from somewhere else. Nope. Look again at the debug logging in org.apache.cataline.startup.HostConfig.checkResources() so I'm not sure what I'm looking at. What I can tell is that some component is furiously checking "reload resources" during the time period covered by that log file. Furiously? Look at the timestamps. This is taking milliseconds at most. This is normal behaviour. I would check the /Tomcat/ log during the same period to see if anything is happening, there. GC logging should rule that out as a factor. So a log message like : 2015-01-12 16:11:07,390 DEBUG Checking context[/ps_9.0_8.53.02_1] redeploy resource /usr/local/tomcat-instance5/webapps/ps_9.0_8.53.02_1.war only means that Tomcat is checking /if/ that resource has been changed, not that it is actually redeploying it ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
On 14/01/2015 18:56, Christopher Schultz wrote: > Possibly. > > I re-read the log file, and there are no logs that actually say that > the context is reloading. It just says "checking context[/path] reload > resource" over and over again. I checked, and Tomcat does not emit any > log messages that look like that... this must come from somewhere else. Nope. Look again at the debug logging in org.apache.cataline.startup.HostConfig.checkResources() > so I'm not sure what I'm looking at. What I can tell is that some > component is furiously checking "reload resources" during the time > period covered by that log file. Furiously? Look at the timestamps. This is taking milliseconds at most. This is normal behaviour. > I would check the /Tomcat/ log during the same period to see if > anything is happening, there. GC logging should rule that out as a factor. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 1/14/15 1:15 PM, David kerber wrote: > On 1/14/2015 1:02 PM, Christopher Schultz wrote: Thone, > >> ... > > Also, yes, autodeploy is equal to "True". I will check with my colleague to see if we can test with false because this will impact all web folders. Do you see any impact if this was "false"? Why is it defaulted to "true"? > > See http://tomcat.apache.org/tomcat-7.0-doc/config/host.html and > http://tomcat.apache.org/tomcat-7.0-doc/config/host.html#Automatic_Application_Deployment > > > > It defaults to true because it's not much overhead and most people > want to allow automatic re-deployment of web applications. It's > easily disabled. > > The real question is "why is Tomcat deciding to reload"? > > The only time Tomcat reloads for us (we run with autoDeploy set to > default in production) is when we "touch" the META-INF/context.xml > file. > > You should check the timestamps on your context.xml files (or > appname.xml files, if you are placing those into > CATALINA_BASE/conf/[engine]/[host]/[appname].xml) and/or your WAR > files (or exploded WAR directories) in your appbase (defaults to > CATALINA_BASE/webapps) to see if those timestamps are changing for > some reason. > > Timestamps should be retained in ms since the epoch, so DST > changes shouldn't have any effect (e.g. in March, your webapps > won't automatically reload just because of clock adjustments). But, > if the system time changes radically for some reason, it may > trigger a reload or if the file dates change for some reason (I > dunno... some weird NFS thing?) then you may also get a reload. > > I wonder if these occasional slow-downs are an artifact of > reloading the web application in the middle of a request. > >> I take it TC is running as a service (or whatever it's called on >> Linux)? And if so, will it auto restart if TC crashes? I know >> it's quite a stretch, but if something was messed up badly >> enough, could it crash the service during the request, causing it >> to auto restart and reload? Possibly. I re-read the log file, and there are no logs that actually say that the context is reloading. It just says "checking context[/path] reload resource" over and over again. I checked, and Tomcat does not emit any log messages that look like that... this must come from somewhere else. so I'm not sure what I'm looking at. What I can tell is that some component is furiously checking "reload resources" during the time period covered by that log file. I would check the /Tomcat/ log during the same period to see if anything is happening, there. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJUtrv5AAoJEBzwKT+lPKRYAIcQAKxw0yIy9G0jsdEnpN2XK1Fi Sk7YSUtjJhzC4eQR/ndeROeuECPqrYDcIal9c351rN+Uwt73RHvei9rmTe5bQ6Ek 8SE0K09yq0nfEBKGaWUJiPGBMWpAuWw2/c4vzTwkgG7/QKVx4Pfu2IE64+/5WaRE SDPDmvVjxwTY031TmPfJ7/Jtp4rw6yxAVd/r1VqM3+CIPakqozYP6V94ZTHUcE9F NwIHdc62zXxTk55Qq5+96vyIYTrV3uAoRA9q8cKkbAAXFOoCa0XRtgfi6qgik3iu iGOhbxpiUxOj3pEO/gRhJEZGEq7MikpoNDqM2ELhpXVlogwuvpoHLaDkd4e23bir MOXU2KO38QrQuCUb+Foee9EhjbNwPLuOtqOhTQU239eYws5Y+ay8EEfhWLgvvz5z 8T7AEbkrqUulJ4txhKktBIKVikyk/TDfwfFdNyttQNXjpp0NHbbX5BvCTCxA+FBt dTAGnUUG6I7ZJ4VfyQpwu8QdG2gSugttk08Ju7lNFeDLoVSHZnc1NB3GbndFizDK SYZumI9shX7uuh+y991UWl5mzYbq841NAEgV7KF1w4M3K/uPXiYfFEqMlhQlSlVx hqkZHU9qoIch7FoBUXikL9oPKLqUWG16AubjbUeLom5g2LUPFOXlj2V219o0Zl6M qRZGo2KZoC3hVAIFkEIQ =f/u5 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
On 1/14/2015 1:02 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, ... Also, yes, autodeploy is equal to "True". I will check with my colleague to see if we can test with false because this will impact all web folders. Do you see any impact if this was "false"? Why is it defaulted to "true"? See http://tomcat.apache.org/tomcat-7.0-doc/config/host.html and http://tomcat.apache.org/tomcat-7.0-doc/config/host.html#Automatic_Application_Deployment It defaults to true because it's not much overhead and most people want to allow automatic re-deployment of web applications. It's easily disabled. The real question is "why is Tomcat deciding to reload"? The only time Tomcat reloads for us (we run with autoDeploy set to default in production) is when we "touch" the META-INF/context.xml file. You should check the timestamps on your context.xml files (or appname.xml files, if you are placing those into CATALINA_BASE/conf/[engine]/[host]/[appname].xml) and/or your WAR files (or exploded WAR directories) in your appbase (defaults to CATALINA_BASE/webapps) to see if those timestamps are changing for some reason. Timestamps should be retained in ms since the epoch, so DST changes shouldn't have any effect (e.g. in March, your webapps won't automatically reload just because of clock adjustments). But, if the system time changes radically for some reason, it may trigger a reload or if the file dates change for some reason (I dunno... some weird NFS thing?) then you may also get a reload. I wonder if these occasional slow-downs are an artifact of reloading the web application in the middle of a request. I take it TC is running as a service (or whatever it's called on Linux)? And if so, will it auto restart if TC crashes? I know it's quite a stretch, but if something was messed up badly enough, could it crash the service during the request, causing it to auto restart and reload? - -chris -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Tuesday, January 13, 2015 2:04 PM To: Tomcat Users List Subject: Re: Slow HTTP Rquest via Tomcat Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, On 1/13/15 1:05 PM, Thone Soungpanya wrote: Hello Andre, I do not think it is an DNS lookup issue but I'll check on it. Yes we actually have logs but it does not tell us much. We added additional logging in our servlet code to tell us when and what the code is doing. I have attached a log for 1 connection which took about 1 minute and 23 seconds to complete. I have eliminated all sensitive information with "XX". During this time, there would be other connections too simultaneously. Some information on the request: 1) HTTP request initiated at 16:11:06 to Tomcat. This is a very confusing log file... from latest-to-oldest. +10 Before #2 happens, Tomcat reloads. Many (all?) web applications were re-loaded. Was that expected? 2) Connection to Third Party System happened at 16:12:22. You can see it on line 38 with following text "2015-01-12 16:12:22,426 INFO 06E4074221FF93C2827079EBC2102847; begin request" Okay. 3) Response back to Tomcat from Third Party System at 16:12:29 This does not appear in the log. So it seems that between point#1 and point #2, there was about a 1 minute and 16 seconds delay before connection to the third party. Point #2 to Point #3 only took 7 seconds. Can you or anyone see what may be the issue from the logs? Maybe if you provide clearer logs, having some real information allowing to distinguish one request processing sequence from another. Tomcat is reloading in the middle of the request? Right, that's what it looks like. And not only this one time. From the (very difficult) look of it, it seems that this system is spending its time reloading its applications almost continuously. Using some nifty features of Notepad++, I have extracted and re-ordered what I believe are log lines relating to one webapp (and I don't really know for sure if this is one request, or several). See attached if it makes it, it's probably clearer. Otherwise, I have also pasted these log lines below. I have selected lines related to a webapp named "ps_1.5_8.53.10", and re-ordered them in a logical older-to-younger top-down sequence. Examining thse lines, we see : - at 2015-01-12 16:11:07,396 : what appears to be the beginning of a redeployment of that webapp - at 2015-01-12 16:11:08,108 (one second later) : what appears to be the beginning of a request to that webapp - at 2015-01-12 16:11:15,918 (7 seconds later), what seems to be the beginning of yet another redeployment of that webapp - at 2015-01-12 16:11:21,037, another request starting In-between, there are a lot of other lines apparently showing similar (other) webapps redeployment
Re: Slow HTTP Rquest via Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, On 1/13/15 7:59 PM, Thone Soungpanya wrote: > Hello Andre, > > Yes sorry the logs files are not clear. We have so much traffic > going through the Tomcat and many clients connecting which it seems > to be logging for different sessions at different time. We really > have no control to sort this out as the connections are > simultaneous. > > Our reordering of the logs and the entries listed may be correct. I > cannot tell myself if it is linked to the same session. We tried to > use Session IDs to add against the entries we make for servlet code > but that somehow does not follow in order. > > As for your comment on web.xml and retriggering the redeployment of > the web app folders, yes, I am wondering why this is happening if > that is the case. There is a web.xml in each WEB-INF folder however > nothing never change. Also, this is all it has in it below where it > is just the servlet name and mapping. Can this really retrigger it > if it never changes? > > http://java.sun.com/xml/ns/j2ee > http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"; version="2.4"> > Peoplesoft Connector > PeopleSoft Connector Applications. > PSConnectorServlet > PSConnectorServlet > PSConnectorServlet > /servlet/PSConnectorServlet > > > Also, yes, autodeploy is equal to "True". I will check with my > colleague to see if we can test with false because this will impact > all web folders. Do you see any impact if this was "false"? Why is > it defaulted to "true"? See http://tomcat.apache.org/tomcat-7.0-doc/config/host.html and http://tomcat.apache.org/tomcat-7.0-doc/config/host.html#Automatic_Application_Deployment It defaults to true because it's not much overhead and most people want to allow automatic re-deployment of web applications. It's easily disabled. The real question is "why is Tomcat deciding to reload"? The only time Tomcat reloads for us (we run with autoDeploy set to default in production) is when we "touch" the META-INF/context.xml file. You should check the timestamps on your context.xml files (or appname.xml files, if you are placing those into CATALINA_BASE/conf/[engine]/[host]/[appname].xml) and/or your WAR files (or exploded WAR directories) in your appbase (defaults to CATALINA_BASE/webapps) to see if those timestamps are changing for some reason. Timestamps should be retained in ms since the epoch, so DST changes shouldn't have any effect (e.g. in March, your webapps won't automatically reload just because of clock adjustments). But, if the system time changes radically for some reason, it may trigger a reload or if the file dates change for some reason (I dunno... some weird NFS thing?) then you may also get a reload. I wonder if these occasional slow-downs are an artifact of reloading the web application in the middle of a request. - -chris > -Original Message- From: André Warnier > [mailto:a...@ice-sa.com] Sent: Tuesday, January 13, 2015 2:04 PM To: > Tomcat Users List Subject: Re: Slow HTTP Rquest via Tomcat > > Christopher Schultz wrote: >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 >> >> Thone, >> >> On 1/13/15 1:05 PM, Thone Soungpanya wrote: >>> Hello Andre, >>> >>> I do not think it is an DNS lookup issue but I'll check on it. >>> >>> Yes we actually have logs but it does not tell us much. We >>> added additional logging in our servlet code to tell us when >>> and what the code is doing. I have attached a log for 1 >>> connection which took about 1 minute and 23 seconds to >>> complete. I have eliminated all sensitive information with >>> "XX". During this time, there would be >>> other connections too simultaneously. >>> >>> Some information on the request: 1) HTTP request initiated at >>> 16:11:06 to Tomcat. >> >> This is a very confusing log file... from latest-to-oldest. > > +10 > >> >> Before #2 happens, Tomcat reloads. Many (all?) web applications >> were re-loaded. Was that expected? >> >>> 2) Connection to Third Party System happened at 16:12:22. You >>> can see it on line 38 with following text "2015-01-12 >>> 16:12:22,426 INFO 06E4074221FF93C2827079EBC2102847; begin >>> request" >> >> Okay. >> >>> 3) Response back to Tomcat from Third Party System at 16:12:29 >> >> This does not appear in the log. >> >>> So it seems that between point#1 and point #2, there was about >>> a 1 minute and 16 seconds delay before connection to the third >>>
RE: Slow HTTP Rquest via Tomcat
Hello Andre, Yes sorry the logs files are not clear. We have so much traffic going through the Tomcat and many clients connecting which it seems to be logging for different sessions at different time. We really have no control to sort this out as the connections are simultaneous. Our reordering of the logs and the entries listed may be correct. I cannot tell myself if it is linked to the same session. We tried to use Session IDs to add against the entries we make for servlet code but that somehow does not follow in order. As for your comment on web.xml and retriggering the redeployment of the web app folders, yes, I am wondering why this is happening if that is the case. There is a web.xml in each WEB-INF folder however nothing never change. Also, this is all it has in it below where it is just the servlet name and mapping. Can this really retrigger it if it never changes? http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"; version="2.4"> Peoplesoft Connector PeopleSoft Connector Applications. PSConnectorServlet PSConnectorServlet PSConnectorServlet /servlet/PSConnectorServlet Also, yes, autodeploy is equal to "True". I will check with my colleague to see if we can test with false because this will impact all web folders. Do you see any impact if this was "false"? Why is it defaulted to "true"? Thanks, Thone -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Tuesday, January 13, 2015 2:04 PM To: Tomcat Users List Subject: Re: Slow HTTP Rquest via Tomcat Christopher Schultz wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Thone, > > On 1/13/15 1:05 PM, Thone Soungpanya wrote: >> Hello Andre, >> >> I do not think it is an DNS lookup issue but I'll check on it. >> >> Yes we actually have logs but it does not tell us much. We added >> additional logging in our servlet code to tell us when and what the >> code is doing. I have attached a log for 1 connection which took >> about 1 minute and 23 seconds to complete. I have eliminated all >> sensitive information with "XX". During this >> time, there would be other connections too simultaneously. >> >> Some information on the request: 1) HTTP request initiated at >> 16:11:06 to Tomcat. > > This is a very confusing log file... from latest-to-oldest. +10 > > Before #2 happens, Tomcat reloads. Many (all?) web applications were > re-loaded. Was that expected? > >> 2) Connection to Third Party System happened at 16:12:22. You can see >> it on line 38 with following text "2015-01-12 16:12:22,426 INFO >> 06E4074221FF93C2827079EBC2102847; begin request" > > Okay. > >> 3) Response back to Tomcat from Third Party System at 16:12:29 > > This does not appear in the log. > >> So it seems that between point#1 and point #2, there was about a 1 >> minute and 16 seconds delay before connection to the third party. >> Point #2 to Point #3 only took 7 seconds. >> >> Can you or anyone see what may be the issue from the logs? > Maybe if you provide clearer logs, having some real information allowing to distinguish one request processing sequence from another. > Tomcat is reloading in the middle of the request? Right, that's what it looks like. And not only this one time. From the (very difficult) look of it, it seems that this system is spending its time reloading its applications almost continuously. Using some nifty features of Notepad++, I have extracted and re-ordered what I believe are log lines relating to one webapp (and I don't really know for sure if this is one request, or several). See attached if it makes it, it's probably clearer. Otherwise, I have also pasted these log lines below. I have selected lines related to a webapp named "ps_1.5_8.53.10", and re-ordered them in a logical older-to-younger top-down sequence. Examining thse lines, we see : - at 2015-01-12 16:11:07,396 : what appears to be the beginning of a redeployment of that webapp - at 2015-01-12 16:11:08,108 (one second later) : what appears to be the beginning of a request to that webapp - at 2015-01-12 16:11:15,918 (7 seconds later), what seems to be the beginning of yet another redeployment of that webapp - at 2015-01-12 16:11:21,037, another request starting In-between, there are a lot of other lines apparently showing similar (other) webapps redeployments, and maybe other lines related to that same webapp, but all quite difficult to identify because there is no common marker. That logfile is very confusing, and as a result take this with a grain of salt, but I get the funny feeling that this webapp, when it responds to a request, may be modif
Re: Slow HTTP Rquest via Tomcat
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, On 1/13/15 1:05 PM, Thone Soungpanya wrote: Hello Andre, I do not think it is an DNS lookup issue but I'll check on it. Yes we actually have logs but it does not tell us much. We added additional logging in our servlet code to tell us when and what the code is doing. I have attached a log for 1 connection which took about 1 minute and 23 seconds to complete. I have eliminated all sensitive information with "XX". During this time, there would be other connections too simultaneously. Some information on the request: 1) HTTP request initiated at 16:11:06 to Tomcat. This is a very confusing log file... from latest-to-oldest. +10 Before #2 happens, Tomcat reloads. Many (all?) web applications were re-loaded. Was that expected? 2) Connection to Third Party System happened at 16:12:22. You can see it on line 38 with following text "2015-01-12 16:12:22,426 INFO 06E4074221FF93C2827079EBC2102847; begin request" Okay. 3) Response back to Tomcat from Third Party System at 16:12:29 This does not appear in the log. So it seems that between point#1 and point #2, there was about a 1 minute and 16 seconds delay before connection to the third party. Point #2 to Point #3 only took 7 seconds. Can you or anyone see what may be the issue from the logs? Maybe if you provide clearer logs, having some real information allowing to distinguish one request processing sequence from another. Tomcat is reloading in the middle of the request? Right, that's what it looks like. And not only this one time. From the (very difficult) look of it, it seems that this system is spending its time reloading its applications almost continuously. Using some nifty features of Notepad++, I have extracted and re-ordered what I believe are log lines relating to one webapp (and I don't really know for sure if this is one request, or several). See attached if it makes it, it's probably clearer. Otherwise, I have also pasted these log lines below. I have selected lines related to a webapp named "ps_1.5_8.53.10", and re-ordered them in a logical older-to-younger top-down sequence. Examining thse lines, we see : - at 2015-01-12 16:11:07,396 : what appears to be the beginning of a redeployment of that webapp - at 2015-01-12 16:11:08,108 (one second later) : what appears to be the beginning of a request to that webapp - at 2015-01-12 16:11:15,918 (7 seconds later), what seems to be the beginning of yet another redeployment of that webapp - at 2015-01-12 16:11:21,037, another request starting In-between, there are a lot of other lines apparently showing similar (other) webapps redeployments, and maybe other lines related to that same webapp, but all quite difficult to identify because there is no common marker. That logfile is very confusing, and as a result take this with a grain of salt, but I get the funny feeling that this webapp, when it responds to a request, may be modifying something (it's own WEB-INF/web.xml ?) in such a way that it triggers a re-deployment of that same very application by Tomcat, at each request. (And similarly for the other ones) For a start, does the tag of that Tomcat instance have an autodeploy="true" attribute ? (see in conf/server.xml). And if so, what about setting that attribute to "false", restarting this Tomcat, and looking at the symptoms again ? (and read : http://tomcat.apache.org/tomcat-8.0-doc/config/host.html#Standard_Implementation section : Automatic Application Deployment ) Appendix (also attached but the attachment may be stripped by the list) : Re-ordered selected log line : Line 1099: 2015-01-12 16:11:07,396 DEBUG Checking context[/ps_1.5_8.53.10] redeploy resource /usr/local/tomcat-instance5/webapps/ps_1.5_8.53.10.war Line 1098: 2015-01-12 16:11:07,396 DEBUG Checking context[/ps_1.5_8.53.10] redeploy resource /usr/local/tomcat-instance5/webapps/ps_1.5_8.53.10 Line 1097: 2015-01-12 16:11:07,396 DEBUG Checking context[/ps_1.5_8.53.10] redeploy resource /usr/local/tomcat-instance5/conf/Catalina/localhost/ps_1.5_8.53.10.xml Line 1096: 2015-01-12 16:11:07,396 DEBUG Checking context[/ps_1.5_8.53.10] redeploy resource /usr/local/tomcat-instance5/webapps/ps_1.5_8.53.10/META-INF/context.xml Line 1095: 2015-01-12 16:11:07,396 DEBUG Checking context[/ps_1.5_8.53.10] redeploy resource /usr/local/tomcat-instance5/conf/context.xml Line 1094: 2015-01-12 16:11:07,396 DEBUG Checking context[/ps_1.5_8.53.10] reload resource /usr/local/tomcat-instance5/webapps/ps_1.5_8.53.10/WEB-INF/web.xml Line 972: 2015-01-12 16:11:08,108 DEBUG The variable [uriBC] has value [/ps_1.5_8.53.10/servlet/PSConnectorServlet] Line 969: 2015-01-12 16:11:08,108 DEBUG Security checking request GET /ps_1.5_8.53.10/servlet/PSConnectorServlet Line 964: 2015-01-12 16:11:08,109 DEBUG --> Returning 'file:/usr/local/tomcat-instance5/webapps/ps_1.5_8.53.10/WEB-INF/class
RE: Slow HTTP Rquest via Tomcat
Hello Chris, Yes, sorry when we pull from a Splunk program we have for the logs, it splits it out in the reverse order. Regarding #3, yes sorry it was missing from the logs but it basically return some information we request around that time. Tomcat is reloading in the middle of the request? -- Yes I was looking into why this is doing so but I recall seeing on the google searches that it is done by default to reload all web folders? Is this not suppose to behavior this way? If not, where would the setting be to turn this off as I couldn't find it anywhere. Thanks, Thone -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, January 13, 2015 11:01 AM To: Tomcat Users List Subject: Re: Slow HTTP Rquest via Tomcat -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, On 1/13/15 1:05 PM, Thone Soungpanya wrote: > Hello Andre, > > I do not think it is an DNS lookup issue but I'll check on it. > > Yes we actually have logs but it does not tell us much. We added > additional logging in our servlet code to tell us when and what the > code is doing. I have attached a log for 1 connection which took about > 1 minute and 23 seconds to complete. I have eliminated all sensitive > information with "XX". During this time, there > would be other connections too simultaneously. > > Some information on the request: 1) HTTP request initiated at > 16:11:06 to Tomcat. This is a very confusing log file... from latest-to-oldest. Before #2 happens, Tomcat reloads. Many (all?) web applications were re-loaded. Was that expected? > 2) Connection to Third Party System happened at 16:12:22. You can see > it on line 38 with following text "2015-01-12 16:12:22,426 INFO > 06E4074221FF93C2827079EBC2102847; begin request" Okay. > 3) Response back to Tomcat from Third Party System at 16:12:29 This does not appear in the log. > So it seems that between point#1 and point #2, there was about a 1 > minute and 16 seconds delay before connection to the third party. > Point #2 to Point #3 only took 7 seconds. > > Can you or anyone see what may be the issue from the logs? Tomcat is reloading in the middle of the request? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJUtWtPAAoJEBzwKT+lPKRY/m8P/i0VaL3x05kGAArWb+RL+BOL jhfezsN5NKw3iosMFnlDkeyoD59AnWZvUqbnybqzCf9aXacT3iUeHEZHq4/mu3ew juawm0phZl19PKYCfA/3PN0XR1jXVWdyN06QJuzA1KlpFDp7+/9WTWvMcLtG3DqC vW3NstywZ7sPTgingHgVegEjcnKeXyDmFuhaUyyBNxo7Tgy/HaNwfB8Y7GSLBvGY /NiJi0klkpbeyz+epbxzBe8JlYiFwosjlJIE0MpXvZgrKtiyD+cj9yaJxNGos5GS nkiD3FGYKQ5cZ5c4dAXMs7es0bPjqRYWBRTiiaa8W2yX57t44Y7ZB6geuteep0/7 URS0NoONdCg1TbR/XE1kqG06g+IrBSzHBUZRzW3A8+NICcLQc/3i+X1zMYfnrD46 MzcBViMpsyJrBbsfpPol49zLx284/+146gYs6djhCdnWcbVNXf12QI/DA/YedThc zJeCYy0MWQlhQ2T1FECgtgtPzSY0j+Wt06Dz+JoTAOXb5qX9kD/hvj+d2VRfWkkn P2qBANSiOBE+JNPXQsA1JjHlWHO+HS4CIqG5VEflQ7aVzbC0zv57aRYqFFpPu1tH Oc+S/m1AGpDdaRGBhjgdJCbtT8Pyi+WzCAA5YqeHjmEBDwo7pnnXoh9HO5t0UOb5 8hWJpjaXWxg+NxjIlv+P =wn8y -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, On 1/13/15 1:05 PM, Thone Soungpanya wrote: > Hello Andre, > > I do not think it is an DNS lookup issue but I'll check on it. > > Yes we actually have logs but it does not tell us much. We added > additional logging in our servlet code to tell us when and what the > code is doing. I have attached a log for 1 connection which took > about 1 minute and 23 seconds to complete. I have eliminated all > sensitive information with "XX". During this > time, there would be other connections too simultaneously. > > Some information on the request: 1) HTTP request initiated at > 16:11:06 to Tomcat. This is a very confusing log file... from latest-to-oldest. Before #2 happens, Tomcat reloads. Many (all?) web applications were re-loaded. Was that expected? > 2) Connection to Third Party System happened at 16:12:22. You can > see it on line 38 with following text "2015-01-12 16:12:22,426 > INFO 06E4074221FF93C2827079EBC2102847; begin request" Okay. > 3) Response back to Tomcat from Third Party System at 16:12:29 This does not appear in the log. > So it seems that between point#1 and point #2, there was about a 1 > minute and 16 seconds delay before connection to the third party. > Point #2 to Point #3 only took 7 seconds. > > Can you or anyone see what may be the issue from the logs? Tomcat is reloading in the middle of the request? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJUtWtPAAoJEBzwKT+lPKRY/m8P/i0VaL3x05kGAArWb+RL+BOL jhfezsN5NKw3iosMFnlDkeyoD59AnWZvUqbnybqzCf9aXacT3iUeHEZHq4/mu3ew juawm0phZl19PKYCfA/3PN0XR1jXVWdyN06QJuzA1KlpFDp7+/9WTWvMcLtG3DqC vW3NstywZ7sPTgingHgVegEjcnKeXyDmFuhaUyyBNxo7Tgy/HaNwfB8Y7GSLBvGY /NiJi0klkpbeyz+epbxzBe8JlYiFwosjlJIE0MpXvZgrKtiyD+cj9yaJxNGos5GS nkiD3FGYKQ5cZ5c4dAXMs7es0bPjqRYWBRTiiaa8W2yX57t44Y7ZB6geuteep0/7 URS0NoONdCg1TbR/XE1kqG06g+IrBSzHBUZRzW3A8+NICcLQc/3i+X1zMYfnrD46 MzcBViMpsyJrBbsfpPol49zLx284/+146gYs6djhCdnWcbVNXf12QI/DA/YedThc zJeCYy0MWQlhQ2T1FECgtgtPzSY0j+Wt06Dz+JoTAOXb5qX9kD/hvj+d2VRfWkkn P2qBANSiOBE+JNPXQsA1JjHlWHO+HS4CIqG5VEflQ7aVzbC0zv57aRYqFFpPu1tH Oc+S/m1AGpDdaRGBhjgdJCbtT8Pyi+WzCAA5YqeHjmEBDwo7pnnXoh9HO5t0UOb5 8hWJpjaXWxg+NxjIlv+P =wn8y -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Slow HTTP Rquest via Tomcat
Hello Chris, The average number of simultaneous requests I would say would be about 1500 connections when this slows down. Is there a max number of connection or memory limitation? We have about 4 GB of memory on the system which 1 GB is set as the XMX value for this JVM instance. Jolt port is Oracle Technology which will handle all web requests and communicate between PeopleSoft application server and web server. Ok, I will try the thread dump of the application and see if it will give some information. Thanks, Thone -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Monday, January 12, 2015 7:41 PM To: Tomcat Users List Subject: Re: Slow HTTP Rquest via Tomcat -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, On 1/12/15 6:03 PM, Thone Soungpanya wrote: > When connecting from Tomcat to third party, we are going through a > servlet code which is customized by us. We package the code into a jar > file which sits on the \WEB-INF\lib folder. When connection, we create > a session and connect through the application server IP address, jolt > port, username and password. The third party system we connect to is a > PeopleSoft ERP system. > > When the problem occurs, I would estimate it to be about 200 or so > active sessions through just that one folder. Now, another thing I > forgot to mention is, we have multiple web application folders with > the same files but just different PeopleTools version to support for > different clients. These are not slowing up and they have about maybe > 10 - 20 sessions. Just throwing this information out there in case. > > Also to note again, I do not think it is the connection to the third > party. While reviewing the logs, I notice when connecting to third > party, it takes about 10 seconds round-trip. We write logs when it > begin connection to end connection. Something is bottlenecking in the > tomcat server before trying to connect. The number of uses sessions is largely irrelevant. What is relevant is the number of simultaneous requests that are being made to your own web application, and how many are trying to create these outgoing connections. (Just curious: what is a "jolt port"?) If you have a 20-50 second pause, there's enough time to trigger a thread dump of the application (see the wiki HOWTO section for how to do that). A thread dump will tell you where the code is hanging. It's not likely to be in Tomcat itself. - -chris > -Original Message- From: cjder...@gmail.com > [mailto:cjder...@gmail.com] On Behalf Of chris derham Sent: > Monday, January 12, 2015 2:43 PM To: Tomcat Users List Subject: Re: > Slow HTTP Rquest via Tomcat > > On 13 January 2015 at 11:38, Thone Soungpanya > wrote: >> Hello, >> >> Can anyone give some advice on what the issue may be? We have a web >> application folder in Tomcat that have been slowing the http >> request's connection to another third party system when there are >> high traffic. It seems to be halting in the Tomcat server for about >> 20 - 50 seconds before attempting to connect to the third party >> system. Once it tries to connect to the third party system, the >> response time is only about 6 - 10 seconds which is normal. >> It does not seem to be an issue at the third party's end as the >> response time is quick once we get connected. >> >> Here are the specs that we have and additional information... >> >> 1) Apache Tomcat 7.0.56 2) JVM version 1.7 3) Running on Linux OS >> Version 2.6.32 >> >> Connection Flow: 1) User submits a HTTP request from web application >> to Tomcat 2) Tomcat calls Java's servlet code along with HTTP request >> parameters 3) Java servlet code connects to third party system via >> application server and jolt port 4) Third party system returns >> requested information >> >> We thought it would be a session issue which we lowered the session >> timeout (just for that folder) in WEB-INF folder to 5 minutes from >> defaulted 30 minutes but it is still doing the same thing. >> >> Any advice on what it may be? Let me know if you need additional >> information. > > How does your code connect to the 3rd party? There are libraries that > do this, please elaborate which you use. I recall using one, and it > had a limit on the number of concurrent connections to a 3rd party. > Perhaps you hit this limit? How many connections are you talking about > when the problem occurs - 10s, 100s, 1000s or more? > > Chris > >> Thank you in advance! >> >> Thone > > - > > > To
Re: Slow HTTP Rquest via Tomcat
Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Thone,
On 1/12/15 6:03 PM, Thone Soungpanya wrote:
When connecting from Tomcat to third party, we are going through a
servlet code which is customized by us. We package the code into a
jar file which sits on the \WEB-INF\lib folder. When connection,
we create a session and connect through the application server IP
address, jolt port, username and password. The third party system
we connect to is a PeopleSoft ERP system.
When the problem occurs, I would estimate it to be about 200 or so
active sessions through just that one folder. Now, another thing I
forgot to mention is, we have multiple web application folders
with the same files but just different PeopleTools version to
support for different clients. These are not slowing up and they
have about maybe 10 - 20 sessions. Just throwing this information
out there in case.
Also to note again, I do not think it is the connection to the
third party. While reviewing the logs, I notice when connecting to
third party, it takes about 10 seconds round-trip. We write logs
when it begin connection to end connection. Something is
bottlenecking in the tomcat server before trying to connect.
The number of uses sessions is largely irrelevant. What is relevant is
the number of simultaneous requests that are being made to your own
web application, and how many are trying to create these outgoing
connections.
(Just curious: what is a "jolt port"?)
If you have a 20-50 second pause, there's enough time to trigger a
thread dump of the application (see the wiki HOWTO section for how to
do that). A thread dump will tell you where the code is hanging. It's
not likely to be in Tomcat itself.
Maybe a DNS lookup issue ?
If the code of the webapp is under control of the OP ("servlet code which is customized by
us"), then it should be easy to add some logging in there, to figure out where the time
really goes ?
That may be a bit easier to do at first, than taking thread dumps.
- -chris
-Original Message- From: cjder...@gmail.com
[mailto:cjder...@gmail.com] On Behalf Of chris derham Sent:
Monday, January 12, 2015 2:43 PM To: Tomcat Users List Subject: Re:
Slow HTTP Rquest via Tomcat
On 13 January 2015 at 11:38, Thone Soungpanya
wrote:
Hello,
Can anyone give some advice on what the issue may be? We have a
web application folder in Tomcat that have been slowing the http
request's connection to another third party system when there
are high traffic. It seems to be halting in the Tomcat server for
about 20 - 50 seconds before attempting to connect to the third
party system. Once it tries to connect to the third party system,
the response time is only about 6 - 10 seconds which is normal.
It does not seem to be an issue at the third party's end as the
response time is quick once we get connected.
Here are the specs that we have and additional information...
1) Apache Tomcat 7.0.56 2) JVM version 1.7 3) Running on Linux
OS Version 2.6.32
Connection Flow: 1) User submits a HTTP request from web
application to Tomcat 2) Tomcat calls Java's servlet code along
with HTTP request parameters 3) Java servlet code connects to
third party system via application server and jolt port 4) Third
party system returns requested information
We thought it would be a session issue which we lowered the
session timeout (just for that folder) in WEB-INF folder to 5
minutes from defaulted 30 minutes but it is still doing the same
thing.
Any advice on what it may be? Let me know if you need additional
information.
How does your code connect to the 3rd party? There are libraries
that do this, please elaborate which you use. I recall using one,
and it had a limit on the number of concurrent connections to a 3rd
party. Perhaps you hit this limit? How many connections are you
talking about when the problem occurs - 10s, 100s, 1000s or more?
Chris
Thank you in advance!
Thone
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJUtJPoAAoJEBzwKT+lPKRYeYAQAIElQByMpZ88BGNpwNPNqJ08
n0uw/bsBAhKV39Yz54qNRFiQ/7CBCDtP9IrghdjrWNDgnocFb2DGPCW37dT3IS3P
76nTzcZIby8XU/CtUP2iPBqxYLbhm8FEsRB6yH4YB/0QRgz5E1SRBBFw7f2LVQFz
8xnnGObrIJn2Ah0sYlpwVH3FejokjLe67wra3SBV1ONQsZzwpm3JCGdpHzXEzUqt
MhSElAoXhk+LupA7ofvPC4GLiS6uvS1o9cOP1wbEK5P5DeFe3E78H/QPmSWf/0Vd
iEIZbrlKRQyKjNPn+9NvKZMWcwcMYuntjdbhQNDOyQ0A/GW9DwV4qNjsUuQbyLi6
sF0UBmygw1dmHnDTJ4F04JNkYsGVG+POlWltKvG/Q6H+aywu86OxJKqWp7L4NL5G
T4g9j4pGYLdX1WDjZLWlG71
Re: Slow HTTP Rquest via Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thone, On 1/12/15 6:03 PM, Thone Soungpanya wrote: > When connecting from Tomcat to third party, we are going through a > servlet code which is customized by us. We package the code into a > jar file which sits on the \WEB-INF\lib folder. When connection, > we create a session and connect through the application server IP > address, jolt port, username and password. The third party system > we connect to is a PeopleSoft ERP system. > > When the problem occurs, I would estimate it to be about 200 or so > active sessions through just that one folder. Now, another thing I > forgot to mention is, we have multiple web application folders > with the same files but just different PeopleTools version to > support for different clients. These are not slowing up and they > have about maybe 10 - 20 sessions. Just throwing this information > out there in case. > > Also to note again, I do not think it is the connection to the > third party. While reviewing the logs, I notice when connecting to > third party, it takes about 10 seconds round-trip. We write logs > when it begin connection to end connection. Something is > bottlenecking in the tomcat server before trying to connect. The number of uses sessions is largely irrelevant. What is relevant is the number of simultaneous requests that are being made to your own web application, and how many are trying to create these outgoing connections. (Just curious: what is a "jolt port"?) If you have a 20-50 second pause, there's enough time to trigger a thread dump of the application (see the wiki HOWTO section for how to do that). A thread dump will tell you where the code is hanging. It's not likely to be in Tomcat itself. - -chris > -Original Message- From: cjder...@gmail.com > [mailto:cjder...@gmail.com] On Behalf Of chris derham Sent: > Monday, January 12, 2015 2:43 PM To: Tomcat Users List Subject: Re: > Slow HTTP Rquest via Tomcat > > On 13 January 2015 at 11:38, Thone Soungpanya > wrote: >> Hello, >> >> Can anyone give some advice on what the issue may be? We have a >> web application folder in Tomcat that have been slowing the http >> request's connection to another third party system when there >> are high traffic. It seems to be halting in the Tomcat server for >> about 20 - 50 seconds before attempting to connect to the third >> party system. Once it tries to connect to the third party system, >> the response time is only about 6 - 10 seconds which is normal. >> It does not seem to be an issue at the third party's end as the >> response time is quick once we get connected. >> >> Here are the specs that we have and additional information... >> >> 1) Apache Tomcat 7.0.56 2) JVM version 1.7 3) Running on Linux >> OS Version 2.6.32 >> >> Connection Flow: 1) User submits a HTTP request from web >> application to Tomcat 2) Tomcat calls Java's servlet code along >> with HTTP request parameters 3) Java servlet code connects to >> third party system via application server and jolt port 4) Third >> party system returns requested information >> >> We thought it would be a session issue which we lowered the >> session timeout (just for that folder) in WEB-INF folder to 5 >> minutes from defaulted 30 minutes but it is still doing the same >> thing. >> >> Any advice on what it may be? Let me know if you need additional >> information. > > How does your code connect to the 3rd party? There are libraries > that do this, please elaborate which you use. I recall using one, > and it had a limit on the number of concurrent connections to a 3rd > party. Perhaps you hit this limit? How many connections are you > talking about when the problem occurs - 10s, 100s, 1000s or more? > > Chris > >> Thank you in advance! >> >> Thone > > - > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJUtJPoAAoJEBzwKT+lPKRYeYAQAIElQByMpZ88BGNpwNPNqJ08 n0uw/bsBAhKV39Yz54qNRFiQ/7CBCDtP9IrghdjrWNDgnocFb2DGPCW37dT3IS3P 76nTzcZIby8XU/CtUP2iPBqxYLbhm8FEsRB6yH4YB/0QRgz5E1SRBBFw7f2LVQFz 8xnnGObrIJn2Ah0sYlpwVH3FejokjLe67wra3SBV1ONQsZzwpm3JCGdpHzXEzUqt MhSElAoXhk+LupA7ofvPC4GLiS6uvS1o9c
RE: Slow HTTP Rquest via Tomcat
Hello Chris, Thank you for your quick response. When connecting from Tomcat to third party, we are going through a servlet code which is customized by us. We package the code into a jar file which sits on the \WEB-INF\lib folder. When connection, we create a session and connect through the application server IP address, jolt port, username and password. The third party system we connect to is a PeopleSoft ERP system. When the problem occurs, I would estimate it to be about 200 or so active sessions through just that one folder. Now, another thing I forgot to mention is, we have multiple web application folders with the same files but just different PeopleTools version to support for different clients. These are not slowing up and they have about maybe 10 - 20 sessions. Just throwing this information out there in case. Also to note again, I do not think it is the connection to the third party. While reviewing the logs, I notice when connecting to third party, it takes about 10 seconds round-trip. We write logs when it begin connection to end connection. Something is bottlenecking in the tomcat server before trying to connect. Hope this helps. Thanks, Thone -Original Message- From: cjder...@gmail.com [mailto:cjder...@gmail.com] On Behalf Of chris derham Sent: Monday, January 12, 2015 2:43 PM To: Tomcat Users List Subject: Re: Slow HTTP Rquest via Tomcat On 13 January 2015 at 11:38, Thone Soungpanya wrote: > Hello, > > Can anyone give some advice on what the issue may be? We have a web > application folder in Tomcat that have been slowing the http request's > connection to another third party system when there are high traffic. It > seems to be halting in the Tomcat server for about 20 - 50 seconds before > attempting to connect to the third party system. Once it tries to connect to > the third party system, the response time is only about 6 - 10 seconds which > is normal. It does not seem to be an issue at the third party's end as the > response time is quick once we get connected. > > Here are the specs that we have and additional information... > > 1) Apache Tomcat 7.0.56 > 2) JVM version 1.7 > 3) Running on Linux OS Version 2.6.32 > > Connection Flow: > 1) User submits a HTTP request from web application to Tomcat > 2) Tomcat calls Java's servlet code along with HTTP request parameters > 3) Java servlet code connects to third party system via application > server and jolt port > 4) Third party system returns requested information > > We thought it would be a session issue which we lowered the session timeout > (just for that folder) in WEB-INF folder to 5 minutes from defaulted 30 > minutes but it is still doing the same thing. > > Any advice on what it may be? Let me know if you need additional information. How does your code connect to the 3rd party? There are libraries that do this, please elaborate which you use. I recall using one, and it had a limit on the number of concurrent connections to a 3rd party. Perhaps you hit this limit? How many connections are you talking about when the problem occurs - 10s, 100s, 1000s or more? Chris > Thank you in advance! > > Thone - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow HTTP Rquest via Tomcat
On 13 January 2015 at 11:38, Thone Soungpanya wrote: > Hello, > > Can anyone give some advice on what the issue may be? We have a web > application folder in Tomcat that have been slowing the http request's > connection to another third party system when there are high traffic. It > seems to be halting in the Tomcat server for about 20 - 50 seconds before > attempting to connect to the third party system. Once it tries to connect to > the third party system, the response time is only about 6 - 10 seconds which > is normal. It does not seem to be an issue at the third party's end as the > response time is quick once we get connected. > > Here are the specs that we have and additional information... > > 1) Apache Tomcat 7.0.56 > 2) JVM version 1.7 > 3) Running on Linux OS Version 2.6.32 > > Connection Flow: > 1) User submits a HTTP request from web application to Tomcat > 2) Tomcat calls Java's servlet code along with HTTP request parameters > 3) Java servlet code connects to third party system via application server > and jolt port > 4) Third party system returns requested information > > We thought it would be a session issue which we lowered the session timeout > (just for that folder) in WEB-INF folder to 5 minutes from defaulted 30 > minutes but it is still doing the same thing. > > Any advice on what it may be? Let me know if you need additional information. How does your code connect to the 3rd party? There are libraries that do this, please elaborate which you use. I recall using one, and it had a limit on the number of concurrent connections to a 3rd party. Perhaps you hit this limit? How many connections are you talking about when the problem occurs - 10s, 100s, 1000s or more? Chris > Thank you in advance! > > Thone - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Slow HTTP Rquest via Tomcat
Hello, Can anyone give some advice on what the issue may be? We have a web application folder in Tomcat that have been slowing the http request's connection to another third party system when there are high traffic. It seems to be halting in the Tomcat server for about 20 - 50 seconds before attempting to connect to the third party system. Once it tries to connect to the third party system, the response time is only about 6 - 10 seconds which is normal. It does not seem to be an issue at the third party's end as the response time is quick once we get connected. Here are the specs that we have and additional information... 1) Apache Tomcat 7.0.56 2) JVM version 1.7 3) Running on Linux OS Version 2.6.32 Connection Flow: 1) User submits a HTTP request from web application to Tomcat 2) Tomcat calls Java's servlet code along with HTTP request parameters 3) Java servlet code connects to third party system via application server and jolt port 4) Third party system returns requested information We thought it would be a session issue which we lowered the session timeout (just for that folder) in WEB-INF folder to 5 minutes from defaulted 30 minutes but it is still doing the same thing. Any advice on what it may be? Let me know if you need additional information. Thank you in advance! Thone
