Re: pick load
Le jeudi 02 septembre 2010 à 11:22 -0400, Christopher Schultz a écrit : -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alexandre, On 9/2/2010 12:08 AM, Alexandre Chapellon wrote: Forget about it this doesn't seems to be related to the jk connector. Right: the jk connector can be tweaked separately. You appear to have other problems. I have the same problems when direct sending requests to tomcat (errors which do not appear under nomal load): 2010-09-01 18:06:53.459 - FAILURE - [tracing] - MANA - Exception dans [Authentification::doAuthentification(String,String)] : [Search::searchPartyByAddInfo(String,object)] Erreur : impossible de recuperer les parties avec l'info additionnel accessCode = CHAP0712A : null 2010-09-01 18:06:53.459 - FAILURE - [tracing] - MANA - [Authentification::performTask(HttpServletRequest,HttpServletResponse)] Exception rencontr?e pendant l'authentification de CHAP0712A. 2010-09-01 18:06:53.467 - FAILURE - [tracing] - MANA - Exception dans [Search::searchParty(String)] : null 2010-09-01 18:06:53.471 - FAILURE - [tracing] - MANA - Exception dans [Client.fetchInformation(String)] :[Search::searchParty(String)] Erreur : impossible de recuperer la partie CHAP0712A : null java.lang.Exception: [Search::searchParty(String)] Erreur : impossible de recuperer la partie CHAP0712A : null at com.mana.om.Client.fetchInformation(Client.java:676) at com.mana.selfcare.Authenticate.performTask(Authenticate.java:207) at com.mana.selfcare.Authenticate.doPost(Authenticate.java:84) That looks like an application error to me. Yes to me too, but what's weired is that thoose errors enver appears under normal load... this really drives me crazy! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkx/wTMACgkQ9CaO5/Lv0PDXFgCgrMYix3jPszsSdOotB2qyZ9+i DBIAnjm44KkSTGLwRtl6GswN/njUC5bD =ZDt3 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: pick load
With load growing I have thoose errors that appear in mod_jk logs: [Sun Aug 29 08:28:27.976 2010] [26242:4106161040] [info] jk_handler::mod_jk.c (2611): Aborting connection for worker=selfcare [Sun Aug 29 12:15:45.829 2010] [27162:4097772432] [info] ajp_service::jk_ajp_common.c (2540): (selfcare) sending request to tomcat failed (recoverable), because of server error (attempt=1) [Sun Aug 29 12:15:45.929 2010] [27162:4097772432] [info] ajp_send_request::jk_ajp_common.c (1490): (selfcare) did not receive END_RESPONSE, closing socket -1 [Sun Aug 29 12:15:45.933 2010] [27162:4097772432] [info] ajp_service::jk_ajp_common.c (2540): (selfcare) sending request to tomcat failed (recoverable), because of server error (attempt=2) [Sun Aug 29 12:15:45.933 2010] [27162:4097772432] [error] ajp_service::jk_ajp_common.c (2559): (selfcare) connecting to tomcat failed. [Sun Aug 29 12:15:45.933 2010] [27162:4097772432] [info] jk_handler::mod_jk.c (2618): Service error=-3 for worker=selfcare [Sun Aug 29 15:04:46.507 2010] [27104:4039052176] [info] ajp_process_callback::jk_ajp_common.c (1882): Writing to client aborted or client network problems ...skipping... I really don't know what to tune Le mardi 31 août 2010 à 17:49 -0500, Caldarale, Charles R a écrit : From: Alexandre Chapellon [mailto:alexandre.chapel...@mana.pf] Subject: Re: pick load We're using an old version of highdeal billing system which apparently (tat's what the support says) doens't support java6. Chris suggested *Tomcat* 6, not Java 6. You can run Tomcat 6 on JRE 5. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
RE: pick load
Forget about it this doesn't seems to be related to the jk connector. I have the same problems when direct sending requests to tomcat (errors which do not appear under nomal load): 2010-09-01 18:06:53.459 - FAILURE - [tracing] - MANA - Exception dans [Authentification::doAuthentification(String,String)] : [Search::searchPartyByAddInfo(String,object)] Erreur : impossible de recuperer les parties avec l'info additionnel accessCode = CHAP0712A : null 2010-09-01 18:06:53.459 - FAILURE - [tracing] - MANA - [Authentification::performTask(HttpServletRequest,HttpServletResponse)] Exception rencontr?e pendant l'authentification de CHAP0712A. 2010-09-01 18:06:53.467 - FAILURE - [tracing] - MANA - Exception dans [Search::searchParty(String)] : null 2010-09-01 18:06:53.471 - FAILURE - [tracing] - MANA - Exception dans [Client.fetchInformation(String)] :[Search::searchParty(String)] Erreur : impossible de recuperer la partie CHAP0712A : null java.lang.Exception: [Search::searchParty(String)] Erreur : impossible de recuperer la partie CHAP0712A : null at com.mana.om.Client.fetchInformation(Client.java:676) at com.mana.selfcare.Authenticate.performTask(Authenticate.java:207) at com.mana.selfcare.Authenticate.doPost(Authenticate.java:84) at javax.servlet.http.HttpServlet.service(HttpServlet.java:638) at javax.servlet.http.HttpServlet.service(HttpServlet.java:720) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:145) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:596) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:139) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:596) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:446) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:594) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:545) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:594) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955) at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2460) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:133) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:596) at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:119) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:594) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:116) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:594) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:127) at org.apache.catalina.core.StandardPipeline $StandardPipelineValveContext.invokeNext(StandardPipeline.java:596) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:433) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:955) at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:157) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket $SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool $ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:534) Le mercredi 01 septembre 2010 à 17:28 -1000, Alexandre Chapellon a écrit : With load growing I have thoose errors that appear in mod_jk logs: [Sun Aug 29 08:28:27.976 2010] [26242:4106161040] [info] jk_handler::mod_jk.c (2611): Aborting connection
Re: pick load
Le mardi 31 août 2010 à 14:39 -0400, Christopher Schultz a écrit : -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alexandre, On 8/30/2010 10:49 PM, Alexandre Chapellon wrote: [Considering] the fair analysis of Charles, we decided to move applications to Poolable connection factory Excellent. To do so I created a factory iun server.xml as follow: Resource name=jdbc/crm auth=Container type=javax.sql.DataSource/ I just realized that you're using Tomcat 4.1. Any possibility of moving up to Tomcat 6.0? I've recently moved several webapps from 4.1 up to 6.0 with very little headache. You'll get the benefit of security and stability updates, as well as performance improvements in the new version. We're using an old version of highdeal billing system which apparently (tat's what the support says) doens't support java6. Migration is planned for next year. ResourceParams name=jdbc/crm parameter nameurl/name valuejdbc:oracle:thin:@1.2.3.4:1521:SID/value Double-check that this driver URL is correct. If you copied it from your old configuration, it's probably fine. Indeed the url what good for one of the pool (oracle9i) but not for the other (Oracle10g) I Changed it, and so get rid of error messages. 4 parameter namefactory/name valueorg.apache.commons.dbcp.BasicDataSourceFactory/value /parameter This parameter should not be necessary. Once you have things working, try removing this parameter. If things are still working fine, then leave this out of your configuration. When trying to connect to the application. Login succeed but I get the following errors in catalina.out: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (invalid argument in call) Can you post the code that fetches a connection from the pool? Did i miss something in xml definitions or is it a problem with the way the app uses the connection Pool? My guess is the location of the driver .jar file. The file oracle.jar (or whatever it is called) should /only/ be in the server's server/lib directory, and /not/ in the webapp's WEB-INF/lib directory. If you have the driver .jar in both places (or only in WEB-INF/lib), you'll get errors like this. I removed the factory param and redundant oracle jar files. Thanks for your help - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkx9TEwACgkQ9CaO5/Lv0PCgHQCfcGRWdjQ8UHYpUizRn49GbD4p D/kAn0V5rZ2rjtOjyfBZcQH0UcmZsXdq =7a6L -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: pick load
Consering the fair analisys of Charles, we decided to move applications to Poolable connection factory To do so I created a factory iun server.xml as follow: Resource name=jdbc/crm auth=Container type=javax.sql.DataSource/ ResourceParams name=jdbc/crm parameter nameurl/name valuejdbc:oracle:thin:@1.2.3.4:1521:SID/value /parameter parameter nameuser/name valuescott/value /parameter parameter namepassword/name valuetiger/value /parameter parameter namemaxActive/name value60/value /parameter parameter namefactory/name valueorg.apache.commons.dbcp.BasicDataSourceFactory/value /parameter parameter namedriverClassName/name valueoracle.jdbc.OracleDriver/value /parameter /ResourceParams and in web.xml of the app, I added at the end: resource-ref description Database connection pool /description res-ref-namejdbc/crm/res-ref-name res-typejavax.sql.DataSource/res-type res-authContainer/res-auth /resource-ref When trying to connect to the application. Login succeed but I get the following errors in catalina.out: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (invalid argument in call) Did i miss something in xml definitions or is it a problem with the way the app uses the connection Pool? regards Le jeudi 26 août 2010 à 22:04 -0500, Caldarale, Charles R a écrit : From: Alexandre Chapellon [mailto:alexandre.chapel...@mana.pf] Subject: Re: pick load To me this sounds like a pool of database connection that is full It's not a full connection pool - you're not using one. It's also likely that your database is a bit sluggish in establishing connections. That version of the JVM sets a global lock while creating a connection, with the expectation that it will occur fairly quickly, allowing the next connection to be established. Note that this has nothing to do with Tomcat. I don't really know how/where to deal with it. Fix your DB, and configure a connection pool rather than calling com.mana.oc.DBConnection.getConnectionCRM directly, so each request doesn't have to acquire a new connection. Also upgrade to supported versions of things, since both the level of Tomcat you're on and the JVM have been unsupported for quite some time. The most recent JVMs do not keep that particular global lock up on the connection request to the DB, so moving to Java 6 might help. If someone has the kind willing to take a look at the full dump here it is: http://pastebin.com/2v3PVTDm A brief look showed only one thread having an active DB connection, with all of the rest stuck on the global lock trying to get a connection in order to authenticate the client or else just waiting for something to do. Since you're not using a connection pool for the authentications, you're serializing the requests, so it's no wonder your throughput is terrible. Even the greatly improved performance of newer Tomcat and JVM versions won't overcome bad webapp design. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: pick load
ab doesn't says much... except the requests completed and most of them did in less than 5 seconds: ### Server Software:Apache Server Hostname:blablablabla.hostname Server Port:443 SSL/TLS Protocol: TLSv1/SSLv3,DHE-RSA-AES256-SHA,1024,256 Document Path: /path/to/Login.jsp Document Length:5576 bytes Concurrency Level: 50 Time taken for tests: 81.949 seconds Complete requests: 1648 Failed requests:0 Write errors: 0 Total transferred: 9752864 bytes HTML transferred: 9189248 bytes Requests per second:20.11 [#/sec] (mean) Time per request: 2486.315 [ms] (mean) Time per request: 49.726 [ms] (mean, across all concurrent requests) Transfer rate: 116.22 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 29 55 114.8 351425 Processing:74 2392 377.9 22893589 Waiting: 73 2390 377.8 22863587 Total:128 2447 363.8 23304256 Percentage of the requests served within a certain time (ms) 50% 2330 66% 2498 75% 2596 80% 2688 90% 2892 95% 3075 98% 3421 99% 3520 100% 4256 (longest request) Not sure it helps... It's the jkmanger that shows 20req/s as max thx Le jeudi 26 août 2010 à 07:56 +0200, Domenico Briganti a écrit : Il giorno mer, 25/08/2010 alle 15.28 -1000, Alexandre Chapellon ha scritto: P.S: right now am using ab to send 2000 request with 50 concurrents. What is the report of ab? Domenico - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: pick load
Le jeudi 26 août 2010 à 09:25 +0200, Rainer Jung a écrit : On 26.08.2010 03:28, Alexandre Chapellon wrote: Hello, I'm quite new to tomcat and have an old webapps running on tomcat 4.1 and jvm 1.4.2 with apach2.2 in front ofthem (using modjk). I'm trying to get ready for a comming pick load I will have to face. I Try to do some benchmark using ab and the jkstatus worker. Whatever the configuration of my connecter (both on the apache or tomcat side) I never go upper than 20 requests / second. Here are few parameters I changed in order to get better performances: -Apache2 (worker): increased ServerLimit (64), ThreadLimit (256), MaxClients (2048), ThreadsPerChild (128) set to a non zero value MaxRequestsPerChild (500) - modjk (1.2.30): set to non-zero value worker.selfcare.connection_pool_timeout=60 -Tomcat AJP13 Connector: acceptCount=50 enableLookups=false maxProcessors=500 bufferSize=4096 socketBuffer=2 Unfortunately this doesn't help and am still stuck with 20req/s when the machines' load is not that high and 60% of CPU at most is used during stress test. I've googled around but can't find anything else about increasing performances of apache/tomcat... Help much appreciated Regards P.S: right now am using ab to send 2000 request with 50 concurrents. Take thread dumps of the Tomcat JVM and check what your applicaion is actually doing (like waiting for locks or externals components). This sounds an excellent idea indeed, and it's surely what I would have done if I new it was possible and how I could do it :) What's the way to do it? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: pick load
Thanks. I got the dump of the running JVM (only the benchmark is running at dump time). I tried to take a look at it, but as am not familiar with java, I prefer talk about it here. -first I noticed the number of TP-Processor is twice the number of concurrent resquests send by ab (why twice? i don't know). About half of them are in Object.wait state TP-Processor40 daemon prio=1 tid=0xdef53168 nid=0x646f in Object.wait() [0xda729000..0xda7291b8] at java.lang.Object.wait(Native Method) - waiting on 0xe17bc808 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:429) at org.apache.tomcat.util.threads.ThreadPool $ControlRunnable.run(ThreadPool.java:661) - locked 0xe17bc808 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:534) and the other half is waiting for a monitor entry. TP-Processor34 daemon prio=1 tid=0xdef3edc0 nid=0x6469 waiting for monitor entry [0xdaa28000..0xdaa291b8] at java.sql.DriverManager.getConnection(DriverManager.java:158) - waiting to lock 0xeff0b350 (a java.lang.Class) at com.mana.oc.DBConnection.getConnectionCRM(DBConnection.java:155) at org.apache.jsp.Login_jsp._jspService(Login_jsp.java:192) .. And only 5 runnable processors. To me this sounds like a pool of database connection that is full but I'm not sure of it and I don't really know how/where to deal with it. If someone has the kind willing to take a look at the full dump here it is: http://pastebin.com/2v3PVTDm Regards Le jeudi 26 août 2010 à 21:36 +0200, Rainer Jung a écrit : On 26.08.2010 21:00, Alexandre Chapellon wrote: Le jeudi 26 août 2010 à 09:25 +0200, Rainer Jung a écrit : On 26.08.2010 03:28, Alexandre Chapellon wrote: Hello, I'm quite new to tomcat and have an old webapps running on tomcat 4.1 and jvm 1.4.2 with apach2.2 in front ofthem (using modjk). I'm trying to get ready for a comming pick load I will have to face. I Try to do some benchmark using ab and the jkstatus worker. Whatever the configuration of my connecter (both on the apache or tomcat side) I never go upper than 20 requests / second. Here are few parameters I changed in order to get better performances: -Apache2 (worker): increased ServerLimit (64), ThreadLimit (256), MaxClients (2048), ThreadsPerChild (128) set to a non zero value MaxRequestsPerChild (500) - modjk (1.2.30): set to non-zero value worker.selfcare.connection_pool_timeout=60 -Tomcat AJP13 Connector: acceptCount=50 enableLookups=false maxProcessors=500 bufferSize=4096 socketBuffer=2 Unfortunately this doesn't help and am still stuck with 20req/s when the machines' load is not that high and 60% of CPU at most is used during stress test. I've googled around but can't find anything else about increasing performances of apache/tomcat... Help much appreciated Regards P.S: right now am using ab to send 2000 request with 50 concurrents. Take thread dumps of the Tomcat JVM and check what your applicaion is actually doing (like waiting for locks or externals components). This sounds an excellent idea indeed, and it's surely what I would have done if I new it was possible and how I could do it :) What's the way to do it? http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: pick load
Le jeudi 26 août 2010 à 22:04 -0500, Caldarale, Charles R a écrit : From: Alexandre Chapellon [mailto:alexandre.chapel...@mana.pf] Subject: Re: pick load To me this sounds like a pool of database connection that is full It's not a full connection pool - you're not using one. It's also likely that your database is a bit sluggish in establishing connections. That version of the JVM sets a global lock while creating a connection, with the expectation that it will occur fairly quickly, allowing the next connection to be established. Note that this has nothing to do with Tomcat. I don't really know how/where to deal with it. Fix your DB, and configure a connection pool rather than calling com.mana.oc.DBConnection.getConnectionCRM directly, so each request doesn't have to acquire a new connection. Also upgrade to supported versions of things, since both the level of Tomcat you're on and the JVM have been unsupported for quite some time. The most recent JVMs do not keep that particular global lock up on the connection request to the DB, so moving to Java 6 might help. If someone has the kind willing to take a look at the full dump here it is: http://pastebin.com/2v3PVTDm A brief look showed only one thread having an active DB connection, with all of the rest stuck on the global lock trying to get a connection in order to authenticate the client or else just waiting for something to do. Since you're not using a connection pool for the authentications, you're serializing the requests, so it's no wonder your throughput is terrible. Even the greatly improved performance of newer Tomcat and JVM versions won't overcome bad webapp design. - Chuck Thank you Chuck I'm so glad to hear this! THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
pick load
Hello, I'm quite new to tomcat and have an old webapps running on tomcat 4.1 and jvm 1.4.2 with apach2.2 in front ofthem (using modjk). I'm trying to get ready for a comming pick load I will have to face. I Try to do some benchmark using ab and the jkstatus worker. Whatever the configuration of my connecter (both on the apache or tomcat side) I never go upper than 20 requests / second. Here are few parameters I changed in order to get better performances: -Apache2 (worker): increased ServerLimit (64), ThreadLimit (256), MaxClients (2048), ThreadsPerChild (128) set to a non zero value MaxRequestsPerChild (500) - modjk (1.2.30): set to non-zero value worker.selfcare.connection_pool_timeout=60 -Tomcat AJP13 Connector: acceptCount=50 enableLookups=false maxProcessors=500 bufferSize=4096 socketBuffer=2 Unfortunately this doesn't help and am still stuck with 20req/s when the machines' load is not that high and 60% of CPU at most is used during stress test. I've googled around but can't find anything else about increasing performances of apache/tomcat... Help much appreciated Regards P.S: right now am using ab to send 2000 request with 50 concurrents.
