RE: Tomcat scalability setting - need help please
What sort of transaction isolation level requirements does your application have? If users are prevented from accessing the functionality of the system when a large query is being run then I would recommend checking whether the database is locking the tables. cinlung wrote: Dear Leon Let me answer your question by the number of the questions: 1. My threadpool settings: are you talking about the java threadpool or the tomcat server setting for the thread pool. If this is tomcat, I am sorry I think I set it to standard from installation. I did not change any tomcat standard settings except for the -xmx and -xms stuff. As for java, the DB thread pool is set for 200 connections at once. What I meant by slows down is that user clicks on a simple form view function that usually takes less dan a blink of an eye to open, now the application just not responding. The browser just keep loading and the load animation for IE keeps spinning, but does not return the appropriate response page and sometimes it just died (blank). When I check the tomcat logs nothing went wrong. This happens if a user run one of the heavy duty app and someone else run another app (even simple ones). Regarding multiple tomcat machine I am trying to avoid it because I would have to rewrite the entire software or get an QOS machine to disperse the weight. 2. For number 2, I did the warning for user to wait and not to close the browser before it is finished. But, a lot of them just don't even bother to read the warning and keep closing it. I was wondering if there is an API for monitoring dead users out there. Or I can always create a new API to monitor user activities by planting an applet on the user site that will keep pinging to the server and if the ping died, that means the user closed the browser. Is this wise? Or I can try to see The java Concurency thing that you mentioned below. Thank you for your advice and sorry for my bad English. -Original Message- From: Leon Rosenberg [mailto:rosenberg.l...@googlemail.com] Sent: Thursday, April 08, 2010 5:31 PM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please Hello Rendra, comments inline. On Thu, Apr 8, 2010 at 11:00 AM, Cin Lung cinl...@gmail.com wrote: Dear All Dev Sorry if repost, I got an error from the mailing list server. Can anyone help me with my problem? I have two biggest problems as follow: 1. Multi Connection Problem: I have a web application that service to multiple users. Everytime the users accessing the server reach 100 users at the same time, the tomcat would slows down. I tried to set -xmx1024 and -xms1024, but it did not have any impact at all. I tried to set the memory cache to 2048 and above, but the tomcat won't start. What are your threadpool settings? What exactly is 'slows down' If it stops responding you are obviously running out of threads. If it just gets slower, you should consider measuring the load on your machine and webapp. Maybe you should consider scaling to multiple tomcats/machines. Is there any way to improve tomcat's performance. Will there be any use of Java NIO Framework in tomcat? I mean apache has Mina, why not combine with tomcat? There are a lot of ways, but first you should identify your problem. 2. User cancellation problem Another thing that really bug me is that user would click on a web application that perform a very extensive task. The user was not patient and just close the browser accessing the app. This did not make that particular job stop. In fact the job is still running until finish and then it got no place to return the result since the user closed the browser. As the result, my server is working really hard and takes up all the resources available, and causes other users to lag. Is there any way to make that particular user task/thread stop working? The common way to handle this is to inform user that the task is going to last a bit more. To achieve this, the original request starts a background job which executes the heavy task. The original request then sends the user to a wait page, which explains that he/she has to wait for the execution. The wait page reloads itself all 1-2 seconds checking whether the background job has finished. If this happens, the wait page sends the user to the result page, where he can examine the result of the background task. There are a lot of classes like Executor, Future etc. which can help you in synchronization of background jobs. Java Concurrency in Practice is a good particular start. regards Leon Thank you in advance Rendra - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
On Fri, Apr 9, 2010 at 2:28 AM, cinl...@gmail.com wrote: As what I meant by exhaustive, I went to the extent of building my own cache scheme and it worked, the process still long, but at least it does not kill the other user, but if two or more user doing the same huge process at the same time it will still consume the server. If by same you mean, that they send the same query in parallel, before a result can be encached, you could solve this problem, by having a central cache which stores the result of a future computation (literally a flag that a result will be available soon and there is no need to send the same query again). Anyway, if you want to achieve a better insight into your application you should consider using a monitoring framework. regards Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
Rendra, --- On Thu, 4/8/10 at 5:28 PM, cinl...@gmail.com cinl...@gmail.com wrote: Do you have better way as how to transport this result to jsp? Please enlighten me. If you *really* need to serve _millions_ of rows of data to a user you will need to implement some form of paging - the query results are cached on the server and served to the user on a page-by-page basis when requested. You mentioned that you implemented your own caching scheme. I suggest you take a look at ehcache: http://ehcache.org/ It can be easily setup to automatically cache to disk and purge stale data. - Bob - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat scalability setting - need help please
Hi George, Trust me the people in my client's company would see that much of data. It is needed by the production. The next best way I can do is to send the resultset object directly to the JSP, but I would not go to that extent just yet. I am going to either build a new cache mechanism or use ehcache. The companies where I cater is data hungry company. Thanks Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Friday, April 09, 2010 12:50 AM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please Clearly instantiating millions of objects is not a strategy for scalability. You're going to have to re-structure your code to reduce the memory footprint of each session. Why is your result set returning a million rows? No human would want to see that much data. You need to restructure your queries and navigation design so that it doesn't do this. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com -Original Message- From: Cin Lung [mailto:cinl...@gmail.com] Sent: Thursday, April 08, 2010 10:53 AM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please Hi George Your remark is almost correct. What I did is that I store the result of the resultset (which can go up to million lines of rows) in a batch of Java beans. Then I set the beans to the HTTP Request and pass them to the receiving JSP. But I do remember to return the connection to the pool. I also try to kill the statements, result sets, etc by setting them to null. But I realize that java might wait for the memory to be cleared by the garbage collector. This goes back to my second problem. If the user closes the browser, the request object form the servlet would lost its way to return the result. And this will hog the tomcat performance for a while. Any tips would greatly be appreciated. TIA Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Thursday, April 08, 2010 11:42 PM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 8:49 AM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please When you run the query in your application how are you doing it, e.g. by calling a stored procedure, or by executing exactly the same SQL statement? Most likely the application is storing result sets on the session. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
Hello, Maybe you could just export those data into Excel files (any other format will do), and provide a download link to those file. Those files could be generated lazily, means generate the first time it is requested. I'm just curious, how do people in your company deal with millions of rows of data with their own eyes? On Fri, Apr 9, 2010 at 12:39 PM, Cin Lung cinl...@gmail.com wrote: Hi George, Trust me the people in my client's company would see that much of data. It is needed by the production. The next best way I can do is to send the resultset object directly to the JSP, but I would not go to that extent just yet. I am going to either build a new cache mechanism or use ehcache. The companies where I cater is data hungry company. Thanks Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Friday, April 09, 2010 12:50 AM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please Clearly instantiating millions of objects is not a strategy for scalability. You're going to have to re-structure your code to reduce the memory footprint of each session. Why is your result set returning a million rows? No human would want to see that much data. You need to restructure your queries and navigation design so that it doesn't do this. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com -Original Message- From: Cin Lung [mailto:cinl...@gmail.com] Sent: Thursday, April 08, 2010 10:53 AM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please Hi George Your remark is almost correct. What I did is that I store the result of the resultset (which can go up to million lines of rows) in a batch of Java beans. Then I set the beans to the HTTP Request and pass them to the receiving JSP. But I do remember to return the connection to the pool. I also try to kill the statements, result sets, etc by setting them to null. But I realize that java might wait for the memory to be cleared by the garbage collector. This goes back to my second problem. If the user closes the browser, the request object form the servlet would lost its way to return the result. And this will hog the tomcat performance for a while. Any tips would greatly be appreciated. TIA Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Thursday, April 08, 2010 11:42 PM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 8:49 AM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please When you run the query in your application how are you doing it, e.g. by calling a stored procedure, or by executing exactly the same SQL statement? Most likely the application is storing result sets on the session. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Sincerely yours and Best Regards, Xie Xiaodong
Re: Tomcat scalability setting - need help please
Hahaha this remark is really funny. I mean it in a good way: I'm just curious, how do people in your company deal with millions of rows of data with their own eyes? Yes they use their eyes, but before that, they print it into one rim of papers first. But customer is the king. And there are some conditions where they really need to print 6 months or more worth of data all at once. This is especially for manufacturing company. And yes I used excel as the result. It is faster, but still can do better. I wonder if I can increase the process time to less than 30 min to produce results with millions of data. Currently, with only one person doing the analysis and no one else uses the app, it takes 2 hours to finish sending the data to customer screen. Regard Rendra GOD is GREAT! -Original Message- From: Xie Xiaodong xxd82...@gmail.com Date: Fri, 9 Apr 2010 12:42:50 To: Tomcat Users Listusers@tomcat.apache.org Subject: Re: Tomcat scalability setting - need help please Hello, Maybe you could just export those data into Excel files (any other format will do), and provide a download link to those file. Those files could be generated lazily, means generate the first time it is requested. I'm just curious, how do people in your company deal with millions of rows of data with their own eyes? On Fri, Apr 9, 2010 at 12:39 PM, Cin Lung cinl...@gmail.com wrote: Hi George, Trust me the people in my client's company would see that much of data. It is needed by the production. The next best way I can do is to send the resultset object directly to the JSP, but I would not go to that extent just yet. I am going to either build a new cache mechanism or use ehcache. The companies where I cater is data hungry company. Thanks Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Friday, April 09, 2010 12:50 AM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please Clearly instantiating millions of objects is not a strategy for scalability. You're going to have to re-structure your code to reduce the memory footprint of each session. Why is your result set returning a million rows? No human would want to see that much data. You need to restructure your queries and navigation design so that it doesn't do this. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com -Original Message- From: Cin Lung [mailto:cinl...@gmail.com] Sent: Thursday, April 08, 2010 10:53 AM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please Hi George Your remark is almost correct. What I did is that I store the result of the resultset (which can go up to million lines of rows) in a batch of Java beans. Then I set the beans to the HTTP Request and pass them to the receiving JSP. But I do remember to return the connection to the pool. I also try to kill the statements, result sets, etc by setting them to null. But I realize that java might wait for the memory to be cleared by the garbage collector. This goes back to my second problem. If the user closes the browser, the request object form the servlet would lost its way to return the result. And this will hog the tomcat performance for a while. Any tips would greatly be appreciated. TIA Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Thursday, April 08, 2010 11:42 PM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 8:49 AM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please When you run the query in your application how are you doing it, e.g. by calling a stored procedure, or by executing exactly the same SQL statement? Most likely the application is storing result sets on the session. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Sincerely yours and Best Regards, Xie Xiaodong
Re: Tomcat scalability setting - need help please
On Fri, Apr 9, 2010 at 12:54 PM, cinl...@gmail.com wrote: And yes I used excel as the result. It is faster, but still can do better. I wonder if I can increase the process time to less than 30 min to produce results with millions of data. Currently, with only one person doing the analysis and no one else uses the app, it takes 2 hours to finish sending the data to customer screen. Maybe you should ask yourself, whether a webapp is really the best solution here. It sounds more like you need a central business logic server and a smart desktop app (swing or something) which can present the data in a better way, as letting the customer wait for 2 hours... Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
I think I'll stick with web apps for now :) I have my own needs and reasons for that. One of them is better maintenance with web apps. Thanks Rendra GOD is GREAT! -Original Message- From: Leon Rosenberg rosenberg.l...@googlemail.com Date: Fri, 9 Apr 2010 13:20:39 To: Tomcat Users Listusers@tomcat.apache.org Subject: Re: Tomcat scalability setting - need help please On Fri, Apr 9, 2010 at 12:54 PM, cinl...@gmail.com wrote: And yes I used excel as the result. It is faster, but still can do better. I wonder if I can increase the process time to less than 30 min to produce results with millions of data. Currently, with only one person doing the analysis and no one else uses the app, it takes 2 hours to finish sending the data to customer screen. Maybe you should ask yourself, whether a webapp is really the best solution here. It sounds more like you need a central business logic server and a smart desktop app (swing or something) which can present the data in a better way, as letting the customer wait for 2 hours... Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
On 09/04/2010 13:08, cinl...@gmail.com wrote: I think I'll stick with web apps for now :) In that case you might consider the following: 1. client requests massive dataset 2. webapp returns a processing started message, starts job in new thread 3. data processing completes and a static file is generated AND 4. a notification is sent to the client stating that the data is ready for download 5. client downloads static file, file is deleted OR 4. file is emailed to client p -Original Message- From: Leon Rosenbergrosenberg.l...@googlemail.com Date: Fri, 9 Apr 2010 13:20:39 To: Tomcat Users Listusers@tomcat.apache.org Subject: Re: Tomcat scalability setting - need help please On Fri, Apr 9, 2010 at 12:54 PM,cinl...@gmail.com wrote: And yes I used excel as the result. It is faster, but still can do better. I wonder if I can increase the process time to less than 30 min to produce results with millions of data. Currently, with only one person doing the analysis and no one else uses the app, it takes 2 hours to finish sending the data to customer screen. Maybe you should ask yourself, whether a webapp is really the best solution here. It sounds more like you need a central business logic server and a smart desktop app (swing or something) which can present the data in a better way, as letting the customer wait for 2 hours... Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
Hei! That is a great idea. All I need now is to socialize the idea. Thanks so much Rendra GOD is GREAT! -Original Message- From: Pid p...@pidster.com Date: Fri, 09 Apr 2010 13:43:42 To: Tomcat Users Listusers@tomcat.apache.org Subject: Re: Tomcat scalability setting - need help please On 09/04/2010 13:08, cinl...@gmail.com wrote: I think I'll stick with web apps for now :) In that case you might consider the following: 1. client requests massive dataset 2. webapp returns a processing started message, starts job in new thread 3. data processing completes and a static file is generated AND 4. a notification is sent to the client stating that the data is ready for download 5. client downloads static file, file is deleted OR 4. file is emailed to client p -Original Message- From: Leon Rosenbergrosenberg.l...@googlemail.com Date: Fri, 9 Apr 2010 13:20:39 To: Tomcat Users Listusers@tomcat.apache.org Subject: Re: Tomcat scalability setting - need help please On Fri, Apr 9, 2010 at 12:54 PM,cinl...@gmail.com wrote: And yes I used excel as the result. It is faster, but still can do better. I wonder if I can increase the process time to less than 30 min to produce results with millions of data. Currently, with only one person doing the analysis and no one else uses the app, it takes 2 hours to finish sending the data to customer screen. Maybe you should ask yourself, whether a webapp is really the best solution here. It sounds more like you need a central business logic server and a smart desktop app (swing or something) which can present the data in a better way, as letting the customer wait for 2 hours... Leon - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rendra, On 4/9/2010 6:54 AM, cinl...@gmail.com wrote: Yes they use their eyes, but before that, they print it into one rim of papers first. But customer is the king. And there are some conditions where they really need to print 6 months or more worth of data all at once. This is especially for manufacturing company. This sounds like reporting, and it doesn't make any sense to me why you'd offer on-demand reporting on this scale via a webapp. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku/QhgACgkQ9CaO5/Lv0PAViQCgmO7TCcfVFZaXqQzoD/9NmNKv GJAAoIoJ12qPFoS9C5Vz+nolRCyK3/nB =Rg1O -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
I'm getting the impression that the output is actually a CSV or something similar. p On 9 April 2010 16:04, Christopher Schultz ch...@christopherschultz.netwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rendra, On 4/9/2010 6:54 AM, cinl...@gmail.com wrote: Yes they use their eyes, but before that, they print it into one rim of papers first. But customer is the king. And there are some conditions where they really need to print 6 months or more worth of data all at once. This is especially for manufacturing company. This sounds like reporting, and it doesn't make any sense to me why you'd offer on-demand reporting on this scale via a webapp. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku/QhgACgkQ9CaO5/Lv0PAViQCgmO7TCcfVFZaXqQzoD/9NmNKv GJAAoIoJ12qPFoS9C5Vz+nolRCyK3/nB =Rg1O -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- -- pidster.com
Re: Tomcat scalability setting - need help please
On 08/04/2010 10:00, Cin Lung wrote: Dear All Dev Sorry if repost, I got an error from the mailing list server. Can anyone help me with my problem? I have two biggest problems as follow: 1. Multi Connection Problem: I have a web application that service to multiple users. Everytime the users accessing the server reach 100 users at the same time, the tomcat would slows down. What does your app do? What tasks is it performing? Does it use a database? I tried to set -xmx1024 and -xms1024, but it did not have any impact at all. Where did you apply those settings? Is Tomcat installed as a service? I tried to set the memory cache to 2048 and above, but the tomcat won't start. Where did you apply that setting? My current server is running AMD Athlon 64 3000+ with 8GB memory running windows server 2003 SP1. Is it running 64bit Windows? I am running tomcat 6 for the app server. Exactly which version of Tomcat 6.0.NN? Before upgrading to Windows 2003 SP1 I also had the same problem. I thought by upgrading would make a difference, but it didn't. What makes you think Tomcat and not your application is the source of your problem? If upgrading the server doesn't make a difference, perhaps this points to something that isn't affected by the processing power of the server as the source of the problem. Is there any way to improve tomcat's performance? It is possible to tune Tomcat, but in the vast majority of cases the application is the problem. Tomcat is used in many high-load situations with great success. Will there be any use of Java NIO Framework in tomcat? There is the NIO Connector. It's not guaranteed to make a difference as the usual source of the problem is in the application. I mean apache has Mina, why not combine with tomcat? Assuming there's anything wrong with Tomcat, which many people here would disagree with. 2. User cancellation problem Another thing that really bug me is that user would click on a web application that perform a very extensive task. The user was not patient and just close the browser accessing the app. This did not make that particular job stop. In fact the job is still running until finish and then it got no place to return the result since the user closed the browser. As the result, my server is working really hard and takes up all the resources available, and causes other users to lag. Is there any way to make that particular user task/thread stop working? Interrupt it? p Thank you in advance Rendra - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
Hello Rendra, comments inline. On Thu, Apr 8, 2010 at 11:00 AM, Cin Lung cinl...@gmail.com wrote: Dear All Dev Sorry if repost, I got an error from the mailing list server. Can anyone help me with my problem? I have two biggest problems as follow: 1. Multi Connection Problem: I have a web application that service to multiple users. Everytime the users accessing the server reach 100 users at the same time, the tomcat would slows down. I tried to set -xmx1024 and -xms1024, but it did not have any impact at all. I tried to set the memory cache to 2048 and above, but the tomcat won't start. What are your threadpool settings? What exactly is 'slows down' If it stops responding you are obviously running out of threads. If it just gets slower, you should consider measuring the load on your machine and webapp. Maybe you should consider scaling to multiple tomcats/machines. Is there any way to improve tomcat's performance. Will there be any use of Java NIO Framework in tomcat? I mean apache has Mina, why not combine with tomcat? There are a lot of ways, but first you should identify your problem. 2. User cancellation problem Another thing that really bug me is that user would click on a web application that perform a very extensive task. The user was not patient and just close the browser accessing the app. This did not make that particular job stop. In fact the job is still running until finish and then it got no place to return the result since the user closed the browser. As the result, my server is working really hard and takes up all the resources available, and causes other users to lag. Is there any way to make that particular user task/thread stop working? The common way to handle this is to inform user that the task is going to last a bit more. To achieve this, the original request starts a background job which executes the heavy task. The original request then sends the user to a wait page, which explains that he/she has to wait for the execution. The wait page reloads itself all 1-2 seconds checking whether the background job has finished. If this happens, the wait page sends the user to the result page, where he can examine the result of the background task. There are a lot of classes like Executor, Future etc. which can help you in synchronization of background jobs. Java Concurrency in Practice is a good particular start. regards Leon Thank you in advance Rendra - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat scalability setting - need help please
Dear Leon Let me answer your question by the number of the questions: 1. My threadpool settings: are you talking about the java threadpool or the tomcat server setting for the thread pool. If this is tomcat, I am sorry I think I set it to standard from installation. I did not change any tomcat standard settings except for the -xmx and -xms stuff. As for java, the DB thread pool is set for 200 connections at once. What I meant by slows down is that user clicks on a simple form view function that usually takes less dan a blink of an eye to open, now the application just not responding. The browser just keep loading and the load animation for IE keeps spinning, but does not return the appropriate response page and sometimes it just died (blank). When I check the tomcat logs nothing went wrong. This happens if a user run one of the heavy duty app and someone else run another app (even simple ones). Regarding multiple tomcat machine I am trying to avoid it because I would have to rewrite the entire software or get an QOS machine to disperse the weight. 2. For number 2, I did the warning for user to wait and not to close the browser before it is finished. But, a lot of them just don't even bother to read the warning and keep closing it. I was wondering if there is an API for monitoring dead users out there. Or I can always create a new API to monitor user activities by planting an applet on the user site that will keep pinging to the server and if the ping died, that means the user closed the browser. Is this wise? Or I can try to see The java Concurency thing that you mentioned below. Thank you for your advice and sorry for my bad English. -Original Message- From: Leon Rosenberg [mailto:rosenberg.l...@googlemail.com] Sent: Thursday, April 08, 2010 5:31 PM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please Hello Rendra, comments inline. On Thu, Apr 8, 2010 at 11:00 AM, Cin Lung cinl...@gmail.com wrote: Dear All Dev Sorry if repost, I got an error from the mailing list server. Can anyone help me with my problem? I have two biggest problems as follow: 1. Multi Connection Problem: I have a web application that service to multiple users. Everytime the users accessing the server reach 100 users at the same time, the tomcat would slows down. I tried to set -xmx1024 and -xms1024, but it did not have any impact at all. I tried to set the memory cache to 2048 and above, but the tomcat won't start. What are your threadpool settings? What exactly is 'slows down' If it stops responding you are obviously running out of threads. If it just gets slower, you should consider measuring the load on your machine and webapp. Maybe you should consider scaling to multiple tomcats/machines. Is there any way to improve tomcat's performance. Will there be any use of Java NIO Framework in tomcat? I mean apache has Mina, why not combine with tomcat? There are a lot of ways, but first you should identify your problem. 2. User cancellation problem Another thing that really bug me is that user would click on a web application that perform a very extensive task. The user was not patient and just close the browser accessing the app. This did not make that particular job stop. In fact the job is still running until finish and then it got no place to return the result since the user closed the browser. As the result, my server is working really hard and takes up all the resources available, and causes other users to lag. Is there any way to make that particular user task/thread stop working? The common way to handle this is to inform user that the task is going to last a bit more. To achieve this, the original request starts a background job which executes the heavy task. The original request then sends the user to a wait page, which explains that he/she has to wait for the execution. The wait page reloads itself all 1-2 seconds checking whether the background job has finished. If this happens, the wait page sends the user to the result page, where he can examine the result of the background task. There are a lot of classes like Executor, Future etc. which can help you in synchronization of background jobs. Java Concurrency in Practice is a good particular start. regards Leon Thank you in advance Rendra - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat scalability setting - need help please
Hi Pid My task is a web-based ERP application. It uses database and the number of user connection to it is up to 200 users. I use MySQL for the database. I apply the -xmx1024m and -xms1024m because I once got a message about Java Heap Out Of Space, so I searched the web and some people suggested that settings and it worked... for a while. Now that the amount of data is increased and the user using the app increased as well, lagging has been happening. The settings above I set it in the Tomcat Service monitor. And yes, I also set tomcat as service. It's running 32 Bit windows 2003 only With 8GB Ram. The tomcat version is 6.0.14. I am not assuming either tomcat is the problem or my software. I am merely trying to find a way out and I have exhausted my resources to make the software as fast as possible. Tomcat setting is the only thing that I have not explore extensively since I am not as expert as you guys. Maybe I can learn a tip or two to make things better. By the way the number of data that is being processed by the heavy app is in millions of rows. I ran the SQL directly to the mysql server and it worked ok (within minutes and not freezing the server). Meanwhile, when the query is being run via tomcat, then it will freeze the server as well. It does not kill the server, just consume all the server resources, but eventually will come back to normal after two hours with the result. Thanks Rendra -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 4:56 PM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please On 08/04/2010 10:00, Cin Lung wrote: Dear All Dev Sorry if repost, I got an error from the mailing list server. Can anyone help me with my problem? I have two biggest problems as follow: 1. Multi Connection Problem: I have a web application that service to multiple users. Everytime the users accessing the server reach 100 users at the same time, the tomcat would slows down. What does your app do? What tasks is it performing? Does it use a database? I tried to set -xmx1024 and -xms1024, but it did not have any impact at all. Where did you apply those settings? Is Tomcat installed as a service? I tried to set the memory cache to 2048 and above, but the tomcat won't start. Where did you apply that setting? My current server is running AMD Athlon 64 3000+ with 8GB memory running windows server 2003 SP1. Is it running 64bit Windows? I am running tomcat 6 for the app server. Exactly which version of Tomcat 6.0.NN? Before upgrading to Windows 2003 SP1 I also had the same problem. I thought by upgrading would make a difference, but it didn't. What makes you think Tomcat and not your application is the source of your problem? If upgrading the server doesn't make a difference, perhaps this points to something that isn't affected by the processing power of the server as the source of the problem. Is there any way to improve tomcat's performance? It is possible to tune Tomcat, but in the vast majority of cases the application is the problem. Tomcat is used in many high-load situations with great success. Will there be any use of Java NIO Framework in tomcat? There is the NIO Connector. It's not guaranteed to make a difference as the usual source of the problem is in the application. I mean apache has Mina, why not combine with tomcat? Assuming there's anything wrong with Tomcat, which many people here would disagree with. 2. User cancellation problem Another thing that really bug me is that user would click on a web application that perform a very extensive task. The user was not patient and just close the browser accessing the app. This did not make that particular job stop. In fact the job is still running until finish and then it got no place to return the result since the user closed the browser. As the result, my server is working really hard and takes up all the resources available, and causes other users to lag. Is there any way to make that particular user task/thread stop working? Interrupt it? p Thank you in advance Rendra - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
On 08/04/2010 12:19, Cin Lung wrote: Hi Pid My task is a web-based ERP application. It uses database and the number of user connection to it is up to 200 users. I use MySQL for the database. Please post your DB config from server.xml, passwords should be omitted. I apply the -xmx1024m and -xms1024m because I once got a message about Java Heap Out Of Space, so I searched the web and some people suggested that settings and it worked... for a while. Now that the amount of data is increased and the user using the app increased as well, lagging has been happening. You can use VisualVM and JConsole to examine the running JVM. Find out what is actually going on inside your application before you fiddle with settings again. Increasing the amount of memory can sometimes hide a memory leak, for example by not properly returning a database connection to a pool. The settings above I set it in the Tomcat Service monitor. And yes, I also set tomcat as service. So when you said, it didn't make any difference did you mean that the settings weren't applied, or that they were but things haven't improved? It's running 32 Bit windows 2003 only With 8GB Ram. The tomcat version is 6.0.14. The current version is 6.0.24. Upgrading is a good idea. I am not assuming either tomcat is the problem or my software. I am merely trying to find a way out and I have exhausted my resources to make the software as fast as possible. Tomcat setting is the only thing that I have not explore extensively since I am not as expert as you guys. Maybe I can learn a tip or two to make things better. I would say that 99% of the tweakable performance is related to your application and 1% is with Tomcat. You need to establish if there's a bottleneck, and if so, where it is. By the way the number of data that is being processed by the heavy app is in millions of rows. I ran the SQL directly to the mysql server and it worked ok (within minutes and not freezing the server). Meanwhile, when the query is being run via tomcat, then it will freeze the server as well. It does not kill the server, just consume all the server resources, but eventually will come back to normal after two hours with the result. So a single query is using up all the resources on the server? Is the database running on the same server as Tomcat? When you run the query in your application how are you doing it, e.g. by calling a stored procedure, or by executing exactly the same SQL statement? Thanks Rendra -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 4:56 PM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please On 08/04/2010 10:00, Cin Lung wrote: Dear All Dev Sorry if repost, I got an error from the mailing list server. Can anyone help me with my problem? I have two biggest problems as follow: 1. Multi Connection Problem: I have a web application that service to multiple users. Everytime the users accessing the server reach 100 users at the same time, the tomcat would slows down. What does your app do? What tasks is it performing? Does it use a database? I tried to set -xmx1024 and -xms1024, but it did not have any impact at all. Where did you apply those settings? Is Tomcat installed as a service? I tried to set the memory cache to 2048 and above, but the tomcat won't start. Where did you apply that setting? My current server is running AMD Athlon 64 3000+ with 8GB memory running windows server 2003 SP1. Is it running 64bit Windows? I am running tomcat 6 for the app server. Exactly which version of Tomcat 6.0.NN? Before upgrading to Windows 2003 SP1 I also had the same problem. I thought by upgrading would make a difference, but it didn't. What makes you think Tomcat and not your application is the source of your problem? If upgrading the server doesn't make a difference, perhaps this points to something that isn't affected by the processing power of the server as the source of the problem. Is there any way to improve tomcat's performance? It is possible to tune Tomcat, but in the vast majority of cases the application is the problem. Tomcat is used in many high-load situations with great success. Will there be any use of Java NIO Framework in tomcat? There is the NIO Connector. It's not guaranteed to make a difference as the usual source of the problem is in the application. I mean apache has Mina, why not combine with tomcat? Assuming there's anything wrong with Tomcat, which many people here would disagree with. 2. User cancellation problem Another thing that really bug me is that user would click on a web application that perform a very extensive task. The user was not patient and just close the browser accessing the app. This did not make that particular job stop. In fact the job is still running until finish and then it got no place to return the result since the user closed the browser. As the result, my
RE: Tomcat scalability setting - need help please
-Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 8:49 AM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please When you run the query in your application how are you doing it, e.g. by calling a stored procedure, or by executing exactly the same SQL statement? Most likely the application is storing result sets on the session. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat scalability setting - need help please
Hi George Your remark is almost correct. What I did is that I store the result of the resultset (which can go up to million lines of rows) in a batch of Java beans. Then I set the beans to the HTTP Request and pass them to the receiving JSP. But I do remember to return the connection to the pool. I also try to kill the statements, result sets, etc by setting them to null. But I realize that java might wait for the memory to be cleared by the garbage collector. This goes back to my second problem. If the user closes the browser, the request object form the servlet would lost its way to return the result. And this will hog the tomcat performance for a while. Any tips would greatly be appreciated. TIA Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Thursday, April 08, 2010 11:42 PM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 8:49 AM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please When you run the query in your application how are you doing it, e.g. by calling a stored procedure, or by executing exactly the same SQL statement? Most likely the application is storing result sets on the session. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat scalability setting - need help please
Clearly instantiating millions of objects is not a strategy for scalability. You're going to have to re-structure your code to reduce the memory footprint of each session. Why is your result set returning a million rows? No human would want to see that much data. You need to restructure your queries and navigation design so that it doesn't do this. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com -Original Message- From: Cin Lung [mailto:cinl...@gmail.com] Sent: Thursday, April 08, 2010 10:53 AM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please Hi George Your remark is almost correct. What I did is that I store the result of the resultset (which can go up to million lines of rows) in a batch of Java beans. Then I set the beans to the HTTP Request and pass them to the receiving JSP. But I do remember to return the connection to the pool. I also try to kill the statements, result sets, etc by setting them to null. But I realize that java might wait for the memory to be cleared by the garbage collector. This goes back to my second problem. If the user closes the browser, the request object form the servlet would lost its way to return the result. And this will hog the tomcat performance for a while. Any tips would greatly be appreciated. TIA Rendra -Original Message- From: George Sexton [mailto:geor...@mhsoftware.com] Sent: Thursday, April 08, 2010 11:42 PM To: 'Tomcat Users List' Subject: RE: Tomcat scalability setting - need help please -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Thursday, April 08, 2010 8:49 AM To: Tomcat Users List Subject: Re: Tomcat scalability setting - need help please When you run the query in your application how are you doing it, e.g. by calling a stored procedure, or by executing exactly the same SQL statement? Most likely the application is storing result sets on the session. George Sexton MH Software, Inc. 303 438-9585 www.mhsoftware.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rendra, At the risk of getting sucked into the insanity... On 4/8/2010 7:19 AM, Cin Lung wrote: It's running 32 Bit windows 2003 only With 8GB Ram. 32-bit Microsoft Windows can access 8GiB of RAM (much more, in fact), but each process is still limited to a 4GiB address space, and practically Windows will never give you that much anyway. If you want to use more than roughly 1800MiB for Tomcat, you'll have to go to 64-bit Microsoft Windows. I am merely trying to find a way out and I have exhausted my resources to make the software as fast as possible. You didn't tell us what exhaustive measures you're already tried. By the way the number of data that is being processed by the heavy app is in millions of rows. I ran the SQL directly to the mysql server and it worked ok (within minutes and not freezing the server). That suggests one of a few things: 1. Remoteness (that is, not running on the server running MySQL) has a penalty 2. Your webapp isn't building the query you think it is 3. Your webapp is doing much more with the results than simply listing them Let's examine those possibilities one at a time, shall we? First, if remoteness is incurring a penalty, it's generally because you're transmitting a lot of data. Are you transmitting millions of rows from the server to the client? Why? Is this something that can be done in a stored procedure on the server, or even on a separate, server-only process? IIRC Connector/J, in its default configuration, downloads the entire result set before returning control to your code. That means that if you are selecting millions of rows, you have to wait for them all to go from the server to your webapp, and they all take up a bunch of memory while you're working with them. Have you verified this is not happening to you? Second, have you dumped-out the query your webapp is building to make sure it's the one you think is running? Third, is your webapp doing anything else with these rows other than, say, performing a simple examination of them? I can't imagine why a web application would need to fetch millions of rows at once. I can understand fetching a small portion of millions of rows at once (e.g. SELECT a,b,c FROM huge_table LIMIT 1,100) and displaying them to the user, but never actually transferring that many rows. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku+VB8ACgkQ9CaO5/Lv0PAxTQCeLdkcbe7sdKz4M3/8ScXCYmRl 4XAAnRbd5z3H4gl7b8Hhxj1n8eAW9G1H =pieL -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rendra, On 4/8/2010 12:53 PM, Cin Lung wrote: Your remark is almost correct. What I did is that I store the result of the resultset (which can go up to million lines of rows) in a batch of Java beans. Then I set the beans to the HTTP Request and pass them to the receiving JSP. This will kill your server every time. It is a bad plan in general and will simply not scale in any way. Consider changing... everything. But I do remember to return the connection to the pool. Super! I also try to kill the statements, result sets, etc by setting them to null. It's better to call [whatever].close(). But I realize that java might wait for the memory to be cleared by the garbage collector. If you're storing all that stuff in beans in the session, you're toast. This goes back to my second problem. If the user closes the browser, the request object form the servlet would lost its way to return the result. And this will hog the tomcat performance for a while. If you are sending millions of rows to your client, it's no wonder they are closing their web browser. :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku+VMAACgkQ9CaO5/Lv0PBl2wCfa2YlxlZe6uacPKSc2coaji7Y bl8AoKguMz08zZNU/plxn8moq5wLLAZt =wM9J -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
Hi Chris I stored the result bean in the http request object, NOT IN THE SESSION OBJECT, hoping that once the result is delivered, the beans will die with the request object since request object. I also called the necessary .close() methods and then set the conn, statement, rs, object itself to null. Do you have better way as how to transport this result to jsp? Please enlighten me. Fyi, I must process and send millions of data since my customers usually run analysis processes from 15 different tables with hundred of thousands of data from each table. It is an analysis of 6 or more months of manufacturing data combined with marketing, purchasing, inventory mutations, pricing, production monitoring etc. As what I meant by exhaustive, I went to the extent of building my own cache scheme and it worked, the process still long, but at least it does not kill the other user, but if two or more user doing the same huge process at the same time it will still consume the server. TIA Rendra GOD is GREAT! -Original Message- From: Christopher Schultz ch...@christopherschultz.net Date: Thu, 08 Apr 2010 18:12:16 To: Tomcat Users Listusers@tomcat.apache.org Subject: Re: Tomcat scalability setting - need help please -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rendra, On 4/8/2010 12:53 PM, Cin Lung wrote: Your remark is almost correct. What I did is that I store the result of the resultset (which can go up to million lines of rows) in a batch of Java beans. Then I set the beans to the HTTP Request and pass them to the receiving JSP. This will kill your server every time. It is a bad plan in general and will simply not scale in any way. Consider changing... everything. But I do remember to return the connection to the pool. Super! I also try to kill the statements, result sets, etc by setting them to null. It's better to call [whatever].close(). But I realize that java might wait for the memory to be cleared by the garbage collector. If you're storing all that stuff in beans in the session, you're toast. This goes back to my second problem. If the user closes the browser, the request object form the servlet would lost its way to return the result. And this will hog the tomcat performance for a while. If you are sending millions of rows to your client, it's no wonder they are closing their web browser. :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku+VMAACgkQ9CaO5/Lv0PBl2wCfa2YlxlZe6uacPKSc2coaji7Y bl8AoKguMz08zZNU/plxn8moq5wLLAZt =wM9J -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat scalability setting - need help please
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rendra, On 4/8/2010 8:28 PM, cinl...@gmail.com wrote: I stored the result bean in the http request object, NOT IN THE SESSION OBJECT, hoping that once the result is delivered, the beans will die with the request object since request object. They will, but if you are selecting millions of rows and storing them in the request, you're probably grabbing /way/ too many rows for a single page to display reasonably: you're wasting memory, network bandwidth, and CPU time to manage all that. It's no wonder your webapp doesn't scale. Fyi, I must process and send millions of data since my customers usually run analysis processes from 15 different tables with hundred of thousands of data from each table. It is an analysis of 6 or more months of manufacturing data combined with marketing, purchasing, inventory mutations, pricing, production monitoring etc. How many rows are actually being sent back to the client? Millions, or less than that? As what I meant by exhaustive, I went to the extent of building my own cache scheme and it worked, the process still long, but at least it does not kill the other user, but if two or more user doing the same huge process at the same time it will still consume the server. That doesn't sound like a caching scheme that works. Back to the original question: Tomcat is scaling just fine: it's allowing users to connect very quickly, while your web application is choking itself and probably the db server. There is no setting in Tomcat to make your webapp run better. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku+g7AACgkQ9CaO5/Lv0PCerwCfXELH2klREvz8jseSI72Z+cS4 a5cAoK8xqknjL8LvAyyaVHyvF6lyDyOu =0ehh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
