No more than 500 threads on Windows 2003?
Hi, I'm currently integrating a SOAP-based middleware on Tomcat (5.5.9) with Sun JDK 1.5.0_02 and get into problems when using maxThreads with a value greater than 500. Tomcat locks up completely without getting back to normal operations after a while. No exception or error is reported in the logs, it just stops. All connections are refused after the lockup. Tomcat has enough memory (>1GB) assigned with the JVM options. If I use maxThreads="500", the server runs fine, but 500 is not enough for my project ;-( I tried to increase the stack size, but this doesn't help.. The connector is defined as: If I change 500 to 700 or more, Tomcat locks up. Is this a Windows problem?? Hardware is not an issue, I think.. (dual Xeon 3,4GHz, 4GB).. Thanks, Michael -- Michael Kleinhenz tarent GmbH . Bahnhofstr. 13 . 53123 Bonn fon +49 (228) / 52 67 5-0 . fax +49 (228) / 52 67 5-25 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: No more than 500 threads on Windows 2003?
maybe. I can't confirm, because it fails so fast, I can't take a look in the manager app to see the exact number of threads when it begins hanging. I experienced the problem with Tomcat 5.0.28+JDK1.4.2 and Tomcat 5.5.9+JDK1.5.0 while my (somewhat slower) notebook can do 800+ threads with Debian and 5.0.28+1.4.2 without problems. Seems to me like a Windows problem, but I can't find anything in the docs nor via Google.. ;-( Windows sucks.. -- Michael delbd schrieb: > Just by curiosity, does it have something like '511 Threads is ok but 512 > fails?' :D > Le Jeudi 30 Juin 2005 15:09, Michael Kleinhenz a écrit : > >>Hi, >> >>I'm currently integrating a SOAP-based middleware on Tomcat (5.5.9) with >>Sun JDK 1.5.0_02 and get into problems when using maxThreads with a >>value greater than 500. >> >>Tomcat locks up completely without getting back to normal operations >>after a while. No exception or error is reported in the logs, it just >>stops. All connections are refused after the lockup. Tomcat has enough >>memory (>1GB) assigned with the JVM options. >> >>If I use maxThreads="500", the server runs fine, but 500 is not enough >>for my project ;-( >> >>I tried to increase the stack size, but this doesn't help.. >> >>The connector is defined as: >> >>> minSpareThreads="50" maxSpareThreads="150" enableLookups="false" >> redirectPort="8443" acceptCount="200" connectionTimeout="2" >> disableUploadTimeout="true"/> >> >>If I change 500 to 700 or more, Tomcat locks up. Is this a Windows >>problem?? >> >>Hardware is not an issue, I think.. (dual Xeon 3,4GHz, 4GB).. >> >>Thanks, >>Michael >> > > -- Dipl.-Technoinform Michael Kleinhenz tarent GmbH . Bahnhofstraße 13 . 53123 Bonn fon: +49 (228) / 52 67 5-0 fax: +49 (228) / 52 67 5-25 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: No more than 500 threads on Windows 2003?
as far as I know, the concept of green and native threads only applies to Unix systems...? -- Michael On Fri, Jul 01, 2005 at 10:44:38AM +0200, delbd wrote: > Maybe you could switch the jvm to green threads ? > Le Vendredi 1 Juillet 2005 09:57, Michael Kleinhenz a écrit : > > > > maybe. I can't confirm, because it fails so fast, I can't take a look in > > the manager app to see the exact number of threads when it begins hanging. > > > > I experienced the problem with Tomcat 5.0.28+JDK1.4.2 and Tomcat > > 5.5.9+JDK1.5.0 while my (somewhat slower) notebook can do 800+ threads > > with Debian and 5.0.28+1.4.2 without problems. > > > > Seems to me like a Windows problem, but I can't find anything in the > > docs nor via Google.. ;-( > > > > Windows sucks.. > > > > -- Michael > > > > delbd schrieb: > > > Just by curiosity, does it have something like '511 Threads is ok but 512 > > > fails?' :D > > > Le Jeudi 30 Juin 2005 15:09, Michael Kleinhenz a écrit : > > > > > >>Hi, > > >> > > >>I'm currently integrating a SOAP-based middleware on Tomcat (5.5.9) with > > >>Sun JDK 1.5.0_02 and get into problems when using maxThreads with a > > >>value greater than 500. > > >> > > >>Tomcat locks up completely without getting back to normal operations > > >>after a while. No exception or error is reported in the logs, it just > > >>stops. All connections are refused after the lockup. Tomcat has enough > > >>memory (>1GB) assigned with the JVM options. > > >> > > >>If I use maxThreads="500", the server runs fine, but 500 is not enough > > >>for my project ;-( > > >> > > >>I tried to increase the stack size, but this doesn't help.. > > >> > > >>The connector is defined as: > > >> > > >> > >> minSpareThreads="50" maxSpareThreads="150" enableLookups="false" > > >> redirectPort="8443" acceptCount="200" connectionTimeout="2" > > >> disableUploadTimeout="true"/> > > >> > > >>If I change 500 to 700 or more, Tomcat locks up. Is this a Windows > > >>problem?? > > >> > > >>Hardware is not an issue, I think.. (dual Xeon 3,4GHz, 4GB).. > > >> > > >>Thanks, > > >>Michael > > >> > > > > > > > > > > > > -- > David Delbecq > Royal Meteorological Institute of Belgium > > - > Is there life after /sbin/halt -p? > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > -- Michael Kleinhenz tarent GmbH . Bahnhofstr. 13 . 53123 Bonn fon +49 (228) / 52 67 5-0 . fax +49 (228) / 52 67 5-25 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: No more than 500 threads on Windows 2003?
mmh.. "You may hear the terms "green threads" and "native threads". This refers to the Unix implementations of Java. When Java 1.0 first came out on Solaris, it did not use the native Solaris library libthread.so to support threads. Instead it used runtime thread support that had been written in Java for an earlier project code-named "Green." That thread library came to be known as "green threads." "Thread support on Windows used the native libraries from the start, but on Solaris green threads were used. That meant that threaded programs running on Solaris didn't get the benefit of the native libraries. In particular, if your Java program was executing on a multi processor Solaris system, it was never the case that different threads could execute truly in parallel by being dispatched on different processors. There are other benefits too, to do with more efficient mapping of I/O system calls." -> Just Java 2, Fourth Edition, by Peter van der Linden So Windows always uses native threads. Seems to me like a Windows system or license limit. I have already asked tech support of the hosting provider who installed the server, but no answer yet. -- Michael On Fri, Jul 01, 2005 at 11:02:44AM +0200, delbd wrote: > I may be wrong, but as far as my knowledge goes on green threads, this is > just a matter of > dedactivating native system thread to use an internal threading / sheduling > system in the jvm. > I see no reason why this wouldn't exist on windows version of jvm. > > Le Vendredi 1 Juillet 2005 10:39, Michael Kleinhenz a écrit : > > > > as far as I know, the concept of green and native threads only applies > > to Unix systems...? > > > > -- Michael > > > > > > On Fri, Jul 01, 2005 at 10:44:38AM +0200, delbd wrote: > > > Maybe you could switch the jvm to green threads ? > > > Le Vendredi 1 Juillet 2005 09:57, Michael Kleinhenz a écrit : > > > > > > > > maybe. I can't confirm, because it fails so fast, I can't take a look > > > > in > > > > the manager app to see the exact number of threads when it begins > > > > hanging. > > > > > > > > I experienced the problem with Tomcat 5.0.28+JDK1.4.2 and Tomcat > > > > 5.5.9+JDK1.5.0 while my (somewhat slower) notebook can do 800+ threads > > > > with Debian and 5.0.28+1.4.2 without problems. > > > > > > > > Seems to me like a Windows problem, but I can't find anything in the > > > > docs nor via Google.. ;-( > > > > > > > > Windows sucks.. > > > > > > > > -- Michael > > > > > > > > delbd schrieb: > > > > > Just by curiosity, does it have something like '511 Threads is ok but > > > > > 512 fails?' :D > > > > > Le Jeudi 30 Juin 2005 15:09, Michael Kleinhenz a écrit : > > > > > > > > > >>Hi, > > > > >> > > > > >>I'm currently integrating a SOAP-based middleware on Tomcat (5.5.9) > > > > >>with > > > > >>Sun JDK 1.5.0_02 and get into problems when using maxThreads with a > > > > >>value greater than 500. > > > > >> > > > > >>Tomcat locks up completely without getting back to normal operations > > > > >>after a while. No exception or error is reported in the logs, it just > > > > >>stops. All connections are refused after the lockup. Tomcat has enough > > > > >>memory (>1GB) assigned with the JVM options. > > > > >> > > > > >>If I use maxThreads="500", the server runs fine, but 500 is not enough > > > > >>for my project ;-( > > > > >> > > > > >>I tried to increase the stack size, but this doesn't help.. > > > > >> > > > > >>The connector is defined as: > > > > >> > > > > >> > > > >> minSpareThreads="50" maxSpareThreads="150" enableLookups="false" > > > > >> redirectPort="8443" acceptCount="200" connectionTimeout="2" > > > > >> disableUploadTimeout="true"/> > > > > >> > > > > >>If I change 500 to 700 or more, Tomcat locks up. Is this a Windows > > > > >>problem?? > > > > >> > > > > >>Hardware is not an issue, I think.. (dual Xeon 3,4GHz, 4GB).. > > > > >> > > > > >>Thanks, > > > > >>Michael > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > David Delbecq > > > Royal Meteorological Institute of Belgium > > > > > > - > > > Is there life after /sbin/halt -p? > > > > > > - > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > -- > David Delbecq > Royal Meteorological Institute of Belgium > > - > Is there life after /sbin/halt -p? > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > -- Michael Kleinhenz tarent GmbH . Bahnhofstr. 13 . 53123 Bonn fon +49 (228) / 52 67 5-0 . fax +49 (228) / 52 67 5-25 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Max thread/session timeouts
> We're running an application on Tomcat that often hits the limit of > 150 threads. Can someone with experience changing this value give me > advice on it? How big can you make this number? How much extra memory > do I need if I say change it to, say, 500? Thanks. just a few facts on threads in Tomcat: . you'll need (Xss*max_threads)+Xmx+x memory in your box, where x is the memory needed by the basic JVM tasks besides the thread space (say, 100MB). . on ia32/Windows, you can only assign 1.6GB (Xmx-)memory to a single JVM. A value above it will break the 2GB barrier and will not run. . you need to find the balance between max_threads and Xmx. A lower Xmx gives you a higher max_threads and vice versa. . Tomcat on Windows doesn't scale well. Use multiple Tomcat instances on one box if you have a large box. In my setup, I run 4 Tomcats on a 2xXeon system with 4GB memory. Each Tomcat gets 1000 threads and 512MB. This is _much_ faster than a single Tomcat with all memory and 4000 threads. . Windows 2003 Server, at least the Standard Edition has major limitations in both TCP connections and memory. There are some workarounds using esoteric registry keys, but none of them works well. Windows is a crippled desktop os that wants to be on the server. If you want really high performance on large boxes, use Linux (Kernel 2.6!). You even get a better administration interface. Hate this inefficient administration-by-clicking ;-) -- Michael -- Dipl.-Technoinform Michael Kleinhenz tarent GmbH . Bahnhofstraße 13 . 53123 Bonn fon: +49 (228) / 52 67 5-0 fax: +49 (228) / 52 67 5-25 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Max thread/session timeouts
Peddireddy,
> your mail is very informative.
> we have a very similar set up (4 Xeon processors 3.5 GB Ram and
> WIn2003 Standard edition) and hardpart is that we cant change this
> setup and switch to Linux.
same problem over here. The computing centre of our client can't provide
Linux servers. I'm struggling with this problem for 3 weeks now.
> I heard about clustering on diferent boxes. Is the process of running
> multiple tomcats on a single machine will be same??
My experience was that with a single Tomcat on a large machine, you'll
never use that box fully. Besides that you can't get the Sun JVM to
accept a large amount of threads while giving them enough memory, Tomcat
seems to have performance problems with 3000+ threads busy at the same
time. The BEA JVM (JRockit) performs better in terms of memory and
threads but is slower than the Sun JVM. If you have rather complex
database operations performed by your webapp, this will get you into
trouble, because if there is a single bottleneck in the whole system,
the requests will pile up in the server, it hits its thread limit, and
ceases operation.
So, I have made good experiences with 4 Tomcats on a 2xXeon with 4GB.
The server's load is somewhere between 70% and 100% while running with
high load and can handle more concurrent requests than a single Tomcat.
> You talked about some "workarounds using esoteric registry keys", can
> you eloborate on this or tell me where I can get details about it and
> the details about limitations placed by windows2003 on a single
> process in terms of number of threads, memory etc.
We use 2003 Standard Edition and get the following limitations:
. Windows only provides 2GB memory to all applications running,
regardless of how much memory you have. All other memory is used for
Windows itself. With a registry key, you can extend this to 3GB. I
don't have the key handy, but it can be found in MSDN.
. The somewhat larger problem is the crappy TCP stack on Windows. A
single Tomcat easily handles 1000 concurrent connections, but if I start
4 Tomcats, I get a WSAENOBUFS error ("An operation on a socket could not
be performed because the system lacked sufficient buffer space or
because a queue was full") at about 3300 concurrent connections.
Because each Tomcat runs with his own instance of the JVM, I suppose
the problem is Windows, the network or anything else that is used by all
Tomcats at the same time. Even worse, after such a crash, there is a
10% chance, the Tomcats not resuming normal operation but crashes
completely.
. Windows has a default limit of 5000 ephemeral ports that can be used
fo outgoing connections. This can be changed with the MaxUserPorts
registry key. Also, the TcpTimedWaitDelay key is relevant for network
performance. Both are described here:
http://msdn.microsoft.com/library/en-us/randz/protocol/tcp_time-wait_delay.asp
I'm now really irritated by this problem. I need a capacity of about
15000 concurrent connections for my project. I also have 3 2xXeon
servers here, but windows (or anything else) doesn't allow more than
~8000 connections until it dies.
-- Michael
--
Dipl.-Technoinform. Michael Kleinhenz
tarent GmbH . Bahnhofstr. 13 . 53123 Bonn
fon +49 (228) / 52 67 5-0 . fax +49 (228) / 52 67 5-25
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Max thread/session timeouts
Peddireddy,
how about your project? I have finally found a solution here ;-) If the
database handles requests faster, the load on my Tomcat(s) are lower,
because of the lower count of active threads. This does not resolve the
problem induced by the apparent connection limit on Windows 2003
Standard Edition, especially in peak situations with ~3000 active
Threads. I've got the computing centre to install Red Hat Enterprise 4
on one of the servers and voila, the errors are gone!
Admittedly, it seems that Linux is a bit slower when getting into a high
thread count, it's sufficiently fast for my project. Also, it seems
faster on low thread counts (<500). A simple servlet building a XML tree
(without any database access or other 3rd party access) runs about 30%
faster on Linux than on Windows on the same machine.
I think the slower response of Linux with high thread counts is induced
by a slower thread handling. An engineer from Red Hat will try to
optimize that tomorrow..
-- Michael
On Fri, Jul 22, 2005 at 11:23:24AM +0200, Michael Kleinhenz wrote:
> Peddireddy,
>
> > your mail is very informative.
> > we have a very similar set up (4 Xeon processors 3.5 GB Ram and
> > WIn2003 Standard edition) and hardpart is that we cant change this
> > setup and switch to Linux.
>
> same problem over here. The computing centre of our client can't provide
> Linux servers. I'm struggling with this problem for 3 weeks now.
>
> > I heard about clustering on diferent boxes. Is the process of running
> > multiple tomcats on a single machine will be same??
>
> My experience was that with a single Tomcat on a large machine, you'll
> never use that box fully. Besides that you can't get the Sun JVM to
> accept a large amount of threads while giving them enough memory, Tomcat
> seems to have performance problems with 3000+ threads busy at the same
> time. The BEA JVM (JRockit) performs better in terms of memory and
> threads but is slower than the Sun JVM. If you have rather complex
> database operations performed by your webapp, this will get you into
> trouble, because if there is a single bottleneck in the whole system,
> the requests will pile up in the server, it hits its thread limit, and
> ceases operation.
>
> So, I have made good experiences with 4 Tomcats on a 2xXeon with 4GB.
> The server's load is somewhere between 70% and 100% while running with
> high load and can handle more concurrent requests than a single Tomcat.
>
> > You talked about some "workarounds using esoteric registry keys", can
> > you eloborate on this or tell me where I can get details about it and
> > the details about limitations placed by windows2003 on a single
> > process in terms of number of threads, memory etc.
>
> We use 2003 Standard Edition and get the following limitations:
>
> . Windows only provides 2GB memory to all applications running,
>regardless of how much memory you have. All other memory is used for
>Windows itself. With a registry key, you can extend this to 3GB. I
>don't have the key handy, but it can be found in MSDN.
>
> . The somewhat larger problem is the crappy TCP stack on Windows. A
>single Tomcat easily handles 1000 concurrent connections, but if I start
>4 Tomcats, I get a WSAENOBUFS error ("An operation on a socket could not
>be performed because the system lacked sufficient buffer space or
>because a queue was full") at about 3300 concurrent connections.
>Because each Tomcat runs with his own instance of the JVM, I suppose
>the problem is Windows, the network or anything else that is used by all
>Tomcats at the same time. Even worse, after such a crash, there is a
>10% chance, the Tomcats not resuming normal operation but crashes
>completely.
>
> . Windows has a default limit of 5000 ephemeral ports that can be used
>fo outgoing connections. This can be changed with the MaxUserPorts
>registry key. Also, the TcpTimedWaitDelay key is relevant for network
>performance. Both are described here:
>
>
> http://msdn.microsoft.com/library/en-us/randz/protocol/tcp_time-wait_delay.asp
>
> I'm now really irritated by this problem. I need a capacity of about
> 15000 concurrent connections for my project. I also have 3 2xXeon
> servers here, but windows (or anything else) doesn't allow more than
> ~8000 connections until it dies.
>
> -- Michael
>
> --
> Dipl.-Technoinform. Michael Kleinhenz
> tarent GmbH . Bahnhofstr. 13 . 53123 Bonn
> fon +49 (228) / 52 67 5-0 . fax +49 (228) / 52 67 5-25
--
Michael Kleinhenz
tarent GmbH . Bahnhofstr. 13 . 53123 Bonn
fon +49 (228) / 52 67 5-0 . fax +49 (228) / 52 67 5-25
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Does Tomcat run better on Linux or Windows?
> Is Tomcat more stable on Linux or Windows 2003? What are the pros/cons > of using it on each platform? If you're planning for a high-performance high-load system, don't use Windows 2003 Standard Edition. It has serious limitations in the TCP/IP stack. I wasn't able to open more than ~3500 concurrent network connections to a database cluster. The common error was "no buffer space available". Linux on the same machine works fine. Would be interesting if Win 2003 EE or DCE has the same limitation. The real pain was not the limitation itself, but that is _not_ documented. The error message is listed as a common error that occurs in many different situations. Great. -- Michael -- Michael Kleinhenz tarent GmbH . Bahnhofstr. 13 . 53123 Bonn fon +49 (228) / 52 67 5-0 . fax +49 (228) / 52 67 5-25 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
