[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #13 from Brandon Chong lovew...@naver.com --- Hi, Francesco? I have a similar problem as you reported above. I'm using CentOS 6.3 And Tomcat 6.0.35. If you figured out this problem already, could you let me know how to? If you give me some advise I'll be very pleased!!! Please feel free to email me at 'lovew...@naver.com'. Thank you in advance. :) -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 Mark Thomas ma...@apache.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |INVALID --- Comment #12 from Mark Thomas ma...@apache.org --- The thread dump shows that you have a large number of new connections that are not sending any data. Hence, since BIO and APR/native both block during the reading of the request line, the threads will remain blocked until the read times out of the client sends more data. You can see this by examining the stack trace. For example: http-apr-80-exec-96 - Thread t@137 java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.Socket.recvbb(Native Method) at org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:575) at org.apache.coyote.http11.InternalAprInputBuffer.parseRequestLine(InternalAprInputBuffer.java:134) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:927) process(AbstractHttp11Processor.java:927) This is when the processor triggers the read of the request line. The parameter it passes in (keptAlive) indicates is this is the first request received on the connection (keptAlive will be false) or a subsequent request (keptAlive will be true). It is important for this issue to know if keptAlive is true or false. parseRequestLine(InternalAprInputBuffer.java:134) This is the reading data from the socket for the request line and parsing it. Since we are at line 134 we know useAvailableData == false (else the if block starting on line 131 would have been executed and the method would have returned false to the caller. If useAvailableData == false then keptAlive == false. This tells us that this is the first request on the connection. fill(InternalAprInputBuffer.java:575) This is the InputBuffer attempting to read enough (actually any) data into the buffer so it can be parsed as a request line. It is blocking as there is no data to read. Putting everything together we know that: - this thread is processing a new connection from a client - the connection has been established but no data has been sent - Tomcat is waiting for data to arrive so it can parse the request line If you have lots of clients doing this then you have some broken clients. Switching to the NIO connector may help since it is non-blocking while reading the request line and request headers (all connectors block while reading request bodies). I have no idea what it is between your CentOS 5 and CentOS 6 installations that is triggering this issue. It may be related to defer accept but still, the clients are misbehaving. There is no Tomcat bug here. If you need further assistance, the Tomcat users mailing list is the place to seek help. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 francesco vigo...@gmail.com changed: What|Removed |Added Status|NEEDINFO|NEW --- Comment #9 from francesco vigo...@gmail.com --- i've used for bot centos 5 and 6, tomcat-native-1.1.24 and apr-1.4.6 *all tested environments are 64 bit but as i've tested yesterday this issue happens also with bio connectors the problem anyway exists since some months i have never got a working tomcat in centos 6 (and maybe in fedora/RH lastest releases, i don't have tested there but should be almost the same), and because centos is one of the top used server distro i think that this issue should be treated as critical i think that people which try to use tomcat in centos into a server with a quite high load, see the resource usage and think that tomcat is slow which is not! they don't fill a bug report because they don't think is a but (but instead it is because threads and connections doesn't close when tcp/ip transmission end) now i have reinstalled again centos 5.8 on that server (i'm stuck to centos 5 :( ) and the server is replying at about 4k connections / seconds with 8 cuncurrentthreadsBusy and 19 connectionCount (using APR) everything it's exactly the same as previously tested in centos 6 (which was unable to go over 200 connections/seconds without beign slow because threads count would always max out ) same sysctl.conf, firewall disabled , same tomcat version and same servlet (servlet is compiled locally then rsync'ed ) i think that someone should take a look at this issue, if i have a bit of time in next months i'll try to setup iptables to try to forward multiple packets to virtual machine with centos 5 and 6, simulating multiple ip connections (if u know a better way to do that let me know ) anyway if someone is running tomcat on centos 6 with a busy site and don't have this problem tell there and you can say then that it's me that i'm wrong, elseway i'm pretty sure that there is a very important issue there -- indipendent of servlet served -- that should need attention :) thank you, Francesco -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #10 from francesco vigo...@gmail.com --- this error happened with java version since 1.6.xx to 1.7.xx last attempt of yesterday has been done using the lastest 7u9-b05 , are months that i always try with newer software available to get tomcat working correctly on centos 6 but nothing at the end i have always to giveup and reinstall centos 5 and with same identical releases of software everything works correctly... the problem here i think is that if you have never tryed with tomcat centos 5 you think that this is just a problem of performance but i can't believe that in centos 6 threads connections remain opened this IS a bug, performance serving small static content (to multiple destinations, apache benchmark tests doesn't show this issue) are 100x worse in centos 6 and i mean 100x!! == 10 000% ! obviously if you have a huge servlet which take 1 second to serve just 1 request you may be not seeing this issue because the bottleneck is not so evident so... -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #11 from Pid bugzi...@pidster.com --- Creating a load test that sends 100s of requests per second isn't so hard. A test case that reproduces a particular outcome is the only way for the Tomcat community to analyse and fix an issue. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #1 from francesco vigo...@gmail.com --- Created attachment 29506 -- https://issues.apache.org/bugzilla/attachment.cgi?id=29506action=edit thread dump thread dump of tomcat when providing the issue described ( incredibly high currentThreadsBusy ) which causes unusable tomcat installation -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: [Bug 54036] currentThreadsBusy issue in centos 6
after further investigation seems that the problem is really at this point: at org.apache.tomcat.jni.Socket.recvbb(Native Method) at org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:575) each threads hang at this position till timeout and timeout on each connections i just wonder why this doesn't happen in centos 5.x and instead is bugged in centos 6.x probably some EOT/EOF signal didn't pass through APR/JNI ? 2012/10/22 bugzi...@apache.org https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #1 from francesco vigo...@gmail.com --- Created attachment 29506 -- https://issues.apache.org/bugzilla/attachment.cgi?id=29506action=edit thread dump thread dump of tomcat when providing the issue described ( incredibly high currentThreadsBusy ) which causes unusable tomcat installation -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #2 from francesco vigo...@gmail.com --- after further investigation seems that the problem is really at this point: at org.apache.tomcat.jni.Socket.recvbb(Native Method) at org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:575) each threads hang at this position till timeout and timeout on each connections i just wonder why this doesn't happen in centos 5.x and instead is bugged in centos 6.x probably some EOT/EOF signal didn't pass through APR/JNI ? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 Mark Thomas ma...@apache.org changed: What|Removed |Added Component|Connectors |Library Version|7.0.32 |unspecified Product|Tomcat 7|Tomcat Native --- Comment #3 from Mark Thomas ma...@apache.org --- Changing component. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #4 from francesco vigo...@gmail.com --- Hi Mark do you mean to change connector? i've tryed changing to BIO but the problem persist.. i have 220 threads blocked at this point: at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501) i have no other solution than change os for tomcat server, but anyway having tomcat UNCOMPATIBLE (because this bug make the system very susceptible to even small DDoS ) with the new CENTOS = 6 is a BIG ISSUE i think. , apache benchmark didn't show the issue in bot APR and BIO , ( keepalive is off (value=1) but the problem show up only when lot of external ip connect) maybe this is due on how the system handle tcp connections closing at lower levels ? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #5 from francesco vigo...@gmail.com --- i understood sorry u meant that you have changed location of the report (i'm not very used to send bug reports) no problem :) -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 Mark Thomas ma...@apache.org changed: What|Removed |Added Status|NEW |NEEDINFO Component|Library |Connectors Version|unspecified |7.0.32 Product|Tomcat Native |Tomcat 7 --- Comment #6 from Mark Thomas ma...@apache.org --- If this occurs across multiple connectors then this is not an APR/native bug. Moving back. Please provide a reproducible test case. Without one, this will get closed as INVALID. -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #7 from francesco vigo...@gmail.com --- you are the good tomcat programmer , how i can write a reproducible case if you cannot reproduce the volume of requests i'm getting (these are TCP connections not udp so i don't know how these can be simulated as coming from multiple different sources (maybe with something like playing with NAT in iptables) ) i am not the only one who has this issue, is someone use tomcat live with centos 6 and have more than 100 visit /seconds should have faced this issue too... the test case doesn't need to be built because happens always on centos = 6 (and probably on RHEL but i haven't tested before) , threads didn't close at the end of connections but at the socket timeout, so if you have high connection rate from different sources the thread pool become exhausted soon.. if you know how to reproduce these multiple connections i'm almost sure that also the HelloWorld servlet would works as test case in centos 6 -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
[Bug 54036] currentThreadsBusy issue in centos 6
https://issues.apache.org/bugzilla/show_bug.cgi?id=54036 --- Comment #8 from Pid bugzi...@pidster.com --- Which versions of APR and Java are installed on the Centos 5 and 6 systems? -- You are receiving this mail because: You are the assignee for the bug. - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
