Re: Odd NIO connector behavior
On 23/12/2011 23:39, Matthew Tyson wrote: Hello, We have been having quite a few problems with using long-polling connections in Tomcat, via the NIO connector. Upgrading to Tomcat 7.0.23 definitely improved things, but we are still seeing major issues. Glad to hear things are getting better. No so glad to hear you are still having problems. The problems only crop up after a couple minutes under some load (modest load, around 2-3 connections per second). That's pretty low load. One very clear problem I am looking at right now is that the service method on a servlet is continually being called, although there is no traffic coming into tomcat from that remote IP (we verified this at the ethernet device). Hmm. Very strange that the service method is being called. There needs to be a complete and valid set of HTTP headers for that to happen and the request/response gets recycled afterwards so the data shouldn't get processed twice. The logging statement at the beginning of the service method is being executed every so often, like so: logger.info(REQUEST: + request.getRemoteAddr() ++ request.getMethod() ++ request.getQueryString()); INFO 2011-12-23 15:30:50,860 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:02,484 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 GET message=%5B%7B%22channel%22%3A%22%2Fmeta%2Fconnect%22%2C%22connectionType%22%3A%22callback-polling%22%2C%22advice%22%3A%7B%22timeout%22%3A0%7D%2C%22id%22%3A%22354%22%2C%22clientId%22%3A%222b611tiekwk6p2mfh5bye3bm6y7l%22%7D%5Djsonp=dojo.io.script.jsonp_dojoIoScript135._jsonpCallback INFO 2011-12-23 15:31:28,512 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:36,571 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null Odd. So there are at least two different requests being processed here. But again, there is no traffic from that IP. I'm not sure if this is some sort of loop, a very long delay, or other connections being mixed up. I'm not aware of any connection mix up issues that might explain this. Probably the last, since I don't see any loop pattern, and it has continued without any traffic for almost a half an hour now. Thoughts? We need more information :) If you can create a simple web application that reproduces this I'd be happy to take a look. I suspect that is non-trivial so I'll suggest a couple of other options. 1. The simple thing is to add a stack trace to that log message so we can see exactly what code path is triggered this. 2. How comfortable are you patching Tomcat and building it from source? I have some debug logging sat in a git branch that I use for debugging similar issues that will generate a lot of logging but show exactly what is happening. I can either provide you with the patch or an updated JAR (or JARs) that you can drop into a 7.0.23 instance. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Creating CSR for Purchasing SSL Certificate from VeriSign
On 23/12/2011 22:42, Mark Eggers wrote: - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Cc: Sent: Friday, December 23, 2011 2:08 PM Subject: Re: Creating CSR for Purchasing SSL Certificate from VeriSign -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Conway, On 12/23/11 4:13 PM, Conway Liu wrote: Also, if we later add another Tomcat site (with a different domain name) on the same Windows 2008 R2 server, do we have to generate another CSR to purchase another SSL certificate? Apache httpd and non-APR Tomcat use different certificate storage formats: httpd uses a fairly simple PEM file format where you can have one or more certs concatenated together in a single file (or separately). When using APR with Tomcat, it uses the same format as httpd. If you aren't using APR, then the underlying Java environment is providing crypto services through a KeyStore which is stored in a completely different format. The certificates themselves are a standardized format, and you can export from one format and import to the other format whenever you want. You just need to figure out the right incantations of keytool and openssl to make that happen. So, have no fear of making a decision now that cannot be undone. - -chris One of the things you can do is get a SAN (subject alternate name) certificate with lots of different host names. If you are running everything from one IP address / port (named virtual hosts), this is pretty nice. Apache HTTPD complains a little about using named virtual hosts for configuration, but it works in practice. It seems that the configuration parsing engine doesn't like this on start up, but it works once running. There's actually a patch in queue to solve that problem. I suspect that Tomcat using the APR libraries will behave the same. It looks like Java since 1.5 has supported SAN certificates as well, but I've not tried this. You could experiment a little before purchasing a SAN certificate. I have some links on how to create your own root authority and signed SAN certificates if people are interested. A note: SSL protects the transport. In the case of HTTP, this means that the SSL handshake occurs before the receiving party can see which host header is in use, because the TCP connection process doesn't include the DNS name, that is resolved away to IP + port beforehand. In the case of virtual hosts, this used to mean that an SSL protected virtual host had to run on its own IP address. SAN cert support, wildcard certs now provide an alternative, but both are dependent on their support in browsers (which is improving). p -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Tomcat HTTPS Connector Stops working after an hour
On 23/12/2011 10:52, Saravanan L wrote: Please find the dump below. One more interesting thing I found out, was when I remove acceptCount=1 My tomcat 7 works fine. Note the 'acceptCount' definition: http://tomcat.apache.org/tomcat-7.0-doc/config/http.html This attribute specifies the size of the queue of requests that are waiting to be processed, rather than the number of requests that can be processed. 1 is rather large and may be impacted by the OS own ability to handle that number of inbound connection requests. The default is 100. So your experiment may have just deferred the occurrence of the same error. p But I had the same property in tomcat 6 and it worked fine on that. I believe this property is causing the problem . Also found out that when it reaches 900 connections with my existing connector [See below] the tomcat https [Http works fine] stops responding and the browser says 'Connection reset by peer'. But when acceptCount is removed it works fine. ---DUMP- Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode): http-apr-80-exec-10 daemon prio=10 tid=0x7fc5a625d800 nid=0x6a2f waiting on condition [0x53e8e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd489cf0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) http-apr-80-exec-9 daemon prio=10 tid=0x408ca000 nid=0x6a2c waiting on condition [0x54b9b000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd489cf0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) http-apr-80-exec-8 daemon prio=10 tid=0x7fc5ac9b9800 nid=0x6a2b waiting on condition [0x53c8c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd489cf0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) http-apr-80-exec-7 daemon prio=10 tid=0x7fc5a63cf800 nid=0x6a1b waiting on condition [0x552a2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd489cf0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) http-apr-80-exec-6 daemon prio=10 tid=0x7fc5a63ce800 nid=0x6a1a waiting on condition [0x41984000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd489cf0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) http-apr-80-exec-5 daemon prio=10 tid=0x4039c800 nid=0x6a17 waiting on condition [0x54696000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd489cf0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) http-apr-80-exec-4 daemon prio=10 tid=0x405c6800 nid=0x69e8 waiting on condition [0x41883000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd489cf0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) http-apr-443-exec-282 daemon prio=10 tid=0x7fc5b1063000 nid=0x69c1 waiting on condition [0x54f9f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x7fc5bd344e98 (a
Re: Tomcat HTTPS Connector Stops working after an hour
On 23/12/2011 14:47, Christopher Schultz wrote: Saravanan, On 12/23/11 7:03 AM, Saravanan L wrote: Please find the latest thread dump attached with mail. Tomcat 7 still crashes without acceptCount. I am wrong on this assumption. The response code is 504. It takes about 2 hours for this occur. The last status in http-apr-443 (In Server status ) is : Max threads: 5000 Seems high, as Chris says. Current thread count: 249 Can you post your full server.xml please, inline and with all XML comments/usernames/passwords removed. Current thread busy: 2 Low. Keeped alive sockets count: 40 Hmm? Where does that come from? Max processing time: 300477 ms Processing time: 8788.765 s Seems low for 2 hours of operation. Where does that number come from? Request count: 9223 Error count: 783 That's about 10%. What are those errors, exactly? Bytes received: 0.22 MB Bytes sent: 5.43 MB Both of those seem low for 2 hours of operation. What is your application/server doing? p What happens if you configure your HTTPS connector with fewer max threads, say, 50? Does the connector lock-up more quickly? What does your load profile look like? Is this observable in a testing environment or only in production? If you could set up a simple jmeter test against a test webapp, that would be ideal for trying to debug this issue. If you switch to NIO or BIO connector, does Tomcat become more stable, or do you experience the same phenomenon? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Hosted environment
On 23/12/2011 18:46, Mark Eggers wrote: - Original Message - From: Pid * p...@pidster.com To: Tomcat Users List users@tomcat.apache.org Cc: Sent: Friday, December 23, 2011 9:42 AM Subject: Re: Hosted environment On 22 Dec 2011, at 18:29, Mark Eggers its_toas...@yahoo.com wrote: - Original Message - From: r.sriram sri...@rstech.net To: Tomcat Users List users@tomcat.apache.org Cc: Sent: Wednesday, December 21, 2011 5:27 PM Subject: Re: Hosted environment snip Hmm, it certainly does work. I currently have about 100 virtual hosts, 8 CATALINA_BASE implementations, and SSL with a SAN Cert. I just finished setting up another instance (took about 15 minutes) to do some application testing. The problem is that the OP is using Plesk, which has it's own ideas about config file layout. In my experience* the related config files are includes in a config dir under the same part of the virtual host tree as the httpdocs dir. Plesk regenerates some of these every time you update any of the configured virtual hosts. So we have to get the OP to edit files that won't be overwritten. p * not particularly recent, but sufficiently painful to be memorable. snip I haven't had (sounds fortunately) any experience with Plesk. So the OP will have to track down where these include files are and add the appropriate mod_jk directives. It also sounds like he'll have to put the general mod_jk directives directly in the main httpd.conf file. I guess you could put workers.properties anywhere, as long as it's located properly in the main httpd.conf file. Ah ha - a little bit of searching does wonders: http://www.vlad-yatsenko.eu/2011/04/plesk-apache-mod_jk-tomcat-good-relationship/ With a little bit of correction (and adding the Tomcat virtual hosts), the above link should work. Yeah, I think vhost.conf is a file that gets regenerated if you use the Plesk GUI to change vhost settings. p -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Running not multithreaded application
On 23/12/2011 02:05, hernan wrote: On 12/14/11 5:45 PM, hernan wrote: I'm using Tomcat 7.0 for developing a new application. A key component in the application have to run an external not multithreaded application. Since I'm not an experienced user in Tomcat, I wonder which implementation alternatives do you recommend for running my external application. I'm trying to avoid launch a new os process from java since I need the result of that process, so in this way I've to deal communicating two processes (may be using serializable objects using sockets). So, is this other program a Java program? Since you said serializable you probably are talking about running another Java program. What makes this external program non-multithreaded? Is it actually not threadsafe? If you can't run it in the same process as Tomcat, then you'll have to either use Runtime.exec or some type of wrapper around it -- or have that process running completely separately (as a daemon) and connect to it via sockets. Anything else requires native code, which you definitely don't want. My external application is a java application that uses java wrapper (using JNI) for GLPK (written in ANSI C) (http://www.gnu.org/s/glpk/), and GLPK is not threadsafe. Yes, I will consider those alternatives, executing a new process when I receive the request (Runtime.exec) and a separated deamon receiving requests launching those processes. Can you not import the application Jars and just interact with them directly then? Why do you have to interact with it via system process? If I do that, due to glpk is not threadsafe, a call to a method can crash tomcat. I don't know what glpk does, can you explain? glpk is a gnu library to solving some kind of mathematical problems http://www.gnu.org/software/glpk/ You can use the library in c, and there exist wrappers for java, python and other programming languages I think you may be over thinking the thread safety issue. If multiple threads can't access the instances of the thing, thread safety is not an issue. If the entire code block was instantiated inside a Servlet doGet method with no external exposure or external scope, there is no problem. Do you know why it is not thread safe? I know about some problems related to process memory allocation/deallocation because I've experienced them. The documentation says that is not thread safe. Yes, but thread-safety is a very broad topic. If the code does something weird like use a mmap'd file with a hard coded path, then starting up separate processes won't help either. Why do you think that I'm over thinking thread safety issue? In Java for example, code that is not inherently thread-safe can be used safely in a multithreaded environment. The SimpleDateFormat class is a good example of this. Because it may be possible to use the code safely, if you take some precautions. Understanding exactly why it is deemed not thread safe would allow you do that. Warning that the code is not thread-safe usually just means that the author is telling you not to share use of that code *between* threads. If the code is utilised entirely within the scope of a single thread there is (usually) no issue. YMMV. p -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: MBean InitialContext() Tomcat query- pls help
On 23/12/2011 04:57, S B wrote: On Thu, Dec 22, 2011 at 8:49 PM, Pid p...@pidster.com wrote: On 22/12/2011 10:34, Konstantin Kolinko wrote: 2011/12/20 S B sbl...@gmail.com: Hi, I created and deployed an MBean in my tomcat. It uses datasource to connect to DB. My questions is: When I create InitialContext() inside MBean's constructor and pass the envContext to DBManager class to lookup datasource it works fine. However when I create InitialContext() in DBManager class, it fails. IIRC what InitialContext() sees as its environment highly depends on what classloader is active. That is TCCL = Thread.getContextClassLoader(). So while it is run from within web application your TCCL = your webapp's classloader. When it is invoked from jconsole it might be a separate Thread, not related to your web application. Like he ^^^ said. :) p -- [key:62590808] Hi Pid/Konstantin, yes you are right. I just verified it. When it is invoked from jconsole, the classloader is: sun.misc.Launcher$AppClassLoader@61e63e3d and when it is run within a web application the clasloader is WebAppClassloader. So, how can I force it to use webapp classloader during new InitialContext() while invoking from jconsole One possible way is to reuse the initialContext created during server startup. (pass it as param during invocation from jconsole). Is there a smarter way ?? I am a bit concerned about how you are registering the bean inside the bean's own constructor. I am not clear on how you are subsequently unregistering that bean, before the reloading operation - and I am not at all clear what the purpose of reloading is anyway. Is there a reason that the MBean itself needs to be recreated? Bear in mind that you are attempting to re-initialise the MBean, while you are still using it. This does not seem like a good idea to me. I would suggest that you either have a separate Manager MBean that does the reloading, or just have the reload method call the actual code on the DB, without re-initialising the bean. p -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Creating CSR for Purchasing SSL Certificate from VeriSign
- Original Message - From: Pid p...@pidster.com To: Tomcat Users List users@tomcat.apache.org Cc: Sent: Saturday, December 24, 2011 1:42 AM Subject: Re: Creating CSR for Purchasing SSL Certificate from VeriSign On 23/12/2011 22:42, Mark Eggers wrote: - Original Message - From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Cc: Sent: Friday, December 23, 2011 2:08 PM Subject: Re: Creating CSR for Purchasing SSL Certificate from VeriSign -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Conway, On 12/23/11 4:13 PM, Conway Liu wrote: Also, if we later add another Tomcat site (with a different domain name) on the same Windows 2008 R2 server, do we have to generate another CSR to purchase another SSL certificate? Apache httpd and non-APR Tomcat use different certificate storage formats: httpd uses a fairly simple PEM file format where you can have one or more certs concatenated together in a single file (or separately). When using APR with Tomcat, it uses the same format as httpd. If you aren't using APR, then the underlying Java environment is providing crypto services through a KeyStore which is stored in a completely different format. The certificates themselves are a standardized format, and you can export from one format and import to the other format whenever you want. You just need to figure out the right incantations of keytool and openssl to make that happen. So, have no fear of making a decision now that cannot be undone. - -chris One of the things you can do is get a SAN (subject alternate name) certificate with lots of different host names. If you are running everything from one IP address / port (named virtual hosts), this is pretty nice. Apache HTTPD complains a little about using named virtual hosts for configuration, but it works in practice. It seems that the configuration parsing engine doesn't like this on start up, but it works once running. There's actually a patch in queue to solve that problem. I suspect that Tomcat using the APR libraries will behave the same. It looks like Java since 1.5 has supported SAN certificates as well, but I've not tried this. You could experiment a little before purchasing a SAN certificate. I have some links on how to create your own root authority and signed SAN certificates if people are interested. A note: SSL protects the transport. In the case of HTTP, this means that the SSL handshake occurs before the receiving party can see which host header is in use, because the TCP connection process doesn't include the DNS name, that is resolved away to IP + port beforehand. In the case of virtual hosts, this used to mean that an SSL protected virtual host had to run on its own IP address. SAN cert support, wildcard certs now provide an alternative, but both are dependent on their support in browsers (which is improving). p SSL SAN cert support in recent browsers is pretty good. For example: http://www.digicert.com/subject-alternative-name-compatibility.htm On my test setup (Apache HTTPD 2.2.21 -- mod_jk 1.2.32 -- Tomcat 6.0.33) and the latest Firefox or Google Chrome, this works pretty well. SSL SNI support on the other hand is not well supported. In particular IE and Safari on Windows/XP do not support SNI. Most older mobile browsers don't support SNI as well. I ran into this issue recently while setting up Apache HTTPD-- AJP --Tomcat with multiple virtual hosts and SSL. . . . . just my two cents. /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Odd NIO connector behavior
On Sat, Dec 24, 2011 at 1:06 AM, Mark Thomas ma...@apache.org wrote: On 23/12/2011 23:39, Matthew Tyson wrote: Hello, We have been having quite a few problems with using long-polling connections in Tomcat, via the NIO connector. Upgrading to Tomcat 7.0.23 definitely improved things, but we are still seeing major issues. Glad to hear things are getting better. No so glad to hear you are still having problems. The problems only crop up after a couple minutes under some load (modest load, around 2-3 connections per second). That's pretty low load. It is. We have just a small portion of connections routed to this server. One very clear problem I am looking at right now is that the service method on a servlet is continually being called, although there is no traffic coming into tomcat from that remote IP (we verified this at the ethernet device). Hmm. Very strange that the service method is being called. There needs to be a complete and valid set of HTTP headers for that to happen and the request/response gets recycled afterwards so the data shouldn't get processed twice. It is very strange. The logging statement at the beginning of the service method is being executed every so often, like so: logger.info(REQUEST: + request.getRemoteAddr() ++ request.getMethod() ++ request.getQueryString()); INFO 2011-12-23 15:30:50,860 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:02,484 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 GET message=%5B%7B%22channel%22%3A%22%2Fmeta%2Fconnect%22%2C%22connectionType%22%3A%22callback-polling%22%2C%22advice%22%3A%7B%22timeout%22%3A0%7D%2C%22id%22%3A%22354%22%2C%22clientId%22%3A%222b611tiekwk6p2mfh5bye3bm6y7l%22%7D%5Djsonp=dojo.io.script.jsonp_dojoIoScript135._jsonpCallback INFO 2011-12-23 15:31:28,512 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:36,571 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null Odd. So there are at least two different requests being processed here. But again, there is no traffic from that IP. I'm not sure if this is some sort of loop, a very long delay, or other connections being mixed up. I'm not aware of any connection mix up issues that might explain this. Probably the last, since I don't see any loop pattern, and it has continued without any traffic for almost a half an hour now. Thoughts? We need more information :) If you can create a simple web application that reproduces this I'd be happy to take a look. I suspect that is non-trivial so I'll suggest a couple of other options. 1. The simple thing is to add a stack trace to that log message so we can see exactly what code path is triggered this. Here is a couple stack traces from this when the problem is occurring: INFO 2011-12-24 10:25:35,578 COMET REQUEST: 75.149.42.46 POST null | TRACE: java.lang.Throwable at org.cometd.server.CometdServlet.service(CometdServlet.java:149) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.company.util.filter.MonitoringFilter.doFilter(MonitoringFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at com.company.util.tomcat.SecureProxyValve.invoke(SecureProxyValve.java:57) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1571) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at
Re: Odd NIO connector behavior
Am 24.12.2011 19:33, schrieb Matthew Tyson: On Sat, Dec 24, 2011 at 1:06 AM, Mark Thomasma...@apache.org wrote: On 23/12/2011 23:39, Matthew Tyson wrote: Hello, We have been having quite a few problems with using long-polling connections in Tomcat, via the NIO connector. Upgrading to Tomcat 7.0.23 definitely improved things, but we are still seeing major issues. Glad to hear things are getting better. No so glad to hear you are still having problems. The problems only crop up after a couple minutes under some load (modest load, around 2-3 connections per second). That's pretty low load. It is. We have just a small portion of connections routed to this server. One very clear problem I am looking at right now is that the service method on a servlet is continually being called, although there is no traffic coming into tomcat from that remote IP (we verified this at the ethernet device). Hmm. Very strange that the service method is being called. There needs to be a complete and valid set of HTTP headers for that to happen and the request/response gets recycled afterwards so the data shouldn't get processed twice. It is very strange. The logging statement at the beginning of the service method is being executed every so often, like so: logger.info(REQUEST: + request.getRemoteAddr() ++ request.getMethod() ++ request.getQueryString()); INFO 2011-12-23 15:30:50,860 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:02,484 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 GET message=%5B%7B%22channel%22%3A%22%2Fmeta%2Fconnect%22%2C%22connectionType%22%3A%22callback-polling%22%2C%22advice%22%3A%7B%22timeout%22%3A0%7D%2C%22id%22%3A%22354%22%2C%22clientId%22%3A%222b611tiekwk6p2mfh5bye3bm6y7l%22%7D%5Djsonp=dojo.io.script.jsonp_dojoIoScript135._jsonpCallback INFO 2011-12-23 15:31:28,512 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:36,571 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null Odd. So there are at least two different requests being processed here. But again, there is no traffic from that IP. I'm not sure if this is some sort of loop, a very long delay, or other connections being mixed up. I'm not aware of any connection mix up issues that might explain this. Probably the last, since I don't see any loop pattern, and it has continued without any traffic for almost a half an hour now. Thoughts? We need more information :) If you can create a simple web application that reproduces this I'd be happy to take a look. I suspect that is non-trivial so I'll suggest a couple of other options. 1. The simple thing is to add a stack trace to that log message so we can see exactly what code path is triggered this. Here is a couple stack traces from this when the problem is occurring: INFO 2011-12-24 10:25:35,578 COMET REQUEST: 75.149.42.46 POST null | TRACE: java.lang.Throwable at org.cometd.server.CometdServlet.service(CometdServlet.java:149) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.company.util.filter.MonitoringFilter.doFilter(MonitoringFilter.java:47) I guess com.company is something your company developed. Monitoring sounds optional. Can you reproduce your error without it? at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at com.company.util.tomcat.SecureProxyValve.invoke(SecureProxyValve.java:57) Again com.company: What is this valve doing? Can you reproduce the problem without it? at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539) at
Re: Odd NIO connector behavior
On Sat, Dec 24, 2011 at 10:33 AM, Matthew Tyson matthewcarlty...@gmail.comwrote: On Sat, Dec 24, 2011 at 1:06 AM, Mark Thomas ma...@apache.org wrote: On 23/12/2011 23:39, Matthew Tyson wrote: Hello, We have been having quite a few problems with using long-polling connections in Tomcat, via the NIO connector. Upgrading to Tomcat 7.0.23 definitely improved things, but we are still seeing major issues. Glad to hear things are getting better. No so glad to hear you are still having problems. The problems only crop up after a couple minutes under some load (modest load, around 2-3 connections per second). That's pretty low load. It is. We have just a small portion of connections routed to this server. One very clear problem I am looking at right now is that the service method on a servlet is continually being called, although there is no traffic coming into tomcat from that remote IP (we verified this at the ethernet device). Hmm. Very strange that the service method is being called. There needs to be a complete and valid set of HTTP headers for that to happen and the request/response gets recycled afterwards so the data shouldn't get processed twice. It is very strange. The logging statement at the beginning of the service method is being executed every so often, like so: logger.info(REQUEST: + request.getRemoteAddr() ++ request.getMethod() ++ request.getQueryString()); INFO 2011-12-23 15:30:50,860 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:02,484 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 GET message=%5B%7B%22channel%22%3A%22%2Fmeta%2Fconnect%22%2C%22connectionType%22%3A%22callback-polling%22%2C%22advice%22%3A%7B%22timeout%22%3A0%7D%2C%22id%22%3A%22354%22%2C%22clientId%22%3A%222b611tiekwk6p2mfh5bye3bm6y7l%22%7D%5Djsonp=dojo.io.script.jsonp_dojoIoScript135._jsonpCallback INFO 2011-12-23 15:31:28,512 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null INFO 2011-12-23 15:31:36,571 org.cometd.server.CometdServlet REQUEST: 75.149.42.46 POST null Odd. So there are at least two different requests being processed here. But again, there is no traffic from that IP. I'm not sure if this is some sort of loop, a very long delay, or other connections being mixed up. I'm not aware of any connection mix up issues that might explain this. Probably the last, since I don't see any loop pattern, and it has continued without any traffic for almost a half an hour now. Thoughts? We need more information :) If you can create a simple web application that reproduces this I'd be happy to take a look. I suspect that is non-trivial so I'll suggest a couple of other options. 1. The simple thing is to add a stack trace to that log message so we can see exactly what code path is triggered this. Here is a couple stack traces from this when the problem is occurring: INFO 2011-12-24 10:25:35,578 COMET REQUEST: 75.149.42.46 POST null | TRACE: java.lang.Throwable at org.cometd.server.CometdServlet.service(CometdServlet.java:149) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.company.util.filter.MonitoringFilter.doFilter(MonitoringFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at com.company.util.tomcat.SecureProxyValve.invoke(SecureProxyValve.java:57) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1571) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Reasons for error Connection reset
Hi, What can be the reasons for the error given below: ERROR [org.apache.tomcat.util.net.PoolTcpEndpoint] Remote Host 172.16.128.73/172.16.128.73 SocketException: Connection reset I am using Tomcat 5.0 bundled inside JBoss 3.2.5. This error appears when I use JDK 1.6.0_29 while it does not appear in logs when I use JDK 1.4.2_05. Best Regards!
