RE: Packet misses in Tomcat
-Original Message- From: Stefan Mayr [mailto:ste...@mayr-stefan.de] Sent: 13 January 2014 23:26 To: users@tomcat.apache.org Subject: Re: Packet misses in Tomcat Am 13.01.2014 07:22, schrieb Divyaprakash Y: -Original Message- From: Stefan Mayr [mailto:ste...@mayr-stefan.de] Sent: 10 January 2014 23:26 To: users@tomcat.apache.org Subject: Re: Packet misses in Tomcat Hi Am 09.01.2014 14:21, schrieb Divyaprakash Y: -Original Message- From: Divyaprakash Y Sent: 08 January 2014 14:35 To: Tomcat Users List Subject: RE: Packet misses in Tomcat ... Strange that this is happening only to me. Looks like something similar was reported on the dev list when voting for Tomcat 7.0.50 .. Thanks. I tried same setup today with the BIO connector, everything worked flawlessly. Will there be any issue with the APR connector(earlier setup) or are there any extra configurations which I missed in my server.xml? This might be the issue seen in https://issues.apache.org/bugzilla/show_bug.cgi?id=55976 . Looks like Mark fixed it today for 7.0.51 (not released yet) - Stefan Thanks Stefan for the information. Should that(Fix on NIO Connector) fix the possible issue in APR Connector as well? I guess not. But maybe you shoudl 7.0.50 give a try. APR was updated in one of the not released versions after your lastest 7.0.42. Also a minimal testcase would help to check if this is reproducable on other peoples machine (like they did in the bugzilla ticket) - Stefan Thanks Stefan, will definitely try Tomcat 7.0.50 as well as try to build a minimal test case.. - DISCLAIMER: This electronic message and any attachments to this electronic message is intended for the exclusive use of the addressee(s) named herein and may contain legally privileged and confidential information. It is the property of Celstream Systems Private Limited. If you are not the intended recipient, you are hereby strictly notified not to copy, forward, distribute or use this message or any attachments thereto. If you have received this message in error, please delete it and all copies thereof, from your system and notify the sender at Celstream Systems or administrat...@celstream.com immediately. -
Re: Packet misses in Tomcat
Am 13.01.2014 07:22, schrieb Divyaprakash Y: -Original Message- From: Stefan Mayr [mailto:ste...@mayr-stefan.de] Sent: 10 January 2014 23:26 To: users@tomcat.apache.org Subject: Re: Packet misses in Tomcat Hi Am 09.01.2014 14:21, schrieb Divyaprakash Y: -Original Message- From: Divyaprakash Y Sent: 08 January 2014 14:35 To: Tomcat Users List Subject: RE: Packet misses in Tomcat ... Strange that this is happening only to me. Looks like something similar was reported on the dev list when voting for Tomcat 7.0.50 .. Thanks. I tried same setup today with the BIO connector, everything worked flawlessly. Will there be any issue with the APR connector(earlier setup) or are there any extra configurations which I missed in my server.xml? This might be the issue seen in https://issues.apache.org/bugzilla/show_bug.cgi?id=55976 . Looks like Mark fixed it today for 7.0.51 (not released yet) - Stefan Thanks Stefan for the information. Should that(Fix on NIO Connector) fix the possible issue in APR Connector as well? I guess not. But maybe you shoudl 7.0.50 give a try. APR was updated in one of the not released versions after your lastest 7.0.42. Also a minimal testcase would help to check if this is reproducable on other peoples machine (like they did in the bugzilla ticket) - Stefan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Packet misses in Tomcat
-Original Message- From: Stefan Mayr [mailto:ste...@mayr-stefan.de] Sent: 10 January 2014 23:26 To: users@tomcat.apache.org Subject: Re: Packet misses in Tomcat Hi Am 09.01.2014 14:21, schrieb Divyaprakash Y: -Original Message- From: Divyaprakash Y Sent: 08 January 2014 14:35 To: Tomcat Users List Subject: RE: Packet misses in Tomcat ... Strange that this is happening only to me. Looks like something similar was reported on the dev list when voting for Tomcat 7.0.50 .. Thanks. I tried same setup today with the BIO connector, everything worked flawlessly. Will there be any issue with the APR connector(earlier setup) or are there any extra configurations which I missed in my server.xml? This might be the issue seen in https://issues.apache.org/bugzilla/show_bug.cgi?id=55976 . Looks like Mark fixed it today for 7.0.51 (not released yet) - Stefan Thanks Stefan for the information. Should that(Fix on NIO Connector) fix the possible issue in APR Connector as well? - DISCLAIMER: This electronic message and any attachments to this electronic message is intended for the exclusive use of the addressee(s) named herein and may contain legally privileged and confidential information. It is the property of Celstream Systems Private Limited. If you are not the intended recipient, you are hereby strictly notified not to copy, forward, distribute or use this message or any attachments thereto. If you have received this message in error, please delete it and all copies thereof, from your system and notify the sender at Celstream Systems or administrat...@celstream.com immediately. -
RE: Packet misses in Tomcat
-Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: 10 January 2014 03:46 To: Tomcat Users List Subject: Re: Packet misses in Tomcat -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Divyaprakash, On 1/8/14, 4:06 AM, Divyaprakash Y wrote: -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 08 January 2014 02:52 To: Tomcat Users List Subject: Re: Packet misses in Tomcat Christopher, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 1/7/14, 5:09 AM, André Warnier wrote: I do not pretend to know your system, nor your application, nor that the following is a definite explanation. But on the base of the currently available data, I would say : - it is quite unlikely that Tomcat 7 is randomly dropping requests. If it was, then I would imagine that this list would be overflowing with cries for help. There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall seeing any significant number of issues mentioning dropped requests. - it also doesn't seem, from your wireshark-related observations, that the requests are being lost outside of Tomcat. - so I would say at this point, that the most likely place for requests to disappear is in your own application. It seems that Tomcat is not logging the request in its access log, so it's more likely that the request is either malformed to such an extent that Tomcat rejects the request altogether or that the request never reaches Tomcat. ... Hi. Of course I am going essentially by what the OP provided earlier as information, and he has not provided much details on the disappearing requests themselves, or on the channel through which these requests were reaching Tomcat. But one thing that he did mention, is that these requests are similar - and even in general the same - as other requests which do get processed normally. As per his own words : For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. If we take this at face value, then it should not be so that these requests are so malformed that Tomcat discards them without further ado. Also - but maybe I'm wrong there - I would expect, if Tomcat discards a request for being malformed - that something would appear in the Tomcat error log. But according to the OP it doesn't. Finally - and there is a bit of an assumption on my part here - I assume that when the OP says that he sees the request with Wireshark (prior to it disappearing in Tomcat), he was running Wireshark on the Tomcat host itself. That would make it unlikely that another external component is at play. All of the above led me to suspect that something in the application itself may be playing a role here. Of course, that all does not necessarily prove that some other component than Tomcat is not dropping some packets/requests. -- Clarifications for the missed information: (I have re-ordered some of your responses so they make more logical sense to me): - There are 2 machines (LAN) where client and servers are installed. Client is used to send the requests to Server. - Wireshark runs on the same machine where Tomcat runs - Tomcat will be accessed directly via HTTP - This issue occurs even if I use loopback address/localhost instead of Server IP Address in the request URL (Install both client and server in the same machine). - A war file with a servlet to POST the requests. What is the client? Is it possible that this is something simple like the TCP backlog queue filling-up and the OS is just discarding requests? Christopher, The client is nothing but a SOAP client which sends the SOAP packets at an interval of 30 seconds to the server. Looks like the situation of TCP backlog queue filling-up won't arise. FYI, the issue arises only when I use APR connector whereas default BIO connector works fine (both HTTP and HTTPS). I have not verified with NIO connector. Any pointer? - DISCLAIMER: This electronic message and any attachments to this electronic message is intended for the exclusive use of the addressee(s) named herein and may contain legally privileged and confidential information. It is the property of Celstream Systems Private Limited. If you are not the intended recipient, you are hereby strictly notified not to copy, forward, distribute or use this message or any attachments thereto. If you have received this message in error, please delete it and all copies thereof, from your system and notify the sender at Celstream Systems
Re: Packet misses in Tomcat
Hi Am 09.01.2014 14:21, schrieb Divyaprakash Y: -Original Message- From: Divyaprakash Y Sent: 08 January 2014 14:35 To: Tomcat Users List Subject: RE: Packet misses in Tomcat -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 08 January 2014 02:52 To: Tomcat Users List Subject: Re: Packet misses in Tomcat Christopher, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 1/7/14, 5:09 AM, André Warnier wrote: I do not pretend to know your system, nor your application, nor that the following is a definite explanation. But on the base of the currently available data, I would say : - it is quite unlikely that Tomcat 7 is randomly dropping requests. If it was, then I would imagine that this list would be overflowing with cries for help. There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall seeing any significant number of issues mentioning dropped requests. - it also doesn't seem, from your wireshark-related observations, that the requests are being lost outside of Tomcat. - so I would say at this point, that the most likely place for requests to disappear is in your own application. ... Strange that this is happening only to me. Looks like something similar was reported on the dev list when voting for Tomcat 7.0.50 .. Thanks. I tried same setup today with the BIO connector, everything worked flawlessly. Will there be any issue with the APR connector(earlier setup) or are there any extra configurations which I missed in my server.xml? This might be the issue seen in https://issues.apache.org/bugzilla/show_bug.cgi?id=55976 . Looks like Mark fixed it today for 7.0.51 (not released yet) - Stefan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Packet misses in Tomcat
-Original Message- From: Divyaprakash Y Sent: 08 January 2014 14:35 To: Tomcat Users List Subject: RE: Packet misses in Tomcat -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 08 January 2014 02:52 To: Tomcat Users List Subject: Re: Packet misses in Tomcat Christopher, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 1/7/14, 5:09 AM, André Warnier wrote: I do not pretend to know your system, nor your application, nor that the following is a definite explanation. But on the base of the currently available data, I would say : - it is quite unlikely that Tomcat 7 is randomly dropping requests. If it was, then I would imagine that this list would be overflowing with cries for help. There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall seeing any significant number of issues mentioning dropped requests. - it also doesn't seem, from your wireshark-related observations, that the requests are being lost outside of Tomcat. - so I would say at this point, that the most likely place for requests to disappear is in your own application. It seems that Tomcat is not logging the request in its access log, so it's more likely that the request is either malformed to such an extent that Tomcat rejects the request altogether or that the request never reaches Tomcat. ... Hi. Of course I am going essentially by what the OP provided earlier as information, and he has not provided much details on the disappearing requests themselves, or on the channel through which these requests were reaching Tomcat. But one thing that he did mention, is that these requests are similar - and even in general the same - as other requests which do get processed normally. As per his own words : For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. If we take this at face value, then it should not be so that these requests are so malformed that Tomcat discards them without further ado. Also - but maybe I'm wrong there - I would expect, if Tomcat discards a request for being malformed - that something would appear in the Tomcat error log. But according to the OP it doesn't. Finally - and there is a bit of an assumption on my part here - I assume that when the OP says that he sees the request with Wireshark (prior to it disappearing in Tomcat), he was running Wireshark on the Tomcat host itself. That would make it unlikely that another external component is at play. All of the above led me to suspect that something in the application itself may be playing a role here. Of course, that all does not necessarily prove that some other component than Tomcat is not dropping some packets/requests. -- Clarifications for the missed information: - Wireshark runs on the same machine where Tomcat runs - Channel through which these requests were reaching Tomcat - HTTP Chanel - Can you describe your deployment? Are you accessing Tomcat directly via HTTP? What networking components are between your test client(s) and Tomcat - A war file with a servlet to POST the requests. - There are 2 machines (LAN) where client and servers are installed. Client is used to send the requests to Server. - Disabled the firewall in the server machine - Tomcat will be accessed directly via HTTP - This issue occurs even if I use loopback address/localhost instead of Server IP Address in the request URL (Install both client and server in the same machine). - Tested with both apache-tomcat-7.0.27.exe and apache-tomcat-7.0.42-windows-x86.zip - Also, Windows event logs will not have any entries when the requests getting missed. Strange that this is happening only to me. Overnight run of my application with Jetty Server(1167 requests) did not miss any requests. So I am hoping that application is not contributing to the missing requests. I may be wrong here. Is there any way to get more logs on request processing in Tomcat? Thanks. I tried same setup today with the BIO connector, everything worked flawlessly. Will there be any issue with the APR connector(earlier setup) or are there any extra configurations which I missed in my server.xml? Thanks. - DISCLAIMER: This electronic message and any attachments to this electronic message is intended for the exclusive use of the addressee(s) named herein and may contain legally privileged and confidential information. It is the property of Celstream Systems Private Limited. If you are not the intended recipient, you are hereby strictly notified not to copy, forward
Re: Packet misses in Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Divyaprakash, On 1/8/14, 4:06 AM, Divyaprakash Y wrote: -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 08 January 2014 02:52 To: Tomcat Users List Subject: Re: Packet misses in Tomcat Christopher, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 1/7/14, 5:09 AM, André Warnier wrote: I do not pretend to know your system, nor your application, nor that the following is a definite explanation. But on the base of the currently available data, I would say : - it is quite unlikely that Tomcat 7 is randomly dropping requests. If it was, then I would imagine that this list would be overflowing with cries for help. There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall seeing any significant number of issues mentioning dropped requests. - it also doesn't seem, from your wireshark-related observations, that the requests are being lost outside of Tomcat. - so I would say at this point, that the most likely place for requests to disappear is in your own application. It seems that Tomcat is not logging the request in its access log, so it's more likely that the request is either malformed to such an extent that Tomcat rejects the request altogether or that the request never reaches Tomcat. ... Hi. Of course I am going essentially by what the OP provided earlier as information, and he has not provided much details on the disappearing requests themselves, or on the channel through which these requests were reaching Tomcat. But one thing that he did mention, is that these requests are similar - and even in general the same - as other requests which do get processed normally. As per his own words : For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. If we take this at face value, then it should not be so that these requests are so malformed that Tomcat discards them without further ado. Also - but maybe I'm wrong there - I would expect, if Tomcat discards a request for being malformed - that something would appear in the Tomcat error log. But according to the OP it doesn't. Finally - and there is a bit of an assumption on my part here - I assume that when the OP says that he sees the request with Wireshark (prior to it disappearing in Tomcat), he was running Wireshark on the Tomcat host itself. That would make it unlikely that another external component is at play. All of the above led me to suspect that something in the application itself may be playing a role here. Of course, that all does not necessarily prove that some other component than Tomcat is not dropping some packets/requests. -- Clarifications for the missed information: (I have re-ordered some of your responses so they make more logical sense to me): - There are 2 machines (LAN) where client and servers are installed. Client is used to send the requests to Server. - Wireshark runs on the same machine where Tomcat runs - Tomcat will be accessed directly via HTTP - This issue occurs even if I use loopback address/localhost instead of Server IP Address in the request URL (Install both client and server in the same machine). - A war file with a servlet to POST the requests. What is the client? Is it possible that this is something simple like the TCP backlog queue filling-up and the OS is just discarding requests? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJSzx+xAAoJEBzwKT+lPKRYwOgP/RnNfmH+/TUzWy5OUy7f2KUR 3D82ozXMQg4QNpV7jqUlacRb8k10WitXWG2wzC7jO0eyoDbiQlQQTZ0GhPwWKsOo 8Ah7wL3YeFEIy4QV8LcmCDspyzp1vJ64aCgr7z+mW+36g1b2ZvFiCeUunA6r+rBw m5vrRb2zVDprARd1hmdlM+JoApQ05XXbc7ZY79HXMn6D3zDu77s2Hvp5m5t6FDx/ mgqHbk4ILrJKf3/u+5ACVngIvUEHqzcc3tFMLlEzqUffZ5n04z/tK+WvHB19dkIS zp2YWJ30+vkzKSi+49cmM3aigdRFSNlTjba7ipdRhjW8vdsLHRnfiTNU41m1WsHD aXrJZqyzusOTGmOclR+Hgau82tn5lkwiQY2mibjCf3Lw5HDyT+sqNBZam/2iRRtp /oTOHMN+aDAHWTuTy0t8ySOD5CHnQOnGJtdSTqUHNN8dJ4sGn9xmZZ5dhXGdUYF3 5ag+NMmRsvwCiXKS9OJK7azqVwajKNjUeeOhHzbv8tF28jVJ6PMYIcVVMJsZEgcF 96BI3GX8emp8wD8pWMZFmlHpKkZ+svDxFs4Aoqjv1gZI8Qq6D8tyX6p4lZjwXqyH GcxVKYNHY4dZZHsMDHWq1CGxL93xH5FUU22M0nI0eJ6h4znU2uOFXScVEIVkiVih e2HI63On94Yk68jKzQk4 =c+yc -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Packet misses in Tomcat
-Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: 08 January 2014 02:52 To: Tomcat Users List Subject: Re: Packet misses in Tomcat Christopher, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 1/7/14, 5:09 AM, André Warnier wrote: I do not pretend to know your system, nor your application, nor that the following is a definite explanation. But on the base of the currently available data, I would say : - it is quite unlikely that Tomcat 7 is randomly dropping requests. If it was, then I would imagine that this list would be overflowing with cries for help. There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall seeing any significant number of issues mentioning dropped requests. - it also doesn't seem, from your wireshark-related observations, that the requests are being lost outside of Tomcat. - so I would say at this point, that the most likely place for requests to disappear is in your own application. It seems that Tomcat is not logging the request in its access log, so it's more likely that the request is either malformed to such an extent that Tomcat rejects the request altogether or that the request never reaches Tomcat. ... Hi. Of course I am going essentially by what the OP provided earlier as information, and he has not provided much details on the disappearing requests themselves, or on the channel through which these requests were reaching Tomcat. But one thing that he did mention, is that these requests are similar - and even in general the same - as other requests which do get processed normally. As per his own words : For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. If we take this at face value, then it should not be so that these requests are so malformed that Tomcat discards them without further ado. Also - but maybe I'm wrong there - I would expect, if Tomcat discards a request for being malformed - that something would appear in the Tomcat error log. But according to the OP it doesn't. Finally - and there is a bit of an assumption on my part here - I assume that when the OP says that he sees the request with Wireshark (prior to it disappearing in Tomcat), he was running Wireshark on the Tomcat host itself. That would make it unlikely that another external component is at play. All of the above led me to suspect that something in the application itself may be playing a role here. Of course, that all does not necessarily prove that some other component than Tomcat is not dropping some packets/requests. -- Clarifications for the missed information: - Wireshark runs on the same machine where Tomcat runs - Channel through which these requests were reaching Tomcat - HTTP Chanel - Can you describe your deployment? Are you accessing Tomcat directly via HTTP? What networking components are between your test client(s) and Tomcat - A war file with a servlet to POST the requests. - There are 2 machines (LAN) where client and servers are installed. Client is used to send the requests to Server. - Disabled the firewall in the server machine - Tomcat will be accessed directly via HTTP - This issue occurs even if I use loopback address/localhost instead of Server IP Address in the request URL (Install both client and server in the same machine). - Tested with both apache-tomcat-7.0.27.exe and apache-tomcat-7.0.42-windows-x86.zip - Also, Windows event logs will not have any entries when the requests getting missed. Strange that this is happening only to me. Overnight run of my application with Jetty Server(1167 requests) did not miss any requests. So I am hoping that application is not contributing to the missing requests. I may be wrong here. Is there any way to get more logs on request processing in Tomcat? Thanks. - DISCLAIMER: This electronic message and any attachments to this electronic message is intended for the exclusive use of the addressee(s) named herein and may contain legally privileged and confidential information. It is the property of Celstream Systems Private Limited. If you are not the intended recipient, you are hereby strictly notified not to copy, forward, distribute or use this message or any attachments thereto. If you have received this message in error, please delete it and all copies thereof, from your system and notify the sender at Celstream Systems or administrat...@celstream.com immediately
Re: Packet misses in Tomcat
Divyaprakash Y wrote: Issue: Few packets do not reach the application servlet but tomcat receives them. The missing packets reach the HTTP layer and thereafter they disappear. This issue is not frequent but occasionally consistent. For the POSTs of missing packet I am not able to find the entry in localhost_access_log. How do you know that the packet reaches tomcat if there is no matching entry in localhost_access.log? Does all other access appear in this file? I could see the packet in the wireshark capture and it has reached HTTP layer so I thought it has reached Tomcat; I may be wrong here. Also, as far as my observation, every hit to the application URLs was getting logged in local access log. And is there anything in the Tomcat error logs ? (Or the Windows Event logs) A HTTP request might be discarded by Tomcat for various reasons (*) before it is ever mapped to an application. In such a case, there is probably also no log of the request in the Access log. But I would expect some error message in the Tomcat error logs. (*) Invalid HTTP request, incomplete request, etc.. If there are really packets being lost somewhere, then for POST requests the request size would not match the Content-length header, and that may be one of these cases. I must say that the packets lost hypothesis sound a bit iffy to me. This is TCP, which should detect missing packets and cause a client connection abort if it was the case. The invalid HTTP request being rejected by Tomcat sounds more probable to me. i I could not find anything (Exception) in tomcat std error log file or any file in the log folder. But I have not checked the Windows event logs. Will do that. Regarding the content length, there is no mismatch as I have validated using Wireshark and also the same packets(In terms of structure) have reached the application previously. W.r.t. the last point, I could see TCP ACK for the received packet in the wireshark and the packet has reached HTTP layer which goes missing thereafter. What do you mean exactly by the packet has reached HTTP layer ? Are you using the word packet in the TCP/IP sense here, or do you mean HTTP request (composed of one or more TCP/IP packets) ? Does one complete HTTP POST request (headers and body) fit into one TCP packet ? And how exactly do you recognise that some particular packet (or HTTP request) has not been processed by the application ? Do these disappearing requests have some special characteristic that allows you to distinguish them from other requests to the same application ? Or is it so that all the requests for that same application disappear ? Do all the ones that disappear have something in common, that other requests (which do not disappear) do not have ? Suggestion : paste the content of your Tomcat's server.xml file in your next message, after removing any private information such as host name or IP, passwords etc. (Do not send it as attachment, this list often removes them). Actually, I meant HTTP request has reached HTTP Layer. I could recognise the miss as some noticeable activity will happen upon the reception of the request in the application because of which it is easy to identify the misses in this case when compared to other requests. For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. I could see this in Tomcat 7.0 and Tomcat 7.0.42. Here is the server.xml details: ?xml version='1.0' encoding='utf-8'? !-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -- !-- Note: A Server is not itself a Container, so you may not define subcomponents such as Valves at this level. Documentation at /docs/config/server.html -- Server port=8105 shutdown=SHUTDOWN !-- Security listener. Documentation at /docs/config/listeners.html Listener className=org.apache.catalina.security.SecurityListener / -- !--APR library loader. Documentation at /docs/apr.html -- ! Listener
RE: Packet misses in Tomcat
Issue: Few packets do not reach the application servlet but tomcat receives them. The missing packets reach the HTTP layer and thereafter they disappear. This issue is not frequent but occasionally consistent. For the POSTs of missing packet I am not able to find the entry in localhost_access_log. How do you know that the packet reaches tomcat if there is no matching entry in localhost_access.log? Does all other access appear in this file? I could see the packet in the wireshark capture and it has reached HTTP layer so I thought it has reached Tomcat; I may be wrong here. Also, as far as my observation, every hit to the application URLs was getting logged in local access log. And is there anything in the Tomcat error logs ? (Or the Windows Event logs) A HTTP request might be discarded by Tomcat for various reasons (*) before it is ever mapped to an application. In such a case, there is probably also no log of the request in the Access log. But I would expect some error message in the Tomcat error logs. (*) Invalid HTTP request, incomplete request, etc.. If there are really packets being lost somewhere, then for POST requests the request size would not match the Content-length header, and that may be one of these cases. I must say that the packets lost hypothesis sound a bit iffy to me. This is TCP, which should detect missing packets and cause a client connection abort if it was the case. The invalid HTTP request being rejected by Tomcat sounds more probable to me. i I could not find anything (Exception) in tomcat std error log file or any file in the log folder. But I have not checked the Windows event logs. Will do that. Regarding the content length, there is no mismatch as I have validated using Wireshark and also the same packets(In terms of structure) have reached the application previously. W.r.t. the last point, I could see TCP ACK for the received packet in the wireshark and the packet has reached HTTP layer which goes missing thereafter. What do you mean exactly by the packet has reached HTTP layer ? Are you using the word packet in the TCP/IP sense here, or do you mean HTTP request (composed of one or more TCP/IP packets) ? Does one complete HTTP POST request (headers and body) fit into one TCP packet ? And how exactly do you recognise that some particular packet (or HTTP request) has not been processed by the application ? Do these disappearing requests have some special characteristic that allows you to distinguish them from other requests to the same application ? Or is it so that all the requests for that same application disappear ? Do all the ones that disappear have something in common, that other requests (which do not disappear) do not have ? Suggestion : paste the content of your Tomcat's server.xml file in your next message, after removing any private information such as host name or IP, passwords etc. (Do not send it as attachment, this list often removes them). Actually, I meant HTTP request has reached HTTP Layer. I could recognise the miss as some noticeable activity will happen upon the reception of the request in the application because of which it is easy to identify the misses in this case when compared to other requests. For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. I could see this in Tomcat 7.0 and Tomcat 7.0.42. Here is the server.xml details: ?xml version='1.0' encoding='utf-8'? !-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -- !-- Note: A Server is not itself a Container, so you may not define subcomponents such as Valves at this level. Documentation at /docs/config/server.html -- Server port=8105 shutdown=SHUTDOWN !-- Security listener. Documentation at /docs/config/listeners.html Listener className=org.apache.catalina.security.SecurityListener / -- !--APR library loader. Documentation at /docs/apr.html --
Re: Packet misses in Tomcat
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 1/7/14, 5:09 AM, André Warnier wrote: I do not pretend to know your system, nor your application, nor that the following is a definite explanation. But on the base of the currently available data, I would say : - it is quite unlikely that Tomcat 7 is randomly dropping requests. If it was, then I would imagine that this list would be overflowing with cries for help. There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall seeing any significant number of issues mentioning dropped requests. - it also doesn't seem, from your wireshark-related observations, that the requests are being lost outside of Tomcat. - so I would say at this point, that the most likely place for requests to disappear is in your own application. It seems that Tomcat is not logging the request in its access log, so it's more likely that the request is either malformed to such an extent that Tomcat rejects the request altogether or that the request never reaches Tomcat. Divyaprakash, can you describe your deployment? Are you accessing Tomcat directly via HTTP? What networking components are between your test client(s) and Tomcat? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJSzECoAAoJEBzwKT+lPKRY0PMQAJHco/XSh8G2JGPoOVliZwGr nwa+Qrn3eFE491EyLO+y+J4A9nQABDjaYtYYCZpnEf4ZDHB8almmumBVghj48MZ4 3YyUVWQdFv9UsLMOAvUR8B8zL4O89WeaLPqoUHqJkEL6/8z3PmDIQqWb5t3Hjmsg T+NI6Mw0ZbjmoxN2AVDdlSae8I63c93dYmAO8w3whifhHv+BGh2lJ+0GTX40LvHu fk/s502w/zN82p3WQ8xJoeltJffXHurUHv8pn98q9uTmHWKErC+GlnnPIHf7cQvg 1rT95so1GhKeeGr+4nCHznQtKwPl3pwFyDo0G7NJYlRrh2qICgSeKfJ5UuC5uSJU xoB6vbcDVZ/jhwmGMlRwnvEdMOAZGZv6UPMViMMuxk++DGHVEBt/RLcwwR3DZEnM 0jyF7vwA+5M9b4sTBbynvpnemw3VS4YVXiRpleoshPrjDDMKz2ZFp2EuMKMPRWOl dIUm7ZED6lJV/HVdYYYtJZ6o78/4jFrNt6WgHI1bofOj2WPbOoUExCcvFLCDh2UL +MkuonxYMocQOobXziNraU4XRl5Ukurz+vMak/KCYEL8tRh64j1H/Dk9DV9eUIeF FWRu0XwuSgPSFoGGlCHlnKupWIL2OGKkuSJEGd7IS3YqbnpaLXr1EldkNb3nlKq2 z8ZF+xyfGdC3qnlz1uRm =DRad -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Packet misses in Tomcat
Christopher, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 1/7/14, 5:09 AM, André Warnier wrote: I do not pretend to know your system, nor your application, nor that the following is a definite explanation. But on the base of the currently available data, I would say : - it is quite unlikely that Tomcat 7 is randomly dropping requests. If it was, then I would imagine that this list would be overflowing with cries for help. There is quite a bit of traffic on this list related to Tomcat 7, but I don't recall seeing any significant number of issues mentioning dropped requests. - it also doesn't seem, from your wireshark-related observations, that the requests are being lost outside of Tomcat. - so I would say at this point, that the most likely place for requests to disappear is in your own application. It seems that Tomcat is not logging the request in its access log, so it's more likely that the request is either malformed to such an extent that Tomcat rejects the request altogether or that the request never reaches Tomcat. ... Hi. Of course I am going essentially by what the OP provided earlier as information, and he has not provided much details on the disappearing requests themselves, or on the channel through which these requests were reaching Tomcat. But one thing that he did mention, is that these requests are similar - and even in general the same - as other requests which do get processed normally. As per his own words : For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. If we take this at face value, then it should not be so that these requests are so malformed that Tomcat discards them without further ado. Also - but maybe I'm wrong there - I would expect, if Tomcat discards a request for being malformed - that something would appear in the Tomcat error log. But according to the OP it doesn't. Finally - and there is a bit of an assumption on my part here - I assume that when the OP says that he sees the request with Wireshark (prior to it disappearing in Tomcat), he was running Wireshark on the Tomcat host itself. That would make it unlikely that another external component is at play. All of the above led me to suspect that something in the application itself may be playing a role here. Of course, that all does not necessarily prove that some other component than Tomcat is not dropping some packets/requests. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Packet misses in Tomcat
Issue: Few packets do not reach the application servlet but tomcat receives them. The missing packets reach the HTTP layer and thereafter they disappear. This issue is not frequent but occasionally consistent. For the POSTs of missing packet I am not able to find the entry in localhost_access_log. How do you know that the packet reaches tomcat if there is no matching entry in localhost_access.log? Does all other access appear in this file? Chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Packet misses in Tomcat
Issue: Few packets do not reach the application servlet but tomcat receives them. The missing packets reach the HTTP layer and thereafter they disappear. This issue is not frequent but occasionally consistent. For the POSTs of missing packet I am not able to find the entry in localhost_access_log. How do you know that the packet reaches tomcat if there is no matching entry in localhost_access.log? Does all other access appear in this file? I could see the packet in the wireshark capture and it has reached HTTP layer so I thought it has reached Tomcat; I may be wrong here. Also, as far as my observation, every hit to the application URLs was getting logged in local access log. Thanks. - DISCLAIMER: This electronic message and any attachments to this electronic message is intended for the exclusive use of the addressee(s) named herein and may contain legally privileged and confidential information. It is the property of Celstream Systems Private Limited. If you are not the intended recipient, you are hereby strictly notified not to copy, forward, distribute or use this message or any attachments thereto. If you have received this message in error, please delete it and all copies thereof, from your system and notify the sender at Celstream Systems or administrat...@celstream.com immediately. - - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Packet misses in Tomcat
Divyaprakash Y wrote: Issue: Few packets do not reach the application servlet but tomcat receives them. The missing packets reach the HTTP layer and thereafter they disappear. This issue is not frequent but occasionally consistent. For the POSTs of missing packet I am not able to find the entry in localhost_access_log. How do you know that the packet reaches tomcat if there is no matching entry in localhost_access.log? Does all other access appear in this file? I could see the packet in the wireshark capture and it has reached HTTP layer so I thought it has reached Tomcat; I may be wrong here. Also, as far as my observation, every hit to the application URLs was getting logged in local access log. And is there anything in the Tomcat error logs ? (Or the Windows Event logs) A HTTP request might be discarded by Tomcat for various reasons (*) before it is ever mapped to an application. In such a case, there is probably also no log of the request in the Access log. But I would expect some error message in the Tomcat error logs. (*) Invalid HTTP request, incomplete request, etc.. If there are really packets being lost somewhere, then for POST requests the request size would not match the Content-length header, and that may be one of these cases. I must say that the packets lost hypothesis sound a bit iffy to me. This is TCP, which should detect missing packets and cause a client connection abort if it was the case. The invalid HTTP request being rejected by Tomcat sounds more probable to me. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Packet misses in Tomcat
Issue: Few packets do not reach the application servlet but tomcat receives them. The missing packets reach the HTTP layer and thereafter they disappear. This issue is not frequent but occasionally consistent. For the POSTs of missing packet I am not able to find the entry in localhost_access_log. How do you know that the packet reaches tomcat if there is no matching entry in localhost_access.log? Does all other access appear in this file? I could see the packet in the wireshark capture and it has reached HTTP layer so I thought it has reached Tomcat; I may be wrong here. Also, as far as my observation, every hit to the application URLs was getting logged in local access log. And is there anything in the Tomcat error logs ? (Or the Windows Event logs) A HTTP request might be discarded by Tomcat for various reasons (*) before it is ever mapped to an application. In such a case, there is probably also no log of the request in the Access log. But I would expect some error message in the Tomcat error logs. (*) Invalid HTTP request, incomplete request, etc.. If there are really packets being lost somewhere, then for POST requests the request size would not match the Content-length header, and that may be one of these cases. I must say that the packets lost hypothesis sound a bit iffy to me. This is TCP, which should detect missing packets and cause a client connection abort if it was the case. The invalid HTTP request being rejected by Tomcat sounds more probable to me. i I could not find anything (Exception) in tomcat std error log file or any file in the log folder. But I have not checked the Windows event logs. Will do that. Regarding the content length, there is no mismatch as I have validated using Wireshark and also the same packets(In terms of structure) have reached the application previously. W.r.t. the last point, I could see TCP ACK for the received packet in the wireshark and the packet has reached HTTP layer which goes missing thereafter. Is there any configuration to get more logs on the operations happening during the mapping of request to the application? - DISCLAIMER: This electronic message and any attachments to this electronic message is intended for the exclusive use of the addressee(s) named herein and may contain legally privileged and confidential information. It is the property of Celstream Systems Private Limited. If you are not the intended recipient, you are hereby strictly notified not to copy, forward, distribute or use this message or any attachments thereto. If you have received this message in error, please delete it and all copies thereof, from your system and notify the sender at Celstream Systems or administrat...@celstream.com immediately. - - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Packet misses in Tomcat
Issue: Few packets do not reach the application servlet but tomcat receives them. The missing packets reach the HTTP layer and thereafter they disappear. This issue is not frequent but occasionally consistent. For the POSTs of missing packet I am not able to find the entry in localhost_access_log. How do you know that the packet reaches tomcat if there is no matching entry in localhost_access.log? Does all other access appear in this file? I could see the packet in the wireshark capture and it has reached HTTP layer so I thought it has reached Tomcat; I may be wrong here. Also, as far as my observation, every hit to the application URLs was getting logged in local access log. And is there anything in the Tomcat error logs ? (Or the Windows Event logs) A HTTP request might be discarded by Tomcat for various reasons (*) before it is ever mapped to an application. In such a case, there is probably also no log of the request in the Access log. But I would expect some error message in the Tomcat error logs. (*) Invalid HTTP request, incomplete request, etc.. If there are really packets being lost somewhere, then for POST requests the request size would not match the Content-length header, and that may be one of these cases. I must say that the packets lost hypothesis sound a bit iffy to me. This is TCP, which should detect missing packets and cause a client connection abort if it was the case. The invalid HTTP request being rejected by Tomcat sounds more probable to me. i I could not find anything (Exception) in tomcat std error log file or any file in the log folder. But I have not checked the Windows event logs. Will do that. Regarding the content length, there is no mismatch as I have validated using Wireshark and also the same packets(In terms of structure) have reached the application previously. W.r.t. the last point, I could see TCP ACK for the received packet in the wireshark and the packet has reached HTTP layer which goes missing thereafter. What do you mean exactly by the packet has reached HTTP layer ? Are you using the word packet in the TCP/IP sense here, or do you mean HTTP request (composed of one or more TCP/IP packets) ? Does one complete HTTP POST request (headers and body) fit into one TCP packet ? And how exactly do you recognise that some particular packet (or HTTP request) has not been processed by the application ? Do these disappearing requests have some special characteristic that allows you to distinguish them from other requests to the same application ? Or is it so that all the requests for that same application disappear ? Do all the ones that disappear have something in common, that other requests (which do not disappear) do not have ? Suggestion : paste the content of your Tomcat's server.xml file in your next message, after removing any private information such as host name or IP, passwords etc. (Do not send it as attachment, this list often removes them). Actually, I meant HTTP request has reached HTTP Layer. I could recognise the miss as some noticeable activity will happen upon the reception of the request in the application because of which it is easy to identify the misses in this case when compared to other requests. For the query regarding All requests, all requests do not disappear. More importantly, sometimes all requests reach the application when I POST same set of requests. To give a rough picture, 1-2 requests fail in a set of 45-50 requests and this behaviour varies [The request which failed in my one test cycle succeeds in another cycle]. I could see this in Tomcat 7.0 and Tomcat 7.0.42. Here is the server.xml details: ?xml version='1.0' encoding='utf-8'? !-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -- !-- Note: A Server is not itself a Container, so you may not define subcomponents such as Valves at this level. Documentation at /docs/config/server.html -- Server port=8105 shutdown=SHUTDOWN !-- Security listener. Documentation at /docs/config/listeners.html Listener className=org.apache.catalina.security.SecurityListener / -- !--APR library loader. Documentation at /docs/apr.html -- ! Listener
