Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bert, On 1/18/12 4:22 PM, Bert Verhees wrote: I have done my testing, and at this moment, it seems that I found the problem, I forgot to close a databaseconnection in a function which is called a zillion times. Just one line of code, and the problem was solved. Consider doing the following: In your development environment, set maxActive in your connection pool to 1. Also set the logAbandoned and similar settings to catch resources that you don't properly close. With maxActive=1, your webapp will stop dead for several minutes while the connection pool starves waiting for the one missing connection to (never) be returned to the pool before killing it and creating a new one. Trust me: you'll notice when that happens. We have the logAbandoned stuff even set in production just in case we don't notice something in development. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8ZhQoACgkQ9CaO5/Lv0PAd9ACfXiOQju6eLIMpB14bZhbkip/B WjoAoJYJLf+vpYqh3IGYumI7zizO6Ggp =cCmq -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bert, On 1/16/12 11:15 AM, Bert Verhees wrote: I am not sure if it is an Axis2 problem or a Tomcat-problem. I have an client-application for test-purposes which runs inside the Eclipse-IDE. The problem only occurs when running Axis2 inside Tomcat6, when running in Axis2 as stand-alone server, it runs better. SO, running Axis2 using it's built-in HTTP server (I didn't know it had one) has acceptable performance, but running Axis2 within Tomcat does not have acceptable performance? I have some console-output to see how long it takes. How about real profiling data? When running in Axis2 standalone, it stays about 4 seconds until 8000 persons stored (which is the complete group to store) This is slow, I must investigate the library directly without webservice if that is a problem of my coding. So we forget that for the moment. So, Axis2 performance is also unaccaptable. The problem which bothers me is when I am using Tomcat. The problem start to occur after posting 1200 patients, it takes 8 seconds. So your observation is that, when running within Tomcat, your code appears to perform more slowly. Is that consistent? I wonder if your code / database slows down considerably as data is added and you are running your in-Tomcat tests after the out-of-Tomcat tests. I try a lot of things to solve the problem. Like what? I put a Thread.sleep(1000); in my client-code between to postings, I thought, maybe the webservice needs some rest to do garbage-clean up. Obviously, pausing for a second per record will limit performance. GC will run when it has to -- you don't have to give it time to do its job. Because it looks very much like a memory blowing up. But it didn't help. What makes you think that memory is blowing up? It probably didn't help because you have mis-identified the problem. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAk8W7QUACgkQ9CaO5/Lv0PDBEQCfTaLMZE7TPBdwAVaFe56MRBDC OccAmNgqaj57ji+z11Z/hVourbydlS0= =H9YI -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bert, On 1/16/12 4:39 PM, Bert Verhees wrote: On 16-01-12 22:26, Pid wrote: The JConsole and VisualVM applications provided with the JDK will tell you how much memory is being consumed by the application in various areas of the Java program's process heap. You may use them to observe behaviour during the operation you are experimenting with. Thanks Pid, I will certainly do that. I did not know about these possibilities. You might also want to take some thread dumps during the slow-processing to see where your code is spending its time. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8W77AACgkQ9CaO5/Lv0PASygCeNQ848/auhyuTzLfohr0vYQH4 vTIAoJ3jNa0mgMD7riVibFVFqngFcrXj =eJcg -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
Sorry, readers of this list for not replying, I am very busy investigating the problem, I try replying later. Bert On 16-01-12 23:55, André Warnier wrote: Hi. I'll follow-up with an out of band message just for now. It was not very clear from your initial explanations whether you were sending all of the 1000, 2000, .. 8000 person's data records at once, in one HTTP POST, or sending the data in a person-by-person way. So my wild guess was based on the symptoms you were providing, which looked very much like a memory problem caused by the processing of increasingly large XML documents using a DOM-like parser. And sorry for missing your references to the platform and Tomcat used. Jumping ahead however, as Pid noted after your memory figures, it would appear that on your system, about 1 GB of swap space on disk /is/ being used, which would be relatively unusual for a Linux system if this application is the only one that is running. If indeed this swap space is being used by the Tomcat process, then it would definitely explain why the processing is slow (but the reason why that is so would still need to be found). (As an idea : as soon as an application starts to need the swap area, you can easily divide its performance by a factor 10). It would be interesting to see what the memory usage of the Tomcat-related processes is, while processing these requests. Your top utility would also give an example of this (use CTRL-F, n , CR to sort the processes by memory usage). The general point is : there is no specific reason, per se, why Tomcat would slow down dramatically when processing 1300 requests, as compared to 1000 similar requests. One would expect the 1300 requests to need approximately 130% of the time needed to process 1000 requests, but not the double amount of time. So if it is not the back-end database, and it is not the Tomcat code per se, that leaves - the web application (including its Axis2 components) - or some fundamentally different way of handling the data, between - on one side the client + the stand-alone Axis2 server - on the other side the client + the Axis2 Web Service webapp this causing, perhaps, Tomcat to become saturated and start trashing I do not really know Axis2, other than what I just read on the Wikipedia page. So I can only make some further guesses at this stage : You mention that when running Axis2 stand-alone, the processing is relatively fast (8000 persons in 4 seconds), while when running the same thing as a Web service under Tomcat, it is infinitely slower. I will assume for now that the background database updates are the same in both cases, and that this is thus not a factor. You also seem to say that when running Axis2 stand-alone, the 8000 persons data is only one POST, while through HTTP it would be 8000 POSTs (or maybe I misunderstood that also). In any case, can it be that you are comparing apples with oranges ? As a loose collection of suggestions : - when you send SOAP requests via HTTP, there is an extra layer of wrapping around the SOAP data, both on the client side (wrap as SOAP-HTTP request) and on the server side (unwrap the SOAP-HTTP wrapping). One would expect some impact of these in any case, if repeated several thousand times in one case, and not in the other. - Axis2 seems to allow for configuring entirely different styles of XML parsing (DOM or stream/event-based). When your client talks to the Axis2 stand-alone server, is it really sending the data the same way, and is the Axis2 server library parsing the data the same way ? Asked another way : is it perhaps so that when you send the data via HTTP to the Web Service, the Web Service code uses a DOM-like parser, while when running in stand-alone mode, the Axis2 server uses a stream-like parsing method ? - maybe the Axis2 Web Service webapp library leaks memory ? - maybe your webapp using that library leaks memory ? - are your client and the server running on the same host ? SOAP is XML. XML processing, particularly in DOM-mode, can use a significant amount of memory (*). If the client (to create the XML data) and the server (to parse it) are running on the same host, and more or less synchronously, you are doubling the amount of memory required to process your request/response, and that could explain the onset of swapping. - when your client sends requests to the Web Servive over HTTP, does it send several requests in parallel ? How many Tomcat threads are available for processing requests, and how many are effectively processing requests in parallel ? - maybe indeed Tomcat is spending a lot of its time doing memory garbage collection, because of one of the above circumstances (*) an estimate is that to represent an XML document in memory using a DOM representation, you easily need 10 times the original amount of data in terms of RAM. Stream-style parsing and processing is much faster and much leaner in terms of
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
I have done my testing, and at this moment, it seems that I found the problem, I forgot to close a databaseconnection in a function which is called a zillion times. Just one line of code, and the problem was solved. At least I am so honest I admit it freely, and it will follow my career until the end of times. Thanks Pid an André for thinking with me. The good news is that tomcat6/axis2 are good products, at least in my situation. Kind regards Bert Verhees - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
Hi, I hope you can help me analyzing this problem: I am using Tomcat6 6.0.32-5ubuntu1.1 together with Axis2 1.6.1 I am not sure if it is an Axis2 problem or a Tomcat-problem. I have an client-application for test-purposes which runs inside the Eclipse-IDE. The problem only occurs when running Axis2 inside Tomcat6, when running in Axis2 as stand-alone server, it runs better. What I do is posting arrays of strings (about 50 elements, about 5Kb max) to an webservice-call. After some validation, the webservice posts the data to a postgresql database. I have some console-output to see how long it takes. storing. Person:26, ID:27 Creating EHR for:213d7c09-949c-4757-a91d-da5510215472 GUIDs saved to table:213d7c09-949c-4757-a91d-da5510215472, 73b60dd4-d745-4267-9c6f-fc8b067bfeb9 stored. Person:26, UID:213d7c09-949c-4757-a91d-da5510215472 PT0.647S As you can see, it starts with less then a second, sometime 0.3 seconds, sometimes 1.1 second, depending on the number of elements in the array to store. After 1000 persons stored, it takes 4 seconds storing. Person:998, ID:989 Creating EHR for:49c96d87-38db-491d-a824-78800615e80a GUIDs saved to table:49c96d87-38db-491d-a824-78800615e80a, 3554514f-ccb6-40dd-92f2-d9140760870e stored. Person:998, UID:49c96d87-38db-491d-a824-78800615e80a PT4.147S When running in Axis2 standalone, it stays about 4 seconds until 8000 persons stored (which is the complete group to store) This is slow, I must investigate the library directly without webservice if that is a problem of my coding. So we forget that for the moment. The problem which bothers me is when I am using Tomcat. The problem start to occur after posting 1200 patients, it takes 8 seconds. When it reaches 1300, it already takes 17 seconds. When at Person 1330, it is already 29 seconds, and at 1340 it is 39 seconds, and than a few persons more and a time-out exception occurs. I receive a java.net.SocketTimeoutException: Read timed out from the client side-code. The default time-out time for Axis2-code is 40 seconds. The printstacktrace is below, but it is client side stacktrace I try a lot of things to solve the problem. I put a Thread.sleep(1000); in my client-code between to postings, I thought, maybe the webservice needs some rest to do garbage-clean up. Because it looks very much like a memory blowing up. But it didn't help. Of course, if you need any information, I will be happy to provide it, please give some hints how to provide it --- Kind regards and much thanks Bert Verhees --- I don't know if it is of any help, but I put the exception-trace below 14.01.2012 13:33:52 INFO (HTTPSender.java:196) - Unable to sendViaPost to url[http://localhost:8080/axis2/services/DemographicService/] java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:146) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413) at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:621) at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193) at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:404) at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:231) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:443) at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:406) at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165) at
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
Bert Verhees wrote: Hi, I hope you can help me analyzing this problem: I am using Tomcat6 6.0.32-5ubuntu1.1 together with Axis2 1.6.1 I am not sure if it is an Axis2 problem or a Tomcat-problem. I have an client-application for test-purposes which runs inside the Eclipse-IDE. The problem only occurs when running Axis2 inside Tomcat6, when running in Axis2 as stand-alone server, it runs better. What I do is posting arrays of strings (about 50 elements, about 5Kb max) Is that per person, or per POST ? to an webservice-call. After some validation, the webservice of which we here know nothing.. posts the data to a postgresql database. I have some console-output to see how long it takes. storing. Person:26, ID:27 Creating EHR for:213d7c09-949c-4757-a91d-da5510215472 GUIDs saved to table:213d7c09-949c-4757-a91d-da5510215472, 73b60dd4-d745-4267-9c6f-fc8b067bfeb9 stored. Person:26, UID:213d7c09-949c-4757-a91d-da5510215472 PT0.647S As you can see, it starts with less then a second, sometime 0.3 seconds, sometimes 1.1 second, depending on the number of elements in the array to store. What array, where ? After 1000 persons stored, it takes 4 seconds Do you mean that you send data in one POST about 1000 persons (each with 50 text data elements) ? storing. Person:998, ID:989 Creating EHR for:49c96d87-38db-491d-a824-78800615e80a GUIDs saved to table:49c96d87-38db-491d-a824-78800615e80a, 3554514f-ccb6-40dd-92f2-d9140760870e stored. Person:998, UID:49c96d87-38db-491d-a824-78800615e80a PT4.147S When running in Axis2 standalone, it stays about 4 seconds until 8000 persons stored (which is the complete group to store) is that 1 POST, or 8000 POSTs to the server ? This is slow, I must investigate the library directly without webservice if that is a problem of my coding. So we forget that for the moment. The problem which bothers me is when I am using Tomcat. The problem start to occur after posting 1200 patients, it takes 8 seconds. When it reaches 1300, it already takes 17 seconds. When at Person 1330, it is already 29 seconds, and at 1340 it is 39 seconds, and than a few persons more and a time-out exception occurs. I receive a java.net.SocketTimeoutException: Read timed out from the client side-code. The default time-out time for Axis2-code is 40 seconds. The printstacktrace is below, but it is client side stacktrace I try a lot of things to solve the problem. I put a Thread.sleep(1000); in my client-code between to postings, I thought, maybe the webservice needs some rest to do garbage-clean up. Because it looks very much like a memory blowing up. But it didn't help. Of course, if you need any information, I will be happy to provide it, please give some hints how to provide it You could start by telling us which version of Tomcat this is, under with Java JVM version, and on which platform (Windows, Linux,..). Then tell us with how much available memory this Tomcat is running. Making some wild guesses here, because you do not really provide any precise information about what exactly you are trying to do : Let's suppose that your client application puts together one POST request, consisting of XML data composed of 1000, 2000, 3000 elements person, each composed of up to 50 text data sub-elements, and then sends that POST to the server. And that the server application receives this big POST request, reads the XML body of it, parses this XML in memory as a large XML document object, before it starts splitting it into 1000, 2000, 3000 person elements in order to update records in the database.. Then, if all the above roughly matches what you are doing, I would not be very surprised if you got into serious memory usage problems, forcing the server to swap this data and slowing down your process dramatically (and exponentially) as the quantity of data increases. Have you tried a design where you send the data to the server in more manageable chunks ? What about a loop where you would send 100 persons's data at one time, and run the loop 80 times ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
Thanks, André, for your reply Is that per person, or per POST ? It is per Person AND per post. The post creates a person-object which is stored in a database, and the person-object is formed out of an array of strings. to an webservice-call. After some validation, the webservice of which we here know nothing.. It has nothing to do with the problem. But you may, of course know what it is all about. It is about forming OpenEHR-objects, the string arrays serve to create path-strings-array and value-string array. The problem is not there, I know, because there are three ways of running the code. One is to call the libraries behind the webservices directly, and also do the complete run of posting and validating. It is exact the same code, it is only passing the webservices. This runs fine. They other one is using axis2 as a standalone service, and the third one is using tomcat as a container for axis. posts the data to a postgresql database. I have some console-output to see how long it takes. storing. Person:26, ID:27 Creating EHR for:213d7c09-949c-4757-a91d-da5510215472 GUIDs saved to table:213d7c09-949c-4757-a91d-da5510215472, 73b60dd4-d745-4267-9c6f-fc8b067bfeb9 stored. Person:26, UID:213d7c09-949c-4757-a91d-da5510215472 PT0.647S As you can see, it starts with less then a second, sometime 0.3 seconds, sometimes 1.1 second, depending on the number of elements in the array to store. What array, where ? After 1000 persons stored, it takes 4 seconds Do you mean that you send data in one POST about 1000 persons (each with 50 text data elements) ? No, it is each time a separate call. storing. Person:998, ID:989 Creating EHR for:49c96d87-38db-491d-a824-78800615e80a GUIDs saved to table:49c96d87-38db-491d-a824-78800615e80a, 3554514f-ccb6-40dd-92f2-d9140760870e stored. Person:998, UID:49c96d87-38db-491d-a824-78800615e80a PT4.147S When running in Axis2 standalone, it stays about 4 seconds until 8000 persons stored (which is the complete group to store) is that 1 POST, or 8000 POSTs to the server ? This is one post. This is slow, I must investigate the library directly without webservice if that is a problem of my coding. So we forget that for the moment. The problem which bothers me is when I am using Tomcat. The problem start to occur after posting 1200 patients, it takes 8 seconds. When it reaches 1300, it already takes 17 seconds. When at Person 1330, it is already 29 seconds, and at 1340 it is 39 seconds, and than a few persons more and a time-out exception occurs. I receive a java.net.SocketTimeoutException: Read timed out from the client side-code. The default time-out time for Axis2-code is 40 seconds. The printstacktrace is below, but it is client side stacktrace I try a lot of things to solve the problem. I put a Thread.sleep(1000); in my client-code between to postings, I thought, maybe the webservice needs some rest to do garbage-clean up. Because it looks very much like a memory blowing up. But it didn't help. Of course, if you need any information, I will be happy to provide it, please give some hints how to provide it You could start by telling us which version of Tomcat this is, under with Java JVM version, and on which platform (Windows, Linux,..). Then tell us with how much available memory this Tomcat is running. I did tell in the message which Tomcat version I use, and I also did tell which platform I use I don't mind telling you again I am using Tomcat6 6.0.32-5ubuntu1.1 together with Axis2 1.6.1 I did not tell you which JVM I use, that is OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10) OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) I don't know how much memory is avalaible for tomcat, but this is what top tells me Mem: 4055608k total, 3277200k used, 778408k free, 2876k buffers Swap: 9764860k total, 1102476k used, 8662384k free, 243536k cached I see there is 8 Gb swap unused, so Tomcat does not bother claiming it. Making some wild guesses here, because you do not really provide any precise information about what exactly you are trying to do : Excuse me for putting you in this situation. I did not know which information to provide. Let's suppose that your client application puts together one POST request, consisting of XML data composed of 1000, 2000, 3000 elements person, each composed of up to 50 text data sub-elements, and then sends that POST to the server. And that the server application receives this big POST request, reads the XML body of it, parses this XML in memory as a large XML document object, before it starts splitting it into 1000, 2000, 3000 person elements in order to update records in the database.. This suggestion is not right, every person is posted with a separate webservice-call. And every person consists of an array with maximum 50 values (house number, city-name, short strings), and another
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
On 16/01/2012 20:12, Bert Verhees wrote: Thanks, André, for your reply Is that per person, or per POST ? It is per Person AND per post. The post creates a person-object which is stored in a database, and the person-object is formed out of an array of strings. to an webservice-call. After some validation, the webservice of which we here know nothing.. It has nothing to do with the problem. But you may, of course know what it is all about. It is about forming OpenEHR-objects, the string arrays serve to create path-strings-array and value-string array. The problem is not there, I know, because there are three ways of running the code. One is to call the libraries behind the webservices directly, and also do the complete run of posting and validating. It is exact the same code, it is only passing the webservices. This runs fine. They other one is using axis2 as a standalone service, and the third one is using tomcat as a container for axis. posts the data to a postgresql database. I have some console-output to see how long it takes. storing. Person:26, ID:27 Creating EHR for:213d7c09-949c-4757-a91d-da5510215472 GUIDs saved to table:213d7c09-949c-4757-a91d-da5510215472, 73b60dd4-d745-4267-9c6f-fc8b067bfeb9 stored. Person:26, UID:213d7c09-949c-4757-a91d-da5510215472 PT0.647S As you can see, it starts with less then a second, sometime 0.3 seconds, sometimes 1.1 second, depending on the number of elements in the array to store. What array, where ? After 1000 persons stored, it takes 4 seconds Do you mean that you send data in one POST about 1000 persons (each with 50 text data elements) ? No, it is each time a separate call. storing. Person:998, ID:989 Creating EHR for:49c96d87-38db-491d-a824-78800615e80a GUIDs saved to table:49c96d87-38db-491d-a824-78800615e80a, 3554514f-ccb6-40dd-92f2-d9140760870e stored. Person:998, UID:49c96d87-38db-491d-a824-78800615e80a PT4.147S When running in Axis2 standalone, it stays about 4 seconds until 8000 persons stored (which is the complete group to store) is that 1 POST, or 8000 POSTs to the server ? This is one post. This is slow, I must investigate the library directly without webservice if that is a problem of my coding. So we forget that for the moment. The problem which bothers me is when I am using Tomcat. The problem start to occur after posting 1200 patients, it takes 8 seconds. When it reaches 1300, it already takes 17 seconds. When at Person 1330, it is already 29 seconds, and at 1340 it is 39 seconds, and than a few persons more and a time-out exception occurs. I receive a java.net.SocketTimeoutException: Read timed out from the client side-code. The default time-out time for Axis2-code is 40 seconds. The printstacktrace is below, but it is client side stacktrace I try a lot of things to solve the problem. I put a Thread.sleep(1000); in my client-code between to postings, I thought, maybe the webservice needs some rest to do garbage-clean up. Because it looks very much like a memory blowing up. But it didn't help. Of course, if you need any information, I will be happy to provide it, please give some hints how to provide it You could start by telling us which version of Tomcat this is, under with Java JVM version, and on which platform (Windows, Linux,..). Then tell us with how much available memory this Tomcat is running. I did tell in the message which Tomcat version I use, and I also did tell which platform I use I don't mind telling you again I am using Tomcat6 6.0.32-5ubuntu1.1 together with Axis2 1.6.1 I did not tell you which JVM I use, that is OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10) OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) I don't know how much memory is avalaible for tomcat, but this is what top tells me Mem: 4055608k total, 3277200k used, 778408k free, 2876k buffers Swap: 9764860k total, 1102476k used, 8662384k free, 243536k cached The system is using swap. What else is running on the system? The top output should also be able to tell you how much memory Tomcat is consuming. I see there is 8 Gb swap unused, so Tomcat does not bother claiming it. That is not clear from the above. The JConsole and VisualVM applications provided with the JDK will tell you how much memory is being consumed by the application in various areas of the Java program's process heap. You may use them to observe behaviour during the operation you are experimenting with. p Making some wild guesses here, because you do not really provide any precise information about what exactly you are trying to do : Excuse me for putting you in this situation. I did not know which information to provide. Let's suppose that your client application puts together one POST request, consisting of XML data composed of 1000, 2000, 3000 elements person, each composed of up to 50
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
On 16-01-12 22:26, Pid wrote: The JConsole and VisualVM applications provided with the JDK will tell you how much memory is being consumed by the application in various areas of the Java program's process heap. You may use them to observe behaviour during the operation you are experimenting with. Thanks Pid, I will certainly do that. I did not know about these possibilities. best regards Bert Verhees - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Unable to sendViaPost to url...... .... java.net.SocketTimeoutException: Read timed out
Hi. I'll follow-up with an out of band message just for now. It was not very clear from your initial explanations whether you were sending all of the 1000, 2000, .. 8000 person's data records at once, in one HTTP POST, or sending the data in a person-by-person way. So my wild guess was based on the symptoms you were providing, which looked very much like a memory problem caused by the processing of increasingly large XML documents using a DOM-like parser. And sorry for missing your references to the platform and Tomcat used. Jumping ahead however, as Pid noted after your memory figures, it would appear that on your system, about 1 GB of swap space on disk /is/ being used, which would be relatively unusual for a Linux system if this application is the only one that is running. If indeed this swap space is being used by the Tomcat process, then it would definitely explain why the processing is slow (but the reason why that is so would still need to be found). (As an idea : as soon as an application starts to need the swap area, you can easily divide its performance by a factor 10). It would be interesting to see what the memory usage of the Tomcat-related processes is, while processing these requests. Your top utility would also give an example of this (use CTRL-F, n , CR to sort the processes by memory usage). The general point is : there is no specific reason, per se, why Tomcat would slow down dramatically when processing 1300 requests, as compared to 1000 similar requests. One would expect the 1300 requests to need approximately 130% of the time needed to process 1000 requests, but not the double amount of time. So if it is not the back-end database, and it is not the Tomcat code per se, that leaves - the web application (including its Axis2 components) - or some fundamentally different way of handling the data, between - on one side the client + the stand-alone Axis2 server - on the other side the client + the Axis2 Web Service webapp this causing, perhaps, Tomcat to become saturated and start trashing I do not really know Axis2, other than what I just read on the Wikipedia page. So I can only make some further guesses at this stage : You mention that when running Axis2 stand-alone, the processing is relatively fast (8000 persons in 4 seconds), while when running the same thing as a Web service under Tomcat, it is infinitely slower. I will assume for now that the background database updates are the same in both cases, and that this is thus not a factor. You also seem to say that when running Axis2 stand-alone, the 8000 persons data is only one POST, while through HTTP it would be 8000 POSTs (or maybe I misunderstood that also). In any case, can it be that you are comparing apples with oranges ? As a loose collection of suggestions : - when you send SOAP requests via HTTP, there is an extra layer of wrapping around the SOAP data, both on the client side (wrap as SOAP-HTTP request) and on the server side (unwrap the SOAP-HTTP wrapping). One would expect some impact of these in any case, if repeated several thousand times in one case, and not in the other. - Axis2 seems to allow for configuring entirely different styles of XML parsing (DOM or stream/event-based). When your client talks to the Axis2 stand-alone server, is it really sending the data the same way, and is the Axis2 server library parsing the data the same way ? Asked another way : is it perhaps so that when you send the data via HTTP to the Web Service, the Web Service code uses a DOM-like parser, while when running in stand-alone mode, the Axis2 server uses a stream-like parsing method ? - maybe the Axis2 Web Service webapp library leaks memory ? - maybe your webapp using that library leaks memory ? - are your client and the server running on the same host ? SOAP is XML. XML processing, particularly in DOM-mode, can use a significant amount of memory (*). If the client (to create the XML data) and the server (to parse it) are running on the same host, and more or less synchronously, you are doubling the amount of memory required to process your request/response, and that could explain the onset of swapping. - when your client sends requests to the Web Servive over HTTP, does it send several requests in parallel ? How many Tomcat threads are available for processing requests, and how many are effectively processing requests in parallel ? - maybe indeed Tomcat is spending a lot of its time doing memory garbage collection, because of one of the above circumstances (*) an estimate is that to represent an XML document in memory using a DOM representation, you easily need 10 times the original amount of data in terms of RAM. Stream-style parsing and processing is much faster and much leaner in terms of memory requirements. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands,