On 16/01/2012 20:12, Bert Verhees wrote: > Thanks, André, for your reply > >> >> Is that per "person", or per POST ? > > It is per Person AND per post. The post creates a person-object which is > stored in a database, and the person-object is formed out of an array of > strings. >> >> to an webservice-call. >>> After some validation, the webservice >> >> of which we here know nothing.. > > It has nothing to do with the problem. But you may, of course know what > it is all about. It is about forming OpenEHR-objects, the string arrays > serve to create path-strings-array and value-string array. The problem > is not there, I know, because there are three ways of running the code. > > One is to call the libraries behind the webservices directly, and also > do the complete run of posting and validating. It is exact the same > code, it is only passing the webservices. This runs fine. > They other one is using axis2 as a standalone service, and the third one > is using tomcat as a container for axis. > >> >> posts the data to a postgresql >>> database. >>> >>> I have some console-output to see how long it takes. >>> >>> storing..... Person:26, ID:27 >>> Creating EHR for:213d7c09-949c-4757-a91d-da5510215472 >>> GUIDs saved to table:213d7c09-949c-4757-a91d-da5510215472, >>> 73b60dd4-d745-4267-9c6f-fc8b067bfeb9 >>> stored..... Person:26, UID:213d7c09-949c-4757-a91d-da5510215472 >>> PT0.647S >>> >>> As you can see, it starts with less then a second, sometime 0.3 >>> seconds, sometimes 1.1 second, depending on the number of elements in >>> the array to store. >> >> What array, where ? >> >>> >>> After 1000 persons stored, it takes 4 seconds >> >> Do you mean that you send data in one POST about 1000 persons (each >> with 50 text data elements) ? > > No, it is each time a separate call. > > >> >>> >>> storing..... Person:998, ID:989 >>> Creating EHR for:49c96d87-38db-491d-a824-78800615e80a >>> GUIDs saved to table:49c96d87-38db-491d-a824-78800615e80a, >>> 3554514f-ccb6-40dd-92f2-d9140760870e >>> stored..... Person:998, UID:49c96d87-38db-491d-a824-78800615e80a >>> PT4.147S >>> >>> When running in Axis2 standalone, it stays about 4 seconds until 8000 >>> persons stored (which is the complete group to store) >> >> is that 1 POST, or 8000 POSTs to the server ? > > This is one post. >> >>> This is slow, I must investigate the library directly without >>> webservice if that is a problem of my coding. So we forget that for >>> the moment. >>> >>> The problem which bothers me is when I am using Tomcat. >>> >>> The problem start to occur after posting 1200 patients, it takes 8 >>> seconds. When it reaches 1300, it already takes 17 seconds. >>> When at Person 1330, it is already 29 seconds, and at 1340 it is 39 >>> seconds, and than a few persons more and a time-out exception occurs. >>> >>> I receive a java.net.SocketTimeoutException: Read timed out from the >>> client side-code. The default time-out time for Axis2-code is 40 >>> seconds. >>> The printstacktrace is below, but it is client side stacktrace >>> >>> I try a lot of things to solve the problem. I put a >>> Thread.sleep(1000); in my client-code between to postings, I thought, >>> maybe the webservice needs some rest to do garbage-clean up. Because >>> it looks very much like a memory blowing up. But it didn't help. >>> >>> Of course, if you need any information, I will be happy to provide >>> it, please give some hints how to provide it >>> >> >> You could start by telling us which version of Tomcat this is, under >> with Java JVM version, and on which platform (Windows, Linux,..). >> Then tell us with how much available memory this Tomcat is running. > > I did tell in the message which Tomcat version I use, and I also did > tell which platform I use > I don't mind telling you again >> I am using Tomcat6 6.0.32-5ubuntu1.1 together with Axis2 1.6.1 > > I did not tell you which JVM I use, that is > OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10) > OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode) > > I don't know how much memory is avalaible for tomcat, but this is what > "top" tells me > Mem: 4055608k total, 3277200k used, 778408k free, 2876k buffers > Swap: 9764860k total, 1102476k used, 8662384k free, 243536k cached
The system is using swap. What else is running on the system? The top output should also be able to tell you how much memory Tomcat is consuming. > I see there is 8 Gb swap unused, so Tomcat does not bother claiming it. That is not clear from the above. The JConsole and VisualVM applications provided with the JDK will tell you how much memory is being consumed by the application in various areas of the Java program's process heap. You may use them to observe behaviour during the operation you are experimenting with. p >> Making some wild guesses here, because you do not really provide any >> precise information about what exactly you are trying to do : > > Excuse me for putting you in this situation. I did not know which > information to provide. > >> >> Let's suppose that your client application puts together one POST >> request, consisting of XML data composed of 1000, 2000, 3000 elements >> "person", each composed of up to 50 text data sub-elements, and then >> sends that POST to the server. >> And that the server application receives this big POST request, reads >> the XML body of it, parses this XML in memory as a large "XML document >> object", before it starts splitting it into 1000, 2000, 3000 "person" >> elements in order to update records in the database.. > > This suggestion is not right, every person is posted with a separate > webservice-call. And every person consists of an array with maximum 50 > values (house number, city-name, short strings), and another array with > the same number of value-descriptors (which are in fact openEHR > archetype-path-strings of about 200 characters), so every person is > maximum 15 Kb. > > I hope with this more precise information, you can give me some clue > about where to look where the things break down. > > Thanks again for your reply and your questions for more information. > > Bert Verhees > >> >> Then, if all the above roughly matches what you are doing, I would not >> be very surprised if you got into serious memory usage problems, >> forcing the server to swap this data and slowing down your process >> dramatically (and exponentially) as the quantity of data increases. >> >> Have you tried a design where you send the data to the server in more >> manageable chunks ? >> What about a loop where you would send 100 persons's data at one time, >> and run the loop 80 times ? >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- [key:62590808]
signature.asc
Description: OpenPGP digital signature
