Thanks for your suggestions. Please find my response below:
On Friday, January 11, 2013 1:16:19 PM UTC+5:30, Дмитрий Дьяченко wrote: > > if i undestand correctly, You have a looong-run application. > Yes, you are right, my application is a really looooong running one. It is expected to run 24 x 7 x 365. > - at start You have 2 ms cycle request-response > The actual request response cycle is a bit more than 2 ms, because after the request is recieved by the application (#9 in my list), it does a few things, which may take longer. But, the time taken to decode the protobuf request is 2 ms at start. > - after a week You have 200 ms cycle > Right, the time taken to decode gradually increases and becomes 200 ms. If left running for longer, it even reached 1 sec in 15 days time. > - Your app is correct and only place where problem may lay is protobuf? > >From the logs which are there in my application, I have seen that only the time taken to decode is gradually increasing. I have also done some memory and thread profiling of my application, but did not find any irregularities (like increase in memory/threads) > > can You do the following? > -- run 'latencytop' at start and after a week? > -- run 'oprofile' at start and after a week? > My application is running on IBM AIX server (using IBM JVM). Are the above tools available for AIX? > -- may be its reasonable to writeout times from crititical points of Ypur > app? smth like: before req? after req? before response - after response > I have logs specific for this purpose which are there at the start and end of these points. Like, before decode and after decode, etc. From these logs I found out the point which was gradually slowing down. > from my [very limited] exp a such problems traditionally becomes from > unintendend sources -- memory allocator, log sub-system, database, swap, > etc... > The only other component involved over here (other than the application and JMV) is the network. But, I don't think network is the issue because the decoding is done after the request object is fully received from the client. Apache MINA is responsible for getting the request from the client. > > Thanks, > Dmitry > > пятница, 11 января 2013 г., 10:50:54 UTC+4 пользователь > debo...@2pirad.comнаписал: >> >> My application uses Google Protocol Buffer with Apache MINA. There are >> clients who connect to the MINA Connector, send request to the server and >> then wait for the response from the server. The flow of control is as >> follows: >> 1. Client sends request >> 2. Request received at MINA filter (<Class> >> extends CumulativeProtocolDecoder) >> 3. doDecode() method is called >> 4. A request object (generated from *.proto file) is created >> using the <RequestObject>.parseFrom(bytes) >> 5. The request is passed on to the IOHandler (<Class> >> extends IoHandlerAdapter) >> 6. messageReceived() method is called >> 7. In this method, the request object (from #7) >> is processed to create the list of requests which has been sent by the >> client. >> * 8. At this point, we have noticed that the time >> taken to process the request object (#7) is gradually degrading with time. >> From a initial period of around 2 ms, the time period is going up to 200 ms >> in just 8 days of continuous usage. And this value gradually increases with >> time.* >> 9. The request list is processed in the >> application >> 10. The response object is created >> 11. The response is sent to the client. >> >> Any suggestion would be highly appreciated. >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/MShFkpi1cmoJ. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
