Dear Jonathan, Is there anything that I can print in log which can give us some more clue for the issue mentioned. As I have mentioned that connectivity is on persistence connection which means I have only single socket connected so any internal statistics on session can be printed. Is it possible that read are given more priority than write?
Thanks And Regards, Nitin Phuria Confidentiality Disclaimer: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.” -----Original Message----- From: Jonathan Valliere [mailto:john...@apache.org] Sent: 05 July 2021 18:38 To: Nitin Phuria Cc: Jonathan Valliere; Users Subject: Re: delay in session.write and messageSent event for same message All writes to the actual OS socket are delayed by reading from other sockets attached to the same IoProcessor. Here is an example: 1. the event poller wakes up and has events for 20 sockets. 2. Socket A - one of those events is a read, your socket reads and processes the data 3. Socket A - you write to IoSession.write 4. Socket A - the WriteRequest is copied to the session write queue 5. Socket A - the IoSession subscribes to the write event from the poller 6. the remaining events for the 19 other sockets are with events are processed 7. the IoProcessor selects new events from the poller 8. 20 new events are returned 9. assuming Socket A is not backlogged, it should appear in the event list as writable but the order is random 10. all 20 events must be processed until Socket A is found and the enqueued WriteRequest is written to the underlying socket On Mon, Jul 5, 2021 at 8:57 AM Nitin Phuria <nit...@integramicro.com> wrote: > Dear Jonathan, > > > > The issue is not persistent and when it comes it is not > for a moment. When it comes the issue will be for 2-3 hours and we see the > time gap only in the session.write and messageSent event and that to with > System B only. If at all it has to be linked with any memory management it > should not affect only this part alone. I need some info on how I can get > some internal details to find out why their is delay happening. Is there > way to print some internal queue details and when MINA framework will be > writting to actual TCP/IP etc.. > > > > But to answer your question we did the heap memory analysis and we didn't > find anything in it. > > > > Message written to System B session (Session.write method) > > 2021-06-30 10:57:10,427 > > Message sent to System B (messageSent event) > > 2021-06-30 10:57:21,749 > > > > Thanks And Regards, > > Nitin Phuria > > > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* > > > > *From:* Jonathan Valliere [mailto:john...@apache.org] > *Sent:* 05 July 2021 18:15 > *To:* nit...@integramicro.com > *Cc:* users@mina.apache.org > *Subject:* Re: delay in session.write and messageSent event for same > message > > > > Have you profiled the garbage collector and heap memory while this happens? > > > > On Mon, Jul 5, 2021 at 7:35 AM Nitin Phuria > <nit...@integramicro.com.invalid> wrote: > > Dear All, > > > > We have Server developed using MINA 2.0.9 where it connects > to two other systems say System-A and System-B. > > > > The connectivity with both these system is persistence which means our > server connects and create only single session with each of these systems > > and on the same session multiple message exchanges (Request/Response) > happens. > > > > The current flow of message is as below: > > Once we connect with System-B and have persistence connection (single > session), System-B will send us message which we have to process in our > server and then send processed message to System-A then get response from > System-A and then pass it back to System-B > > > > We have provision that if load increases we can have multiple instances of > System-A and our server can establish persistence connection (single > session) with each of these System-A instances and manage the load. > > > > Below is the code for establishing the Connectivity with System-B and it is > persistence connection (Single connection) > > > > SocketAddress socketAddress = new > InetSocketAddress(this.hostName,this.port); > > NioSocketConnector connector = new NioSocketConnector(); > > connector.setConnectTimeoutMillis(30 * 1000L); > > connector.getSessionConfig().setTcpNoDelay(true); > > > connector.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE,this.echoTime) > ; > > this.connector.setHandler(systemBHandler); // systemB Handler is our > IoHandler implementation for System-B > > IoFilter codecFilter = new > ProtocolCodecFilter((ProtocolCodecFactory)systemBProcFactory);// systemB > ProcFactory is our Codec Implementation > > DefaultIoFilterChainBuilder chain = this.connector.getFilterChain(); > > chain.clear(); > > chain.addLast("codec", codecFilter); > > > > With above when we connect to System B we get session object on which we > exchange multiple req/res messages. > > > > We have added logs for the message time tracking as below > > - when we write message to session via session.write we log the message id > (internal to our system) in logger with timestamp > > - When the message is sent the messageSent event will be triggered so we > log > the timestamp in that event along with message id (internal to our system) > > We have seen that the time gap between the session.write and messageSent > event is mostly in few milliseconds (1 to 5 milliseconds) > > But Sometimes we have seen that this gap is increasing to almost 10-12 > seconds which is surprising > > > > > # > > Time Stamp Details > > Sample Message Id > > > 118320938392 (normal behaviour) > > 118320743879 (abnormal behaviour) > > > 1 > > Received message from System B > > 2021-07-02 20:31:59,917 > > 2021-06-30 10:57:10,233 > > > 2 > > Message sent to System A > > 2021-07-02 20:31:59,917 > > 2021-06-30 10:57:10,328 > > > 3 > > Received message from System A > > 2021-07-02 20:32:00,005 > > 2021-06-30 10:57:10,426 > > > 4 > > Message written to System B session (Session.write method) > > 2021-07-02 20:32:00,006 > > 2021-06-30 10:57:10,427 > > > 5 > > Message sent to System B (messageSent event) > > 2021-07-02 20:32:00,006 > > 2021-06-30 10:57:21,749 > > > > Can someone provide more details on why this is happening. This is not a > regular behaviour but suddenly one bad day it happens and we are clueless > to > identify the root-cause. After session.write it has to go to Codec encoder > which is our implementation as below and when the actual message is written > to tcp/ip the messageSent event will be triggered by Mina framework. So > what > is going wrong and why it is taking so long to write to TCP/IP is driving > me > crazy. It will be good to get some clarity or soem guideline to trace the > issue. > > > > public void encode(IoSession session, Object msg, ProtocolEncoderOutput > out) > throws Exception > > { > > if(msg instanceof AppMsg) > > { > > AppMsg appMsg = (AppMsg) msg; > > IoBuffer buffer = > > IoBuffer.allocate(appMsg.getLengthBytes().length+appMsg.getMsgBytes().length > ); > > buffer.put(appMsg.getLengthBytes()); > > buffer.put(appMsg.getMsgBytes()); > > buffer.flip(); > > out.write(buffer); > > } > > else if(msg instanceof NullMessage) > > { > > NullMessage m=(NullMessage)msg; > > final byte[] bytes = m.getBytes(); > > IoBuffer buffer= > IoBuffer.allocate(bytes.length); > > buffer.put(bytes); > > buffer.flip(); > > out.write(buffer); > > } > > } > > > > Currently we have our Mina Based Application running on server having 16 > Core with 64 GB RAM. Every day we exchange almost 1 to 1.2 million messages > via our application in 4 hours time frame. > > > > Thanks And Regards, > > Nitin Phuria > > > > Confidentiality Disclaimer: "The information contained in this electronic > message (email) and any attachments to this email are intended for the > exclusive use of the addressee(s) and access to this email by anyone else > is > unauthorized. The email may contain proprietary, confidential or privileged > information or information relating to Integra Group. If you are not the > intended recipient, please notify the sender by telephone, fax, or return > email and delete this communication and any attachments thereto, > immediately > from your computer. Any dissemination, distribution, or copying of this > communication and the attachments thereto (in whole or part), in any > manner, > is strictly prohibited and actionable at law. The recipient acknowledges > that emails are susceptible to alteration and their integrity cannot be > guaranteed and that Company does not guarantee that any e-mail is > virus-free > and accept no liability for any damage caused by any virus transmitted by > this email." > > > > > -- > *Confidentiality Disclaimer**: "The information contained in this > electronic message > (email) and any attachments to this email are intended > for the exclusive use of > the addressee(s) and access to this email by > anyone else is unauthorized. The > email may contain proprietary, > confidential or privileged information or > information relating to Integra > Group. If you are not the intended recipient, > please notify the sender by > telephone, fax, or return email and delete this > communication and any > attachments thereto, immediately from your computer. Any > dissemination, > distribution, or copying of this communication and the > attachments thereto > (in whole or part), in any manner, is strictly prohibited > and actionable at > law. The recipient acknowledges that emails are susceptible > to alteration > and their integrity cannot be guaranteed and that Company does > not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email."* > > > *Confidentiality Disclaimer**: “The information contained in this > electronic message (email) and any attachments to this email are intended > for the exclusive use of the addressee(s) and access to this email by > anyone else is unauthorized. The email may contain proprietary, > confidential or privileged information or information relating to Integra > Group. If you are not the intended recipient, please notify the sender by > telephone, fax, or return email and delete this communication and any > attachments thereto, immediately from your computer. Any dissemination, > distribution, or copying of this communication and the attachments thereto > (in whole or part), in any manner, is strictly prohibited and actionable at > law. The recipient acknowledges that emails are susceptible to alteration > and their integrity cannot be guaranteed and that Company does not > guarantee that any e-mail is virus-free and accept no liability for any > damage caused by any virus transmitted by this email.”* -- *Confidentiality Disclaimer**: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.”* --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@mina.apache.org For additional commands, e-mail: users-h...@mina.apache.org
