Have you profiled the garbage collector and heap memory while this happens?
On Mon, Jul 5, 2021 at 7:35 AM Nitin Phuria <nit...@integramicro.com.invalid>
wrote:
> Dear All,
>
>
>
> We have Server developed using MINA 2.0.9 where it connects
> to two other systems say System-A and System-B.
>
>
>
> The connectivity with both these system is persistence which means our
> server connects and create only single session with each of these systems
>
> and on the same session multiple message exchanges (Request/Response)
> happens.
>
>
>
> The current flow of message is as below:
>
> Once we connect with System-B and have persistence connection (single
> session), System-B will send us message which we have to process in our
> server and then send processed message to System-A then get response from
> System-A and then pass it back to System-B
>
>
>
> We have provision that if load increases we can have multiple instances of
> System-A and our server can establish persistence connection (single
> session) with each of these System-A instances and manage the load.
>
>
>
> Below is the code for establishing the Connectivity with System-B and it is
> persistence connection (Single connection)
>
>
>
> SocketAddress socketAddress = new
> InetSocketAddress(this.hostName,this.port);
>
> NioSocketConnector connector = new NioSocketConnector();
>
> connector.setConnectTimeoutMillis(30 * 1000L);
>
> connector.getSessionConfig().setTcpNoDelay(true);
>
>
> connector.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE,this.echoTime)
> ;
>
> this.connector.setHandler(systemBHandler); // systemB Handler is our
> IoHandler implementation for System-B
>
> IoFilter codecFilter = new
> ProtocolCodecFilter((ProtocolCodecFactory)systemBProcFactory);// systemB
> ProcFactory is our Codec Implementation
>
> DefaultIoFilterChainBuilder chain = this.connector.getFilterChain();
>
> chain.clear();
>
> chain.addLast("codec", codecFilter);
>
>
>
> With above when we connect to System B we get session object on which we
> exchange multiple req/res messages.
>
>
>
> We have added logs for the message time tracking as below
>
> - when we write message to session via session.write we log the message id
> (internal to our system) in logger with timestamp
>
> - When the message is sent the messageSent event will be triggered so we
> log
> the timestamp in that event along with message id (internal to our system)
>
> We have seen that the time gap between the session.write and messageSent
> event is mostly in few milliseconds (1 to 5 milliseconds)
>
> But Sometimes we have seen that this gap is increasing to almost 10-12
> seconds which is surprising
>
>
>
>
> #
>
> Time Stamp Details
>
> Sample Message Id
>
>
> 118320938392 (normal behaviour)
>
> 118320743879 (abnormal behaviour)
>
>
> 1
>
> Received message from System B
>
> 2021-07-02 20:31:59,917
>
> 2021-06-30 10:57:10,233
>
>
> 2
>
> Message sent to System A
>
> 2021-07-02 20:31:59,917
>
> 2021-06-30 10:57:10,328
>
>
> 3
>
> Received message from System A
>
> 2021-07-02 20:32:00,005
>
> 2021-06-30 10:57:10,426
>
>
> 4
>
> Message written to System B session (Session.write method)
>
> 2021-07-02 20:32:00,006
>
> 2021-06-30 10:57:10,427
>
>
> 5
>
> Message sent to System B (messageSent event)
>
> 2021-07-02 20:32:00,006
>
> 2021-06-30 10:57:21,749
>
>
>
> Can someone provide more details on why this is happening. This is not a
> regular behaviour but suddenly one bad day it happens and we are clueless
> to
> identify the root-cause. After session.write it has to go to Codec encoder
> which is our implementation as below and when the actual message is written
> to tcp/ip the messageSent event will be triggered by Mina framework. So
> what
> is going wrong and why it is taking so long to write to TCP/IP is driving
> me
> crazy. It will be good to get some clarity or soem guideline to trace the
> issue.
>
>
>
> public void encode(IoSession session, Object msg, ProtocolEncoderOutput
> out)
> throws Exception
>
> {
>
> if(msg instanceof AppMsg)
>
> {
>
> AppMsg appMsg = (AppMsg) msg;
>
> IoBuffer buffer =
>
> IoBuffer.allocate(appMsg.getLengthBytes().length+appMsg.getMsgBytes().length
> );
>
> buffer.put(appMsg.getLengthBytes());
>
> buffer.put(appMsg.getMsgBytes());
>
> buffer.flip();
>
> out.write(buffer);
>
> }
>
> else if(msg instanceof NullMessage)
>
> {
>
> NullMessage m=(NullMessage)msg;
>
> final byte[] bytes = m.getBytes();
>
> IoBuffer buffer=
> IoBuffer.allocate(bytes.length);
>
> buffer.put(bytes);
>
> buffer.flip();
>
> out.write(buffer);
>
> }
>
> }
>
>
>
> Currently we have our Mina Based Application running on server having 16
> Core with 64 GB RAM. Every day we exchange almost 1 to 1.2 million messages
> via our application in 4 hours time frame.
>
>
>
> Thanks And Regards,
>
> Nitin Phuria
>
>
>
> Confidentiality Disclaimer: "The information contained in this electronic
> message (email) and any attachments to this email are intended for the
> exclusive use of the addressee(s) and access to this email by anyone else
> is
> unauthorized. The email may contain proprietary, confidential or privileged
> information or information relating to Integra Group. If you are not the
> intended recipient, please notify the sender by telephone, fax, or return
> email and delete this communication and any attachments thereto,
> immediately
> from your computer. Any dissemination, distribution, or copying of this
> communication and the attachments thereto (in whole or part), in any
> manner,
> is strictly prohibited and actionable at law. The recipient acknowledges
> that emails are susceptible to alteration and their integrity cannot be
> guaranteed and that Company does not guarantee that any e-mail is
> virus-free
> and accept no liability for any damage caused by any virus transmitted by
> this email."
>
>
>
>
> --
> *Confidentiality Disclaimer**: "The information contained in this
> electronic message
> (email) and any attachments to this email are intended
> for the exclusive use of
> the addressee(s) and access to this email by
> anyone else is unauthorized. The
> email may contain proprietary,
> confidential or privileged information or
> information relating to Integra
> Group. If you are not the intended recipient,
> please notify the sender by
> telephone, fax, or return email and delete this
> communication and any
> attachments thereto, immediately from your computer. Any
> dissemination,
> distribution, or copying of this communication and the
> attachments thereto
> (in whole or part), in any manner, is strictly prohibited
> and actionable at
> law. The recipient acknowledges that emails are susceptible
> to alteration
> and their integrity cannot be guaranteed and that Company does
> not
> guarantee that any e-mail is virus-free and accept no liability for any
> damage caused by any virus transmitted by this email."*
>
