Dear Jonathan,
Thanks for bringing more clarity on Processor Thread and it's relation with IoSession. We are also suspecting the network connection and in a crisis one should pull all the strings to check where you get the lead to identify the problem and that's what we are doing. Thanks a tone for all your replies and giving very useful information. We will check further on network connection. Thanks And Regards, Nitin Phuria Confidentiality Disclaimer: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.” From: Jonathan Valliere [mailto:jon.valli...@emoten.com] Sent: 09 April 2020 20:03 To: Nitin Phuria Cc: users@mina.apache.org Subject: Re: MINA: ExecutorFilter on NioSocketConnector Every IoSession is bound to one Processor Thread. So only by creating more IoSessions can you take advantage of multiple Processor Threads. ExecutorFilter only works for received messages. So it generally does nothing on the client side however it will cause the received messages to be executed in multiple threads. The purpose of the ExecutorFilter is to do potentially blocking tasks in there so as not to block the Processor Thread from handling tasks from other IoSessions. Since you’re seeing a huge jump from 15ms to several minutes and a very small number of messages it is most likely a problem with your network connection. On Thu, Apr 9, 2020 at 10:26 AM Nitin Phuria <nit...@integramicro.com> wrote: Dear Jonathan, Our Server and System-B are on WAN network with leased line. One more clarification required As per documentation NioSocketConnector <http://mina.apache.org/mina-project/gen-docs/2.0.10/apidocs/org/apache/mina/transport/socket/nio/NioSocketConnector.html#NioSocketConnector--> () Constructor for <http://mina.apache.org/mina-project/gen-docs/2.0.10/apidocs/org/apache/mina/transport/socket/nio/NioSocketConnector.html> NioSocketConnector with default configuration (multiple thread model). So when we create NioSocketConnector connector = new NioSocketConnector(); I expect it to create n+1 processor where n is number of core on the hardware. In our case it is 17 (16 core machine) When we took threaddump of our Server process I could see only one thread for this connector and in the log (log4j) when we print the thread name it prints NioProcessor-x where x is some number. So does it means that n+1 processors use single thread to work. Also you mentioned that The ExecutorFilter only works for the Server receiver side. http://apache-mina.10907.n7.nabble.com/OrderedThreadPoolExecutor-shutdown-problem-in-MINA-2-0-0-M4-td34417.html But I could see from this above URL that we can add it to Connector also and it says "My understanding is that the Executor filter just adds an executor in the chain which will be used to spread the load on many threads. This is an optimization, rather than something you need to use. If you don't use it, your program will work fine." Could you provide more insight and your thoughts. Thanks And Regards, Nitin Phuria Confidentiality Disclaimer: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.” From: Jonathan Valliere [mailto:jon.valli...@emoten.com] Sent: 09 April 2020 19:34 To: Nitin Phuria Cc: users@mina.apache.org Subject: Re: MINA: ExecutorFilter on NioSocketConnector How far are the servers? TCP Packet loss could explain this. You can get the Queue size from the IoSession object. On Thu, Apr 9, 2020 at 10:02 AM Nitin Phuria <nit...@integramicro.com> wrote: Dear Jonathan, Anytime we write message to System-B session with session.write method the messageSent event has to be generated with it actually gets written on TCP/IP org.apache.mina.core.session.IoSession.write(Object) We see that there is time difference of 10-14 seconds between timestamp of session.write for particular message and messageSent event getting generated for the same message. In non-peak load this time difference is in 10-15 milliseconds and in peak load (60-70 messages per second) it goes till 10-14 seconds. Is there anything that we can log to see what is happening or see the queue size etc.. for further better understaning. Thanks And Regards, Nitin Phuria Confidentiality Disclaimer: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.” From: Jonathan Valliere [mailto:jon.valli...@emoten.com] Sent: 09 April 2020 18:31 To: Nitin Phuria Cc: Nitin Phuria; users@mina.apache.org Subject: Re: MINA: ExecutorFilter on NioSocketConnector The ExecutorFilter only works for the Server receiver side. When you say the responses are slow, what do you consider slow? All writes are put into a queue dedicated for each session. The Session is then enabled for writing by adding the write interest to the poll mechanism. Reads and Writes IO happen in the same thread. After the Session finishes reading it will then call the poll mechanism for new events to process. Hopefully it should get the write event for your Session then begin to flush the queue. Processor Thread Loop —> Select Poll Events —> Process Events (read/write) —> Loop On Thu, Apr 9, 2020 at 8:53 AM Nitin Phuria <nit...@integramicro.com> wrote: Dear Jonathan, Thanks for the reply, can you provide more details on the "All writes are queued up and scheduled later using the poll/kqueue system." How this poll & kqueue system works? Do we have any control on this from MINA framework? We are not facing any problem with the higher CPU consumption but we see that when there is high load from System-B to our server the reads from System-B and write back to System-B on the single persistence session is becoming slow. So we were looking at i/o operations and how to make them fast. With your explanation provided in below email about ExecutorFilter, If we add it in our connector after codec filter will help us reduce the I/O overhead on reactor/processor thread. I hope my understanding is right kindly confirm? Thanks And Regards, Nitin Phuria Confidentiality Disclaimer: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.” -----Original Message----- From: Jonathan Valliere [mailto:jon.valli...@emoten.com] Sent: 09 April 2020 18:09 To: Nitin Phuria Cc: users@mina.apache.org Subject: Re: MINA: ExecutorFilter on NioSocketConnector Mina is a queue and flush framework. All writes are queued up and scheduled later using the poll/kqueue system. The ExecutorFilter uses threading to process incoming messages using a pool of threads instead of the reactor/processor thread. This frees the processor thread to do IO. Have you profiled the application to figure out which code is consuming the CPU? On Thu, Apr 9, 2020 at 3:39 AM Nitin Phuria <nit...@integramicro.com.invalid> wrote: > Dear All, > > > > We have Server developed using MINA 2.0.16 where it > connects > to two other systems say System-A and System-B. > > > > The connectivity with both these system is persistence which means our > server connects and create only single session with each of these systems > and on the same session multiple message exchanges (Request/Response) > happens. > > > > The current flow of message is as below: > > Once we connect with System-B and have persistence connection (single > session), System-B will send us message which we have to process in our > server and then send processed message to System-A then get response from > System-A and then pass it back to System-B > > > > We have provision that if load increases we can have multiple instances of > System-A and our server can establish persistence connection (single > session) with each of these System-A instances and manage the load. > > > > We are facing problem with load coming from System-B to our server as it is > coming on single session and we could see that the read/write on the > session > are slowed down and lot of queue build-up is happening for read/write. > > > > Below is the code for establishing the Connectivity with System-B where we > have not added the ExecutorFilter > > > > SocketAddress socketAddress = new InetSocketAddress(this.hostName, > this.port); > > NioSocketConnector connector = new NioSocketConnector(); > > connector.setConnectTimeoutMillis(30 * 1000L); > > connector.getSessionConfig().setTcpNoDelay(true); > > > connector.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE,this.echoTime) > ; > > this.connector.setHandler(systemBHandler); // systemBHandler is our > IoHandler implementationfor System-B > > IoFilter codecFilter = new > ProtocolCodecFilter((ProtocolCodecFactory)systemBProcFactory);// > systemBProcFactory is our Codec Implementation > > DefaultIoFilterChainBuilder chain = this.connector.getFilterChain(); > > chain.clear(); > > chain.addLast("codec", codecFilter); > > > > https://cwiki.apache.org/confluence/display/MINA/Configuring+Thread+Model > > > From this link we could understand as below > > > > If We didn't add any ExecutorFilter events are forwarded to an IoHandler > via > method invocations. This means your business logic in your IoHandler > implementation will run in the I/O processor thread. We call this thread > model a 'single thread model'. The single thread model is known to be > adequate for low-latency network applications with CPU-bound business logic > (e.g. game servers). > > > > Do we need to add the ExecutorFilter for above mentioned NioSocketConnector > to System-B so that I/O processor thread will only do the I/O job and all > other business logic will be given to worker threads in ExecutorFilter. > > > > Is there any way to increase the number of threads for my I/O to do the > read/write as we have restriction to create only one session with System-B. > > > > If we add ExecutorFilter with IoEventType for read/write then will that > make the read/write faster with System-B on same single session. > > > > Or we add ExecutorFilter with default IoEventType (All Events) will that > make the read/write faster with System-B on same single session. > > > > Pls help us understand the working on ExecutorFilter in above scenario > where > NioSocketConnector has only single session to do all the read/write with > huge load of almost 60-70 messages per second with average message size of > 3-4KiloBytes. > > > > Currently we have server having 16 Core with 64 GB RAM. > > > > Thanks And Regards, > > Nitin Phuria > > > > Confidentiality Disclaimer: "The information contained in this electronic > message (email) and any attachments to this email are intended for the > exclusive use of the addressee(s) and access to this email by anyone else > is > unauthorized. The email may contain proprietary, confidential or privileged > information or information relating to Integra Group. If you are not the > intended recipient, please notify the sender by telephone, fax, or return > email and delete this communication and any attachments thereto, > immediately > from your computer. Any dissemination, distribution, or copying of this > communication and the attachments thereto (in whole or part), in any > manner, > is strictly prohibited and actionable at law. The recipient acknowledges > that emails are susceptible to alteration and their integrity cannot be > guaranteed and that Company does not guarantee that any e-mail is > virus-free > and accept no liability for any damage caused by any virus transmitted by > this email." > > > -- > *Confidentiality Disclaimer**: "The information contained in this > electronic message > (email) and any attachments to this email are intended > for the exclusive use of > the addressee(s) and access to this email by > anyone else is unauthorized. The > email may contain proprietary, > confidential or privileged information or > information relating to Integra > Group. If you are not the intended recipient, > please notify the sender by > telephone, fax, or return email and delete this > communication and any > attachments thereto, immediately from your computer. Any > dissemination, > distribution, or copying of this communication and the > attachments thereto > (in whole or part), in any manner, is strictly prohibited > and actionable at > law. The recipient acknowledges that emails are susceptible > to alteration > and their integrity cannot be guaranteed and that Company does > not > guarantee that any e-mail is virus-free and accept no liability for any > > damage caused by any virus transmitted by this email."* > -- CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. -- *Confidentiality Disclaimer**: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.”* -- CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. Confidentiality Disclaimer: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.” -- CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. Confidentiality Disclaimer: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.” -- CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. -- *Confidentiality Disclaimer**: “The information contained in this electronic message (email) and any attachments to this email are intended for the exclusive use of the addressee(s) and access to this email by anyone else is unauthorized. The email may contain proprietary, confidential or privileged information or information relating to Integra Group. If you are not the intended recipient, please notify the sender by telephone, fax, or return email and delete this communication and any attachments thereto, immediately from your computer. Any dissemination, distribution, or copying of this communication and the attachments thereto (in whole or part), in any manner, is strictly prohibited and actionable at law. The recipient acknowledges that emails are susceptible to alteration and their integrity cannot be guaranteed and that Company does not guarantee that any e-mail is virus-free and accept no liability for any damage caused by any virus transmitted by this email.”*