With a java heap dump you could easily figure out who is holding on to those sockets:
https://dzone.com/articles/memory-analysis-how-to-obtain-java-heat-dump I recommend using Eclipse MAT do analyze the dump: https://www.eclipse.org/mat/ Stefan Den ons. 5. des. 2018 kl. 13:26 skrev Krishan Babbar < kb00449...@techmahindra.com>: > Hi Jonathan, > > I found following counts in /proc/$PID/fd folder. > > Socket: 7807 > Pipe: 76 > anon_inode:[eventpoll]: 38 > /usr/lib/jvm/jdk1.8.0_92/ : 5 > /appl/myapplicationfolder : 6 > > Thanks & Regards, > Krishan Babbar > > > ============================================================================================================================ > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > > =========================================================================================================================== > > -----Original Message----- > From: Jonathan Valliere <john...@apache.org> > Sent: Wednesday, December 5, 2018 4:34 PM > To: users@mina.apache.org > Cc: Atul Kandhari <atul.kandh...@techmahindra.com> > Subject: Re: Getting "Too many open files" warnings11 > > Can you find out the open file by type instead of just the number? How > many are storage files, how many are sockets? > > On Wed, Dec 5, 2018 at 2:10 AM Stefan Magnus Landrø < > stefan.lan...@gmail.com> > wrote: > > > Perform a heap dump to find out what’s going on > > > > Sendt fra min iPhone > > > > > 5. des. 2018 kl. 07:36 skrev Krishan Babbar > > ><kb00449...@techmahindra.com > > >: > > > > > > Any idea why file count is increasing day by day? Can we close any > > socket/file handle manually? > > > > > > Date > > > > > > Time > > > > > > Open files count > > > > > > 2-Dec > > > > > > 16:08 PM > > > > > > 7047 > > > > > > 3-Dec > > > > > > 8:29 AM > > > > > > 7209 > > > > > > 3-Dec > > > > > > 14:45 PM > > > > > > 7289 > > > > > > 3-Dec > > > > > > 16:00 PM > > > > > > 7308 > > > > > > 3-Dec > > > > > > 16:11 PM > > > > > > 7310 > > > > > > 3-Dec > > > > > > 18:01 PM > > > > > > 7330 > > > > > > 3-Dec > > > > > > 20:00 PM > > > > > > 7358 > > > > > > 3-Dec > > > > > > 22:00 PM > > > > > > 7394 > > > > > > 4-Dec > > > > > > 8:25 AM > > > > > > 7501 > > > > > > 4-Dec > > > > > > 11:09 AM > > > > > > 7543 > > > > > > 4-Dec > > > > > > 12:00 PM > > > > > > 7551 > > > > > > 4-Dec > > > > > > 14:00 PM > > > > > > 7594 > > > > > > 4-Dec > > > > > > 16:00 PM > > > > > > 7614 > > > > > > 4-Dec > > > > > > 18:00 PM > > > > > > 7635 > > > > > > 4-Dec > > > > > > 20:00 PM > > > > > > 7668 > > > > > > 4-Dec > > > > > > 22:00 PM > > > > > > 7689 > > > > > > > > > > > > Thanks & Regards, > > > Krishan Babbar > > > > > > > > ====================================================================== > > ====================================================== > > > Disclaimer: This message and the information contained herein is > > proprietary and confidential and subject to the Tech Mahindra policy > > statement, you may review the policy at > > http://www.techmahindra.com/Disclaimer.html externally > > http://tim.techmahindra.com/tim/disclaimer.html internally within > > TechMahindra. > > > > > ====================================================================== > > ===================================================== > > > > > > From: Krishan Babbar > > > Sent: Monday, December 3, 2018 11:27 AM > > > To: users@mina.apache.org > > > Cc: Atul Kandhari <atul.kandh...@techmahindra.com> > > > Subject: RE: Getting "Too many open files" warnings11 > > > > > > > > > Thanks Jonathan, > > > > > > > > > > > > Ok, but I did not see any message like "acceptor was sleeping". > > > > > > > > > > > > I noticed that my open files count for my Java application is keep > > increasing day by day as shown below. Why is it not decreasing again? > > > > > > What configuration am I missing? What can be the reason? > > > > > > > > > Date > > > > > > Time > > > > > > Open file count > > > > > > 30-Nov > > > > > > 20:00 PM > > > > > > 2273 > > > > > > 30-Nov > > > > > > 22:00 PM > > > > > > 2544 > > > > > > 1-Dec > > > > > > 12:00 AM > > > > > > 2790 > > > > > > 1-Dec > > > > > > 2:00 AM > > > > > > 2987 > > > > > > 1-Dec > > > > > > 4:00 AM > > > > > > 3283 > > > > > > 1-Dec > > > > > > 12:00 PM > > > > > > 4828 > > > > > > 1-Dec > > > > > > 14:00 PM > > > > > > 5200 > > > > > > 1-Dec > > > > > > 16:00 PM > > > > > > 5635 > > > > > > 2-Dec > > > > > > 10:17 AM > > > > > > 6992 > > > > > > 2-Dec > > > > > > 12:00 PM > > > > > > 7008 > > > > > > 2-Dec > > > > > > 14:02 PM > > > > > > 7024 > > > > > > 2-Dec > > > > > > 16:08 PM > > > > > > 7047 > > > > > > > > > > > > > > > > > > > > > In my RequestHandler, I have following settings, means setting > > > session > > timeout to 1800 seconds (30 Mins) > > > @Override > > > public void messageReceived(IoSession session, Object message) > > throws Exception { > > > if (session != null) { > > > this.session = session; > > > this.session.getConfig().setBothIdleTime(1800); > > > LOGGER.info("Session Idle time in sec: " + > > this.session.getConfig().getBothIdleTime()); > > > } > > > > > > this.message = message; > > > > > > } > > > > > > > > > > > > Yes, I can simulate multiple devices with lowering nlimit and test. > > > What > > would be the next? > > > > > > > > > > > > Thanks & Regards, > > > > > > Krishan Babbar > > > > > > > > > > > > > > ====================================================================== > > ====================================================== > > > > > > Disclaimer: This message and the information contained herein is > > proprietary and confidential and subject to the Tech Mahindra policy > > statement, you may review the policy at > > http://www.techmahindra.com/Disclaimer.html externally > > http://tim.techmahindra.com/tim/disclaimer.html internally within > > TechMahindra. > > > > > > > > ====================================================================== > > ===================================================== > > > > > > > > > > > > -----Original Message----- > > > From: Jonathan Valliere <jon.valli...@emoten.com<mailto: > > jon.valli...@emoten.com>> > > > Sent: Friday, November 30, 2018 7:20 PM > > > To: users@mina.apache.org<mailto:users@mina.apache.org> > > > Cc: Atul Kandhari <atul.kandh...@techmahindra.com<mailto: > > atul.kandh...@techmahindra.com>> > > > Subject: Re: Getting "Too many open files" warnings11 > > > > > > > > > > > > Mina is patched to prevent that exception from occurring in a cycle. > > > That > > > > > > was a long time ago, your version is patched. You probably noticed a > > > > > > message saying the acceptor was sleeping. > > > > > > > > > > > > Do you have any kind of testing utility to create fake users for > > > your > > application? I would recommend lowering the nlimit and trying to > > reproduce > > > > > > entirely artificially. There shouldn’t be any correlation to > > > > > > > > > > > > On Fri, Nov 30, 2018 at 8:39 AM Krishan Babbar < > > kb00449...@techmahindra.com<mailto:kb00449...@techmahindra.com>> > > > > > > wrote: > > > > > > > > > > > >> Hi Jonathan, > > > > > >> > > > > > >> Version - 2.0.16 > > > > > >> Java Application process was using around 1% of CPU and 3.4% of > Memory. > > > > > >> > > > > > >> After mitigate, were you able to provide the solution? > > > > > >> I mean what would be the solution if you know? > > > > > >> > > > > > >> Thanks & Regards, > > > > > >> Krishan Babbar > > > > > >> > > > > > >> -----Original Message----- > > > > > >> From: Jonathan Valliere <jon.valli...@emoten.com<mailto: > > jon.valli...@emoten.com>> > > > > > >> Sent: Friday, November 30, 2018 6:56 PM > > > > > >> To: users@mina.apache.org<mailto:users@mina.apache.org> > > > > > >> Cc: Atul Kandhari <atul.kandh...@techmahindra.com<mailto: > > atul.kandh...@techmahindra.com>> > > > > > >> Subject: Re: Getting "Too many open files" warnings11 > > > > > >> > > > > > >> What version of Mina are you using? We added code to mitigate this > > > > > >> specific exemption a while back. > > > > > >> > > > > > >> Do you know how much cpu Java was consuming at that point in time? > > > > > >> > > > > > >> On Fri, Nov 30, 2018 at 8:24 AM Krishan Babbar < > > > > > >> kb00449...@techmahindra.com<mailto:kb00449...@techmahindra.com>> > > > > > >> wrote: > > > > > >> > > > > > >>> Thanks Stefan, > > > > > >>> > > > > > >>> It would be great if you could answers other questions too. > > > > > >>> -What do you think, why application stops receiving packets from > > devices? > > > > > >>> Is it only due to exception "Too many open files" or it can be due > > > > > >>> to any other reasons as well? Please guide. > > > > > >>> -Is my MINA configuration fine shown in below Java code? Please > > > > > >>> suggest if I need to fine tune it more. > > > > > >>> > > > > > >>> Thanks & Regards, > > > > > >>> Krishan Babbar > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> ================================================================== > > >>> == > > > > > >>> == ====================================================== > > > > > >>> Disclaimer: This message and the information contained herein is > > > > > >>> proprietary and confidential and subject to the Tech Mahindra > > >>> policy > > > > > >>> statement, you may review the policy at > > > > > >>> http://www.techmahindra.com/Disclaimer.html externally > > > > > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > > > > > >>> TechMahindra. > > > > > >>> > > > > > >>> ================================================================== > > >>> == > > > > > >>> == ===================================================== > > > > > >>> > > > > > >>> -----Original Message----- > > > > > >>> From: Stefan Magnus Landrø <stefan.lan...@gmail.com<mailto: > > stefan.lan...@gmail.com>> > > > > > >>> Sent: Friday, November 30, 2018 5:59 PM > > > > > >>> To: users@mina.apache.org<mailto:users@mina.apache.org> > > > > > >>> Cc: Atul Kandhari <atul.kandh...@techmahindra.com<mailto: > > atul.kandh...@techmahindra.com>> > > > > > >>> Subject: Re: Getting "Too many open files" warnings11 > > > > > >>> > > > > > >>> > > > > > >>> https://unix.stackexchange.com/questions/108603/do-changes-in-etc- > > >>> se > > > > > >>> cu > > > > > >>> rity-limits-conf-require-a-reboot > > > > > >>> > > > > > >>> > > > > > >>> Sendt fra min iPhone > > > > > >>> > > > > > >>>> 30. nov. 2018 kl. 11:17 skrev Krishan Babbar > > > > > >>>> <kb00449...@techmahindra.com > > > > > >>>> : > > > > > >>>> > > > > > >>>> Dear All, > > > > > >>>> > > > > > >>>> I am working on an IoT project. I have created a Java application > > > > > >>>> using > > > > > >>> MINA library. All the devices are connecting to this Java > > > > > >>> application using TCP protocol. I am getting around 3000 packets > > >>> per > > > > > >>> hour from about 50 devices as of now and it will increase soon. > > > > > >>>> > > > > > >>>> I am facing one issue on production environment. After running > > >>>> say > > > > > >>>> 1-2 > > > > > >>> days, my application stops receiving packets from all the devices > > > > > >>> but application process keep running. To make it working again, I > > > > > >>> need to restart my Java application. I noticed following warnings > > >>> in > > logs. > > > > > >>>> Warnings > > > > > >>>> [NioSocketAcceptor-3] WARN > > > > > >>>> org.apache.mina.util.DefaultExceptionMonitor > > > > > >>> 30/11/2018 05:25:40 - Unexpected exception. > > > > > >>>> java.io.IOException: Too many open files > > > > > >>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native > > >>>> Method) > > > > > >>>> at sun.nio.ch > > > > > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > > > > > >>>> at sun.nio.ch > > > > > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > > > > > >>>> at > > > > > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioS > > >>> oc > > > > > >>> ke > > > > > >>> tAcceptor.java:194) > > > > > >>>> at > > > > > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioS > > >>> oc > > > > > >>> ke > > > > > >>> tAcceptor.java:51) > > > > > >>>> at > > > > > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.pr > > >>> oc > > > > > >>> es > > > > > >>> sHandles(AbstractPollingIoAcceptor.java:544) > > > > > >>>> at > > > > > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.ru > > >>> n( > > > > > >>> Ab > > > > > >>> stractPollingIoAcceptor.java:484) > > > > > >>>> at > > > > > >>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunn > > >>> ab > > > > > >>> le > > > > > >>> .java:64) > > > > > >>>> at > > > > > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut > > >>> or > > > > > >>> .j > > > > > >>> ava:1142) > > > > > >>>> at > > > > > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > > > >>> java:617) > > > > > >>>> at java.lang.Thread.run(Thread.java:745) > > > > > >>>> > > > > > >>>> Due to above warnings in logs, yesterday, after doing some > > >>>> Google, > > > > > >>>> I > > > > > >>> changed file limits in "/etc/security/limits.conf" file to 100000 > > >>> as > > > > > >>> shown below, but I did not restart my Java application. > > > > > >>>> * hard nofile 100000 > > > > > >>>> * soft nofile 100000 > > > > > >>>> > > > > > >>>> Today. I got the same warnings again and I was not getting any > > > > > >>>> packet > > > > > >>> from any of the devices. So I restarted my Java application and > > > > > >>> devices started sending packets again. > > > > > >>>> Now, my questions are > > > > > >>>> > > > > > >>>> - After application stopped receiving packets, I restarted > my > > > > > >>> Java application today. Will it consider new file limits i.e. > > >>> 100000 > > > > > >>> and not give Exceptions again? Or do I need to reboot machine as > well? > > > > > >>>> > > > > > >>>> - What do you think, why application stops receiving > packets > > > > > >>> from devices? Is it only due to exception "Too many open files" or > > > > > >>> it can be due to any other reasons as well? Please guide. > > > > > >>>> > > > > > >>>> - Is my MINA configuration fine shown in below Java code? > > > > > >>> Please suggest if I need to fine tune it more. > > > > > >>>> > > > > > >>>> > > > > > >>>> Below is given our code for MINA configuration. > > > > > >>>> import java.io.IOException; > > > > > >>>> import java.net.InetSocketAddress; import > > >>>> org.apache.log4j.Logger; > > > > > >>>> import org.apache.mina.core.service.AbstractIoAcceptor; > > > > > >>>> import org.apache.mina.core.session.IdleStatus; > > > > > >>>> import org.apache.mina.filter.logging.LoggingFilter; > > > > > >>>> import org.apache.mina.transport.socket.nio.NioDatagramAcceptor; > > > > > >>>> > > > > > >>>> import com.techm.tnt.protocol.gateway.util.ServerConfigLoader; > > > > > >>>> > > > > > >>>> /** > > > > > >>>> * The abstract class for tcp/ip and udp listeners. > > > > > >>>> */ > > > > > >>>> public abstract class AbstractListener { > > > > > >>>> > > > > > >>>> /** The logger instance.*/ > > > > > >>>> private static Logger LOGGER = > > > > > >>>> Logger.getLogger(AbstractListener.class); > > > > > >>>> > > > > > >>>> /** The buffer size constant. */ > > > > > >>>> protected static final int BUFFER_SIZE = 2048; > > > > > >>>> > > > > > >>>> /** > > > > > >>>> * The method to initialize udp and tcp/ip listeners. > > > > > >>>> * > > > > > >>>> * @param portKey > > > > > >>>> * the port of the udp or tcp/ip listener. > > > > > >>>> * @throws IOException > > > > > >>>> * the io exception to be thrown in case of > > > > > >>> error. > > > > > >>>> */ > > > > > >>>> public void init(String portKey) throws IOException > > >>>> { > > > > > >>>> LOGGER.info("Entering init()"); > > > > > >>>> AbstractIoAcceptor acceptor = > > > > > >>> initInternal(); > > > > > >>>> acceptor.setHandler(new > > > > > >>>> RequestHandler()); > > > > > >>>> > > > > > >>> acceptor.getFilterChain().addLast("logger", new LoggingFilter()); > > > > > >>>> > > > > > >>> acceptor.getSessionConfig().setReadBufferSize(BUFFER_SIZE); > > > > > >>>> > > > > > >>> acceptor.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10); > > > > > >>>> if (acceptor instanceof > > > > > >>> NioDatagramAcceptor) { > > > > > >>>> > > > > > >>>> ((NioDatagramAcceptor) > > > > > >>> acceptor).getSessionConfig().setReuseAddress(true); > > > > > >>>> } > > > > > >>>> acceptor.bind(new > > > > > >>> InetSocketAddress(Integer.parseInt(ServerConfigLoader.getInstance(). > > > > > >>> ge > > > > > >>> tProperty(portKey)))); > > > > > >>>> LOGGER.info("Exiting init()"); > > > > > >>>> } > > > > > >>>> > > > > > >>>> /** > > > > > >>>> * The abstract method to be over ridden by > > > > > >>>> appropriate > > > > > >>> listener (udp / > > > > > >>>> * tcp/ip) for invoking appropriate listener. > > > > > >>>> * > > > > > >>>> * @return the appropriate (udp/ tcp/ip) io acceptor. > > > > > >>>> */ > > > > > >>>> public abstract AbstractIoAcceptor initInternal(); > > > > > >>>> } > > > > > >>>> > > > > > >>>> Appreciate all tour help. > > > > > >>>> > > > > > >>>> Thanks & Regards, > > > > > >>>> Krishan Babbar > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> ================================================================= > > >>>> = > > > > > >>>> == == ====================================================== > > > > > >>>> Disclaimer: This message and the information contained herein is > > > > > >>> proprietary and confidential and subject to the Tech Mahindra > > >>> policy > > > > > >>> statement, you may review the policy at > > > > > >>> http://www.techmahindra.com/Disclaimer.html externally > > > > > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > > > > > >>> TechMahindra. > > > > > >>>> ================================================================= > > >>>> = > > > > > >>>> == == ===================================================== > > > > > >>>> > > > > > >>>> ================================================================= > > >>>> = > > > > > >>>> == == ====================================================== > > > > > >>>> > > > > > >>>> Disclaimer: This message and the information contained herein is > > > > > >>> proprietary and confidential and subject to the Tech Mahindra > > >>> policy > > > > > >>> statement, you may review the policy at > > > > > >>> http://www.techmahindra.com/Disclaimer.html < > > > > > >>> http://www.techmahindra.com/Disclaimer.html> externally > > > > > >>> http://tim.techmahindra.com/tim/disclaimer.html < > > > > > >>> http://tim.techmahindra.com/tim/disclaimer.html> internally within > > > > > >>> TechMahindra. > > > > > >>>> > > > > > >>>> ================================================================= > > >>>> = > > > > > >>>> == == ====================================================== > > > > > >>> > > > > > >> -- > > > > > >> > > > > > >> CONFIDENTIALITY NOTICE: The contents of this email message and any > > > > > >> attachments are intended solely for the addressee(s) and may > > >> contain > > > > > >> confidential and/or privileged information and may be legally > > > > > >> protected from disclosure. > > > > > >> > > > > > >> =================================================================== > > >> === > > > > > >> ====================================================== > > > > > >> > > > > > >> Disclaimer: This message and the information contained herein is > > > > > >> proprietary and confidential and subject to the Tech Mahindra > > >> policy > > > > > >> statement, you may review the policy at > > > > > >> http://www.techmahindra.com/Disclaimer.html < > > > > > >> http://www.techmahindra.com/Disclaimer.html> externally > > > > > >> http://tim.techmahindra.com/tim/disclaimer.html < > > > > > >> http://tim.techmahindra.com/tim/disclaimer.html> internally within > > > > > >> TechMahindra. > > > > > >> > > > > > >> > > > > > >> =================================================================== > > >> === > > > > > >> ====================================================== > > > > > >> > > > > > > -- > > > > > > > > > > > > CONFIDENTIALITY NOTICE: The contents of this email message and any > > attachments are intended solely for the addressee(s) and may contain > > confidential and/or privileged information and may be legally > > protected from disclosure. > > > > > ====================================================================== > > ====================================================== > > > > > > Disclaimer: This message and the information contained herein is > > proprietary and confidential and subject to the Tech Mahindra policy > > statement, you may review the policy at > > http://www.techmahindra.com/Disclaimer.html < > > http://www.techmahindra.com/Disclaimer.html> externally > > http://tim.techmahindra.com/tim/disclaimer.html < > > http://tim.techmahindra.com/tim/disclaimer.html> internally within > > TechMahindra. > > > > > > > > ====================================================================== > > ====================================================== > > > > ============================================================================================================================ > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html < > http://www.techmahindra.com/Disclaimer.html> externally > http://tim.techmahindra.com/tim/disclaimer.html < > http://tim.techmahindra.com/tim/disclaimer.html> internally within > TechMahindra. > > > ============================================================================================================================ > -- BEKK Open http://open.bekk.no TesTcl - a unit test framework for iRules http://testcl.com
