Can you find out the open file by type instead of just the number? How many are storage files, how many are sockets?
On Wed, Dec 5, 2018 at 2:10 AM Stefan Magnus Landrø <[email protected]> wrote: > Perform a heap dump to find out what’s going on > > Sendt fra min iPhone > > > 5. des. 2018 kl. 07:36 skrev Krishan Babbar <[email protected] > >: > > > > Any idea why file count is increasing day by day? Can we close any > socket/file handle manually? > > > > Date > > > > Time > > > > Open files count > > > > 2-Dec > > > > 16:08 PM > > > > 7047 > > > > 3-Dec > > > > 8:29 AM > > > > 7209 > > > > 3-Dec > > > > 14:45 PM > > > > 7289 > > > > 3-Dec > > > > 16:00 PM > > > > 7308 > > > > 3-Dec > > > > 16:11 PM > > > > 7310 > > > > 3-Dec > > > > 18:01 PM > > > > 7330 > > > > 3-Dec > > > > 20:00 PM > > > > 7358 > > > > 3-Dec > > > > 22:00 PM > > > > 7394 > > > > 4-Dec > > > > 8:25 AM > > > > 7501 > > > > 4-Dec > > > > 11:09 AM > > > > 7543 > > > > 4-Dec > > > > 12:00 PM > > > > 7551 > > > > 4-Dec > > > > 14:00 PM > > > > 7594 > > > > 4-Dec > > > > 16:00 PM > > > > 7614 > > > > 4-Dec > > > > 18:00 PM > > > > 7635 > > > > 4-Dec > > > > 20:00 PM > > > > 7668 > > > > 4-Dec > > > > 22:00 PM > > > > 7689 > > > > > > > > Thanks & Regards, > > Krishan Babbar > > > > > ============================================================================================================================ > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > > > =========================================================================================================================== > > > > From: Krishan Babbar > > Sent: Monday, December 3, 2018 11:27 AM > > To: [email protected] > > Cc: Atul Kandhari <[email protected]> > > Subject: RE: Getting "Too many open files" warnings11 > > > > > > Thanks Jonathan, > > > > > > > > Ok, but I did not see any message like "acceptor was sleeping". > > > > > > > > I noticed that my open files count for my Java application is keep > increasing day by day as shown below. Why is it not decreasing again? > > > > What configuration am I missing? What can be the reason? > > > > > > Date > > > > Time > > > > Open file count > > > > 30-Nov > > > > 20:00 PM > > > > 2273 > > > > 30-Nov > > > > 22:00 PM > > > > 2544 > > > > 1-Dec > > > > 12:00 AM > > > > 2790 > > > > 1-Dec > > > > 2:00 AM > > > > 2987 > > > > 1-Dec > > > > 4:00 AM > > > > 3283 > > > > 1-Dec > > > > 12:00 PM > > > > 4828 > > > > 1-Dec > > > > 14:00 PM > > > > 5200 > > > > 1-Dec > > > > 16:00 PM > > > > 5635 > > > > 2-Dec > > > > 10:17 AM > > > > 6992 > > > > 2-Dec > > > > 12:00 PM > > > > 7008 > > > > 2-Dec > > > > 14:02 PM > > > > 7024 > > > > 2-Dec > > > > 16:08 PM > > > > 7047 > > > > > > > > > > > > > > In my RequestHandler, I have following settings, means setting session > timeout to 1800 seconds (30 Mins) > > @Override > > public void messageReceived(IoSession session, Object message) > throws Exception { > > if (session != null) { > > this.session = session; > > this.session.getConfig().setBothIdleTime(1800); > > LOGGER.info("Session Idle time in sec: " + > this.session.getConfig().getBothIdleTime()); > > } > > > > this.message = message; > > > > } > > > > > > > > Yes, I can simulate multiple devices with lowering nlimit and test. What > would be the next? > > > > > > > > Thanks & Regards, > > > > Krishan Babbar > > > > > > > > > ============================================================================================================================ > > > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > > > > > =========================================================================================================================== > > > > > > > > -----Original Message----- > > From: Jonathan Valliere <[email protected]<mailto: > [email protected]>> > > Sent: Friday, November 30, 2018 7:20 PM > > To: [email protected]<mailto:[email protected]> > > Cc: Atul Kandhari <[email protected]<mailto: > [email protected]>> > > Subject: Re: Getting "Too many open files" warnings11 > > > > > > > > Mina is patched to prevent that exception from occurring in a cycle. That > > > > was a long time ago, your version is patched. You probably noticed a > > > > message saying the acceptor was sleeping. > > > > > > > > Do you have any kind of testing utility to create fake users for your > application? I would recommend lowering the nlimit and trying to reproduce > > > > entirely artificially. There shouldn’t be any correlation to > > > > > > > > On Fri, Nov 30, 2018 at 8:39 AM Krishan Babbar < > [email protected]<mailto:[email protected]>> > > > > wrote: > > > > > > > >> Hi Jonathan, > > > >> > > > >> Version - 2.0.16 > > > >> Java Application process was using around 1% of CPU and 3.4% of Memory. > > > >> > > > >> After mitigate, were you able to provide the solution? > > > >> I mean what would be the solution if you know? > > > >> > > > >> Thanks & Regards, > > > >> Krishan Babbar > > > >> > > > >> -----Original Message----- > > > >> From: Jonathan Valliere <[email protected]<mailto: > [email protected]>> > > > >> Sent: Friday, November 30, 2018 6:56 PM > > > >> To: [email protected]<mailto:[email protected]> > > > >> Cc: Atul Kandhari <[email protected]<mailto: > [email protected]>> > > > >> Subject: Re: Getting "Too many open files" warnings11 > > > >> > > > >> What version of Mina are you using? We added code to mitigate this > > > >> specific exemption a while back. > > > >> > > > >> Do you know how much cpu Java was consuming at that point in time? > > > >> > > > >> On Fri, Nov 30, 2018 at 8:24 AM Krishan Babbar < > > > >> [email protected]<mailto:[email protected]>> > > > >> wrote: > > > >> > > > >>> Thanks Stefan, > > > >>> > > > >>> It would be great if you could answers other questions too. > > > >>> -What do you think, why application stops receiving packets from > devices? > > > >>> Is it only due to exception "Too many open files" or it can be due > > > >>> to any other reasons as well? Please guide. > > > >>> -Is my MINA configuration fine shown in below Java code? Please > > > >>> suggest if I need to fine tune it more. > > > >>> > > > >>> Thanks & Regards, > > > >>> Krishan Babbar > > > >>> > > > >>> > > > >>> > > > >>> ==================================================================== > > > >>> == ====================================================== > > > >>> Disclaimer: This message and the information contained herein is > > > >>> proprietary and confidential and subject to the Tech Mahindra policy > > > >>> statement, you may review the policy at > > > >>> http://www.techmahindra.com/Disclaimer.html externally > > > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > > > >>> TechMahindra. > > > >>> > > > >>> ==================================================================== > > > >>> == ===================================================== > > > >>> > > > >>> -----Original Message----- > > > >>> From: Stefan Magnus Landrø <[email protected]<mailto: > [email protected]>> > > > >>> Sent: Friday, November 30, 2018 5:59 PM > > > >>> To: [email protected]<mailto:[email protected]> > > > >>> Cc: Atul Kandhari <[email protected]<mailto: > [email protected]>> > > > >>> Subject: Re: Getting "Too many open files" warnings11 > > > >>> > > > >>> > > > >>> https://unix.stackexchange.com/questions/108603/do-changes-in-etc-se > > > >>> cu > > > >>> rity-limits-conf-require-a-reboot > > > >>> > > > >>> > > > >>> Sendt fra min iPhone > > > >>> > > > >>>> 30. nov. 2018 kl. 11:17 skrev Krishan Babbar > > > >>>> <[email protected] > > > >>>> : > > > >>>> > > > >>>> Dear All, > > > >>>> > > > >>>> I am working on an IoT project. I have created a Java application > > > >>>> using > > > >>> MINA library. All the devices are connecting to this Java > > > >>> application using TCP protocol. I am getting around 3000 packets per > > > >>> hour from about 50 devices as of now and it will increase soon. > > > >>>> > > > >>>> I am facing one issue on production environment. After running say > > > >>>> 1-2 > > > >>> days, my application stops receiving packets from all the devices > > > >>> but application process keep running. To make it working again, I > > > >>> need to restart my Java application. I noticed following warnings in > logs. > > > >>>> Warnings > > > >>>> [NioSocketAcceptor-3] WARN > > > >>>> org.apache.mina.util.DefaultExceptionMonitor > > > >>> 30/11/2018 05:25:40 - Unexpected exception. > > > >>>> java.io.IOException: Too many open files > > > >>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > > >>>> at sun.nio.ch > > > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) > > > >>>> at sun.nio.ch > > > >>> .ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > > > >>>> at > > > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSoc > > > >>> ke > > > >>> tAcceptor.java:194) > > > >>>> at > > > >>> org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept(NioSoc > > > >>> ke > > > >>> tAcceptor.java:51) > > > >>>> at > > > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.proc > > > >>> es > > > >>> sHandles(AbstractPollingIoAcceptor.java:544) > > > >>>> at > > > >>> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run( > > > >>> Ab > > > >>> stractPollingIoAcceptor.java:484) > > > >>>> at > > > >>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnab > > > >>> le > > > >>> .java:64) > > > >>>> at > > > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor > > > >>> .j > > > >>> ava:1142) > > > >>>> at > > > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > >>> java:617) > > > >>>> at java.lang.Thread.run(Thread.java:745) > > > >>>> > > > >>>> Due to above warnings in logs, yesterday, after doing some Google, > > > >>>> I > > > >>> changed file limits in "/etc/security/limits.conf" file to 100000 as > > > >>> shown below, but I did not restart my Java application. > > > >>>> * hard nofile 100000 > > > >>>> * soft nofile 100000 > > > >>>> > > > >>>> Today. I got the same warnings again and I was not getting any > > > >>>> packet > > > >>> from any of the devices. So I restarted my Java application and > > > >>> devices started sending packets again. > > > >>>> Now, my questions are > > > >>>> > > > >>>> - After application stopped receiving packets, I restarted my > > > >>> Java application today. Will it consider new file limits i.e. 100000 > > > >>> and not give Exceptions again? Or do I need to reboot machine as well? > > > >>>> > > > >>>> - What do you think, why application stops receiving packets > > > >>> from devices? Is it only due to exception "Too many open files" or > > > >>> it can be due to any other reasons as well? Please guide. > > > >>>> > > > >>>> - Is my MINA configuration fine shown in below Java code? > > > >>> Please suggest if I need to fine tune it more. > > > >>>> > > > >>>> > > > >>>> Below is given our code for MINA configuration. > > > >>>> import java.io.IOException; > > > >>>> import java.net.InetSocketAddress; import org.apache.log4j.Logger; > > > >>>> import org.apache.mina.core.service.AbstractIoAcceptor; > > > >>>> import org.apache.mina.core.session.IdleStatus; > > > >>>> import org.apache.mina.filter.logging.LoggingFilter; > > > >>>> import org.apache.mina.transport.socket.nio.NioDatagramAcceptor; > > > >>>> > > > >>>> import com.techm.tnt.protocol.gateway.util.ServerConfigLoader; > > > >>>> > > > >>>> /** > > > >>>> * The abstract class for tcp/ip and udp listeners. > > > >>>> */ > > > >>>> public abstract class AbstractListener { > > > >>>> > > > >>>> /** The logger instance.*/ > > > >>>> private static Logger LOGGER = > > > >>>> Logger.getLogger(AbstractListener.class); > > > >>>> > > > >>>> /** The buffer size constant. */ > > > >>>> protected static final int BUFFER_SIZE = 2048; > > > >>>> > > > >>>> /** > > > >>>> * The method to initialize udp and tcp/ip listeners. > > > >>>> * > > > >>>> * @param portKey > > > >>>> * the port of the udp or tcp/ip listener. > > > >>>> * @throws IOException > > > >>>> * the io exception to be thrown in case of > > > >>> error. > > > >>>> */ > > > >>>> public void init(String portKey) throws IOException { > > > >>>> LOGGER.info("Entering init()"); > > > >>>> AbstractIoAcceptor acceptor = > > > >>> initInternal(); > > > >>>> acceptor.setHandler(new > > > >>>> RequestHandler()); > > > >>>> > > > >>> acceptor.getFilterChain().addLast("logger", new LoggingFilter()); > > > >>>> > > > >>> acceptor.getSessionConfig().setReadBufferSize(BUFFER_SIZE); > > > >>>> > > > >>> acceptor.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10); > > > >>>> if (acceptor instanceof > > > >>> NioDatagramAcceptor) { > > > >>>> > > > >>>> ((NioDatagramAcceptor) > > > >>> acceptor).getSessionConfig().setReuseAddress(true); > > > >>>> } > > > >>>> acceptor.bind(new > > > >>> InetSocketAddress(Integer.parseInt(ServerConfigLoader.getInstance(). > > > >>> ge > > > >>> tProperty(portKey)))); > > > >>>> LOGGER.info("Exiting init()"); > > > >>>> } > > > >>>> > > > >>>> /** > > > >>>> * The abstract method to be over ridden by > > > >>>> appropriate > > > >>> listener (udp / > > > >>>> * tcp/ip) for invoking appropriate listener. > > > >>>> * > > > >>>> * @return the appropriate (udp/ tcp/ip) io acceptor. > > > >>>> */ > > > >>>> public abstract AbstractIoAcceptor initInternal(); > > > >>>> } > > > >>>> > > > >>>> Appreciate all tour help. > > > >>>> > > > >>>> Thanks & Regards, > > > >>>> Krishan Babbar > > > >>>> > > > >>>> > > > >>>> > > > >>>> ================================================================== > > > >>>> == == ====================================================== > > > >>>> Disclaimer: This message and the information contained herein is > > > >>> proprietary and confidential and subject to the Tech Mahindra policy > > > >>> statement, you may review the policy at > > > >>> http://www.techmahindra.com/Disclaimer.html externally > > > >>> http://tim.techmahindra.com/tim/disclaimer.html internally within > > > >>> TechMahindra. > > > >>>> ================================================================== > > > >>>> == == ===================================================== > > > >>>> > > > >>>> ================================================================== > > > >>>> == == ====================================================== > > > >>>> > > > >>>> Disclaimer: This message and the information contained herein is > > > >>> proprietary and confidential and subject to the Tech Mahindra policy > > > >>> statement, you may review the policy at > > > >>> http://www.techmahindra.com/Disclaimer.html < > > > >>> http://www.techmahindra.com/Disclaimer.html> externally > > > >>> http://tim.techmahindra.com/tim/disclaimer.html < > > > >>> http://tim.techmahindra.com/tim/disclaimer.html> internally within > > > >>> TechMahindra. > > > >>>> > > > >>>> ================================================================== > > > >>>> == == ====================================================== > > > >>> > > > >> -- > > > >> > > > >> CONFIDENTIALITY NOTICE: The contents of this email message and any > > > >> attachments are intended solely for the addressee(s) and may contain > > > >> confidential and/or privileged information and may be legally > > > >> protected from disclosure. > > > >> > > > >> ====================================================================== > > > >> ====================================================== > > > >> > > > >> Disclaimer: This message and the information contained herein is > > > >> proprietary and confidential and subject to the Tech Mahindra policy > > > >> statement, you may review the policy at > > > >> http://www.techmahindra.com/Disclaimer.html < > > > >> http://www.techmahindra.com/Disclaimer.html> externally > > > >> http://tim.techmahindra.com/tim/disclaimer.html < > > > >> http://tim.techmahindra.com/tim/disclaimer.html> internally within > > > >> TechMahindra. > > > >> > > > >> > > > >> ====================================================================== > > > >> ====================================================== > > > >> > > > > -- > > > > > > > > CONFIDENTIALITY NOTICE: The contents of this email message and any > attachments are intended solely for the addressee(s) and may contain > confidential and/or privileged information and may be legally protected > from disclosure. > > > ============================================================================================================================ > > > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html < > http://www.techmahindra.com/Disclaimer.html> externally > http://tim.techmahindra.com/tim/disclaimer.html < > http://tim.techmahindra.com/tim/disclaimer.html> internally within > TechMahindra. > > > > > ============================================================================================================================ >
