Re: Too many open files on Solaris
I've narrowed and hopefully solved the problem described below. At the end, Solaris+Java 1.4.2+Turbine framework combination is problematic. In short, Turbine uses resources/ui/skins/default/skin.props (on my system) file for some configurations etc, and doesn't close it! After a while, that file is opened 500-600 times and of course causes Too many open files exception when Tomcat tries to open a socket. The funny thing is that my application doesn't use the functionalities provided by skin.props; it comes 'configured' by default. Simply removing all references to it solved my Too many open files problem. Additionally, the same application on Linux, with the same version of Java and Tomcat doesn't have that problem. And since I perform my tests on Linux, it was pretty hard to find the solution. I hope this post will eventually help somebody. Cheers, Davor Davor Cengija wrote: > My application is hitting that 'too many open files' limit on Solaris. > > Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint > acceptSocket > SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] > ignored exception: java.net.SocketException: Too many open files > java.net.SocketException: Too many open files > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) > at java.net.ServerSocket.implAccept(ServerSocket.java:448) > at java.net.ServerSocket.accept(ServerSocket.java:419) > at > org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:107) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:387) > at > org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) > at java.lang.Thread.run(Thread.java:534) > Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint > closeServerSocket > SEVERE: Caught exception trying to unlock accept on 8080 > java.net.SocketException: Too many open files > Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint > acceptSocket > WARNING: Reinitializing ServerSocket > > I realize that increasing file descriptor limit per process could help a > bit, but my system administrator simply doesn't want to do that (since it > requires restart). > > Now, the problem is how to avoid too many open files exception. Any > suggestions regarding Tomcat (or Apache) configuration, coding practice, > some runtime operating system parameters would be greatly appreciated. > > Here's my configuration: > > Default Solaris 8 (I believe) installation on 2CPU+4GB RAM box. TCP > connection wait timeout (I believe that the correct wording) is decreased > from 240sec to 60sec and that's the only parameter changed on Solaris. > > Tomcat 4.1.30 with j2sdk 1.4.2_04 (server mode) with Apache 2.0 as a > proxy. Every single request goes through Apache which then routes it to > Tomcat. 600.000 hits and cca 3GB of data daily, with peaks during working > hours, of course, about 50.000 hits per hour. The delivered content > consists mostly of small images (3-20k) delivered directly from a database > (servlet sets content type etc, img src="/preview?id=123" style) > > Application lives about 24hours and then dies with the exception shown > above. As far as I can see, all input- and output- streams are flushed and > closed, at least in my application. Cannot guarantee for the underlining > libraries. > > Somehow I think that I cannot squeeze the application anymore and that > we're hitting tomcat+solaris limit. > > Ahm... whoever managed to completelly read this rather long post I hope > will have some suggestions. > > Thanks in advance! > > Cheers, > Davor -- Davor Cengija, [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Too many open files on Solaris
We are experiencing the same issue with our application. The application is accessed through a 2K box running Tomcat 4.0. The application itself resides on an AIX 5.21 box. The problem is sporadic. The only way our end users can get around it is to exit the application and log back in. -Original Message- From: James Black [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 16, 2004 8:40 AM To: Tomcat Users List Subject: Re: Too many open files on Solaris HasanA wrote: >I am having similar problem and i am running Tomcat 4.0 on Open vms >7.2. We have increased number of files open by a process to some >ridiculous value but this error is still comming up. > >Can someone also help with this issue? > > If you are opening up database connections or files on the local system, you may not be closing the connection. You can use a program called losf, which is available from www.sunfreeware.com and see what it tells you about which process has the descriptors open. -- "Love is mutual self-giving that ends in self-recovery." Fulton Sheen James Black[EMAIL PROTECTED] - This electronic message may contain information that is confidential and/or legally privileged. It is intended only for the use of the individual(s) and entity named as recipients in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from any computer. Do not deliver, distribute or copy this message, and do not disclose its contents or take any action in reliance on the information it contains. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Too many open files on Solaris
QM wrote: > : > : My application is hitting that 'too many open files' limit on > : > : Solaris. > : > [I just left this line in for the archives/searching. Read on] > [ yet again ;) ] Yeah baby! :-)) > > This may be rectified in a positive way, long-term: matching -- or at > least, semi-matching -- test/prod hardware. One of my clients had such > a setup. It was beautiful. It is bad that some customers learn the hard way. They finally accepted to set up a test machine. Ahmmm... it is always difficult to fight big companies intertia. Thanks, QM. -- Davor Cengija, [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Too many open files on Solaris
: > : My application is hitting that 'too many open files' limit on Solaris. : > [I just left this line in for the archives/searching. Read on] [ yet again ;) ] : Very mission critical. But the problem is that (strange enough) sysadmin : needs to file a formal restart request (!) in which he needs to explain why : a restart is needed. You just explained it. ;) The suggestions I gave were very quick-hit, yes; because by now 1/ your problem is more political than technical; and 2/ the technical fixes for your app are likely long-term. Some of my clients are very conservative companies that have similar policies. Telling someone, "this is a huge problem that will eventually cost you money (in terms of loss of client faith)" tends to light a fire under the right people. Unfortunately, you (or someone) will then have to explain to upper mgt why this didn't bite you till you went to production. I'm not saying it's your (or anyone's) fault; but someone will have to lay out the facts and point out a solution. This may be rectified in a positive way, long-term: matching -- or at least, semi-matching -- test/prod hardware. One of my clients had such a setup. It was beautiful. Good luck. -QM -- software -- http://www.brandxdev.net tech news -- http://www.RoarNetworX.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Too many open files on Solaris
Andy Eastham wrote: > The first thing that came up in Google was this, which might help: > > http://support.bea.com/support_news/product_troubleshooting/Too_Many_Open_Fi > les_Pattern.html This is a good reference. Here's a clickable link: http://tinyurl.com/2acgv Thanks -- Davor Cengija, [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Too many open files on Solaris
QM wrote: > On Wed, Jun 16, 2004 at 10:59:12AM +0200, Davor Cengija wrote: > : My application is hitting that 'too many open files' limit on Solaris. > > [I just left this line in for the archives/searching. Read on] Me too :-) Ok, this post might sound a little bit funny. Read below. > : but my system administrator simply doesn't want to do that (since it > : requires restart). > > If this application is mission-critical, your sysadmin may want to > rethink his/her approach. =) Very mission critical. But the problem is that (strange enough) sysadmin needs to file a formal restart request (!) in which he needs to explain why a restart is needed. > You could also check out the "ulimit" command. > > What sort of load-testing was done on the app before it went live? It was tested on linux but not on Solaris (long story). Linux performed quite fine. Thanks for the suggestions. I've already done similar steps you're suggesting (your post is a good reference now :-). > How much memory does the container use during peak periods? You could Java is started with minimum 1GB and maximum 2GB memory footprint. As soon as Tomcat starts, top says taht memory SIZE is 1.2G and RES about 700MB. When the application is about to crash, RES is about 1.1GB or slightly less. > setup a second Tomcat instance (either on the current host, or a > different host) and have Apache/mod_jk load-balance between the two. Hehe, really interesting. My next step is to set up a second Tomcat instance + mod_jk on the same box. Another Solaris box is on its way but won't be available very soon. Ok, to resume: QM, your suggestions might be a good quick-fix-how-to for this kind of problems. I already tried them and still have the problems (although the application's lifespan is streched from 2hrs to 24hrs for the current load). My next step is mod_jk balancer. Ah... my sad story continues :-) Kind regards, Davor -- Davor Cengija, [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Too many open files on Solaris
HasanA wrote: I am having similar problem and i am running Tomcat 4.0 on Open vms 7.2. We have increased number of files open by a process to some ridiculous value but this error is still comming up. Can someone also help with this issue? If you are opening up database connections or files on the local system, you may not be closing the connection. You can use a program called losf, which is available from www.sunfreeware.com and see what it tells you about which process has the descriptors open. -- "Love is mutual self-giving that ends in self-recovery." Fulton Sheen James Black[EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Too many open files on Solaris
The first thing that came up in Google was this, which might help: http://support.bea.com/support_news/product_troubleshooting/Too_Many_Open_Fi les_Pattern.html Andy > -Original Message- > From: news [mailto:[EMAIL PROTECTED] On Behalf Of Davor Cengija > Sent: 16 June 2004 09:59 > To: [EMAIL PROTECTED] > Subject: Too many open files on Solaris > > My application is hitting that 'too many open files' limit on Solaris. > > Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint > acceptSocket > SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] > ignored exception: java.net.SocketException: Too many open files > java.net.SocketException: Too many open files > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) > at java.net.ServerSocket.implAccept(ServerSocket.java:448) > at java.net.ServerSocket.accept(ServerSocket.java:419) > at > org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(Default > ServerSocketFactory.java:107) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.ja > va:387) > at > org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.j > ava:683) > at java.lang.Thread.run(Thread.java:534) > Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint > closeServerSocket > SEVERE: Caught exception trying to unlock accept on 8080 > java.net.SocketException: Too many open files > Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint > acceptSocket > WARNING: Reinitializing ServerSocket > > I realize that increasing file descriptor limit per process could help a > bit, but my system administrator simply doesn't want to do that (since it > requires restart). > > Now, the problem is how to avoid too many open files exception. Any > suggestions regarding Tomcat (or Apache) configuration, coding practice, > some runtime operating system parameters would be greatly appreciated. > > Here's my configuration: > > Default Solaris 8 (I believe) installation on 2CPU+4GB RAM box. TCP > connection wait timeout (I believe that the correct wording) is decreased > from 240sec to 60sec and that's the only parameter changed on Solaris. > > Tomcat 4.1.30 with j2sdk 1.4.2_04 (server mode) with Apache 2.0 as a > proxy. > Every single request goes through Apache which then routes it to Tomcat. > 600.000 hits and cca 3GB of data daily, with peaks during working hours, > of > course, about 50.000 hits per hour. The delivered content consists mostly > of small images (3-20k) delivered directly from a database (servlet sets > content type etc, img src="/preview?id=123" style) > > Application lives about 24hours and then dies with the exception shown > above. As far as I can see, all input- and output- streams are flushed and > closed, at least in my application. Cannot guarantee for the underlining > libraries. > > Somehow I think that I cannot squeeze the application anymore and that > we're > hitting tomcat+solaris limit. > > Ahm... whoever managed to completelly read this rather long post I hope > will > have some suggestions. > > Thanks in advance! > > Cheers, > Davor > -- > Davor Cengija, [EMAIL PROTECTED] > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Too many open files on Solaris
On Wed, Jun 16, 2004 at 10:59:12AM +0200, Davor Cengija wrote: : My application is hitting that 'too many open files' limit on Solaris. [I just left this line in for the archives/searching. Read on] : but my system administrator simply doesn't want to do that (since it : requires restart). If this application is mission-critical, your sysadmin may want to rethink his/her approach. =) You could also check out the "ulimit" command. What sort of load-testing was done on the app before it went live? : Now, the problem is how to avoid too many open files exception. The short version: don't open so many files. ;) The longer version: I'm only half-joking here. Remember, everything in Unix is (accessible as/represented by) a file, including network connections. You could limit the number of network connections in your server.xml elements -- or better still, limit Apache's connections to the container. Reduce the number of DB pool connections, etc. You could also check your app for file I/O, such as your code handling temp files. Slim chance, and it's more of a fine-tuning measure; but it will help on a political/friendliness level when you can honestly tell your sysadmin, "I've done all I can do. Your turn." How much memory does the container use during peak periods? You could setup a second Tomcat instance (either on the current host, or a different host) and have Apache/mod_jk load-balance between the two. -QM -- software -- http://www.brandxdev.net tech news -- http://www.RoarNetworX.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Too many open files on Solaris
I am having similar problem and i am running Tomcat 4.0 on Open vms 7.2. We
have increased number of files open by a process to some ridiculous value
but this error is still comming up.
Can someone also help with this issue?
Thanks
Ajwat
-Original Message-
From: Davor Cengija [mailto:[EMAIL PROTECTED]
Sent: 16 June 2004 09:59
To: [EMAIL PROTECTED]
Subject: Too many open files on Solaris
My application is hitting that 'too many open files' limit on Solaris.
Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080]
ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultSe
rverSocketFactory.java:107)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java
:387)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:683)
at java.lang.Thread.run(Thread.java:534)
Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
closeServerSocket
SEVERE: Caught exception trying to unlock accept on 8080
java.net.SocketException: Too many open files
Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket
WARNING: Reinitializing ServerSocket
I realize that increasing file descriptor limit per process could help a
bit, but my system administrator simply doesn't want to do that (since it
requires restart).
Now, the problem is how to avoid too many open files exception. Any
suggestions regarding Tomcat (or Apache) configuration, coding practice,
some runtime operating system parameters would be greatly appreciated.
Here's my configuration:
Default Solaris 8 (I believe) installation on 2CPU+4GB RAM box. TCP
connection wait timeout (I believe that the correct wording) is decreased
from 240sec to 60sec and that's the only parameter changed on Solaris.
Tomcat 4.1.30 with j2sdk 1.4.2_04 (server mode) with Apache 2.0 as a proxy.
Every single request goes through Apache which then routes it to Tomcat.
600.000 hits and cca 3GB of data daily, with peaks during working hours, of
course, about 50.000 hits per hour. The delivered content consists mostly
of small images (3-20k) delivered directly from a database (servlet sets
content type etc, img src="/preview?id=123" style)
Application lives about 24hours and then dies with the exception shown
above. As far as I can see, all input- and output- streams are flushed and
closed, at least in my application. Cannot guarantee for the underlining
libraries.
Somehow I think that I cannot squeeze the application anymore and that we're
hitting tomcat+solaris limit.
Ahm... whoever managed to completelly read this rather long post I hope will
have some suggestions.
Thanks in advance!
Cheers,
Davor
--
Davor Cengija, [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
This communication together with any attachments transmitted with it ("this E-Mail")
is intended only for the use of the addressee and may contain information which is
privileged and confidential. If the reader of this E-Mail is not the intended
recipient or the employee or agent responsible for delivering it to the intended
recipient you are hereby notified that any use, dissemination, forwarding, printing or
copying of this E-Mail is strictly prohibited. Addressees should check this E-mail
for viruses. The Company makes no representations as regards the absence of viruses
in this E-Mail. If you have received this E-Mail in error please notify our ISe
Response Team immediately by telephone on +44 (0)20 8896 5828 or via e-mail at [EMAIL
PROTECTED] Please then immediately delete, erase or otherwise destroy this E-Mail and
any copies of it.
Any opinions expressed in this E-Mail are those of the author and do not necessarily
constitute the views of the Company. Nothing in this E-Mail shall bind the Company in
any contract or obligation.
For the purposes of this E-Mail "the Company" means The Carphone Warehouse Group Plc
and/or any of its subsidiaries.
Please feel free to visit our website: http:// www.carphonewarehouse.com or
http://www.phonehouse.com
The Carphone Warehouse Group Plc (Registered in England No. 3253714) 1 Portal Way,
London W3 6RS
Too many open files on Solaris
My application is hitting that 'too many open files' limit on Solaris. Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored exception: java.net.SocketException: Too many open files java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:107) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:387) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:557) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) at java.lang.Thread.run(Thread.java:534) Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint closeServerSocket SEVERE: Caught exception trying to unlock accept on 8080 java.net.SocketException: Too many open files Jun 15, 2004 11:23:01 AM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket WARNING: Reinitializing ServerSocket I realize that increasing file descriptor limit per process could help a bit, but my system administrator simply doesn't want to do that (since it requires restart). Now, the problem is how to avoid too many open files exception. Any suggestions regarding Tomcat (or Apache) configuration, coding practice, some runtime operating system parameters would be greatly appreciated. Here's my configuration: Default Solaris 8 (I believe) installation on 2CPU+4GB RAM box. TCP connection wait timeout (I believe that the correct wording) is decreased from 240sec to 60sec and that's the only parameter changed on Solaris. Tomcat 4.1.30 with j2sdk 1.4.2_04 (server mode) with Apache 2.0 as a proxy. Every single request goes through Apache which then routes it to Tomcat. 600.000 hits and cca 3GB of data daily, with peaks during working hours, of course, about 50.000 hits per hour. The delivered content consists mostly of small images (3-20k) delivered directly from a database (servlet sets content type etc, img src="/preview?id=123" style) Application lives about 24hours and then dies with the exception shown above. As far as I can see, all input- and output- streams are flushed and closed, at least in my application. Cannot guarantee for the underlining libraries. Somehow I think that I cannot squeeze the application anymore and that we're hitting tomcat+solaris limit. Ahm... whoever managed to completelly read this rather long post I hope will have some suggestions. Thanks in advance! Cheers, Davor -- Davor Cengija, [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
