[
https://issues.apache.org/jira/browse/STORM-250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924174#comment-13924174
]
Robert Joseph Evans commented on STORM-250:
-------------------------------------------
Reading through things lsof seems to report 'can't identify protocol' when the
socket is closed, but the file descriptor is not closed.
see
http://stackoverflow.com/questions/7911840/seeing-too-many-lsof-cant-identify-protocol
There could be other situations too. My guess would be that for some reason a
code path is being executed for thrift where the socket is not fully closed and
we end up relying on the garbage collector to clean up the connection. Java
streams typically have a finalizer in them that will close the underlying file
handle when called, but It takes two passes of the gc before they can be
collected. I have seen similar issues in other projects where a burst of
traffic would make it difficult for them all to shut down.
I am not sure what is causing it here though. I cannot reproduce it with the
version of storm that I have been running, but I have not tried it with 0.9.0.1.
> Storm UI: Too many open files
> -----------------------------
>
> Key: STORM-250
> URL: https://issues.apache.org/jira/browse/STORM-250
> Project: Apache Storm (Incubating)
> Issue Type: Bug
> Environment: # cat /etc/redhat-release
> CentOS release 6.2 (Final)
> # uname -a
> Linux {{redacted}} 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011
> x86_64 x86_64 x86_64 GNU/Linux
> # ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 30514
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1024
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> Reporter: Vito Laurenza
> Priority: Minor
> Labels: storm, ui
> Attachments: lsof.txt
>
>
> Occasionally, the ui daemon runs out of available file handles and can't open
> a socket to the nimbus:
> # grep -A1 ERROR ui.log
> 2014-02-28 07:44:14 o.a.t.t.TSocket [ERROR] Could not configure socket.
> java.net.SocketException: Too many open files
> --
> 2014-03-01 16:44:10 o.a.t.t.TSocket [ERROR] Could not configure socket.
> java.net.SocketException: Too many open files
> --
> 2014-03-03 01:29:04 o.a.t.t.TSocket [ERROR] Could not configure socket.
> java.net.SocketException: Too many open files
> --
> 2014-03-03 01:34:04 o.a.t.t.TSocket [ERROR] Could not configure socket.
> java.net.SocketException: Too many open files
> --
> 2014-03-04 09:39:01 o.a.t.t.TSocket [ERROR] Could not configure socket.
> java.net.SocketException: Too many open files
> --
> 2014-03-04 09:45:33 o.a.t.t.TSocket [ERROR] Could not configure socket.
> java.net.SocketException: Too many open files
--
This message was sent by Atlassian JIRA
(v6.2#6252)
