Hi Florent,
this definitely sounds like a bug. Please create a bug report for this at
https://github.com/Graylog2/graylog2-server/issues and include some
information about the connections (e. g. which TCP ports are being used) to
the description so that we can identify the culprit (e. g. a certain type
of input or the REST API).
Cheers,
Jochen
On Thursday, 3 September 2015 11:18:15 UTC+2, Florent B wrote:
>
> Hi everyone,
>
> I have a problem with my up-to-date Graylog setup.
>
> After a few days, my 3 Graylog server are getting "Too many opened
> files" errors (I restart them and it runs for a few days).
>
> So I investigate this. I can see that lsof is returning a very high
> number of opened files :
>
> # lsof | wc -l
> 167577
>
> I can see that most of these are "ESTABLISHED" connections with my
> Graylog server instance :
>
> # lsof | grep ESTABLISHED | awk '{print $2}' | sort | uniq -c | sort -n
> 1 20285
> 5 2125
> 74 2395 <-- Graylog-web java process
> 560 2069 <-- Mongodb process
> 120288 2279 <-- Graylog-server java process
>
> If I sort those connections, I got very high number of ESTABLISHED
> connections from all clients sending logs to Graylog :
>
> # lsof -n | grep ESTABLISHED | awk '{print $10}' | cut -d '>' -f 2 | cut
> -d ':' -f 1 | sort | uniq -c | sort -n
> [...]
> 4509 10.111.XX.XA
> 4509 10.111.XX.XB
> 4676 10.111.XX.XC
> 4676 10.111.XX.XD
> 4676 10.111.XX.XE
> 4676 10.111.XX.XF
> 4839 10.111.XX.XG
> 4843 10.111.XX.XH
> 5010 10.111.XX.XI
> 5511 10.111.XX.XJ
> 5678 10.111.XX.XK
> 6012 10.111.XX.XL
>
> All clients are running Debian Wheezy, same as servers, without any
> specific configuration.
>
> Of course, connection are not "symmetric", clients have correct number
> of ESTABLISHED connections :
>
> # lsof | grep "ESTABLISHED" | wc -l
> 6
>
> What I don't understand, is that when I list ESTABLISHED connections on
> server, using "netstat" command, I can't see all those "zombie"
> connections :
>
> # netstat -apn | grep ESTABLISHED |wc -l
> 730
>
> 730 is more "correct" that my 120288 from lsof. There is a huge
> difference between lsof and netstat ESTABLISHED connections;
>
> So where is the problem ? It seems that Graylog-server does not close
> properly connections, and leave socket opened or something like this. Or
> Java does not clean correctly, I don't know...
>
> Except this, Graylog is running fine with us :-)
>
> Thank you all for your help.
>
> Florent
>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/7d8aa75b-5074-4b21-86c7-f9ba1af75c53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
