Re: [dnsdist] expected file descriptor usage
>> What will happen when it hits the limit >> of 16k? > > We might refuse new TCP connections, or if we fail to properly handle a > case you might see dnsdist restarting after an unhandled exception. Now we got to experience what happens when dnsdist reaches the 16k limit in case anyone else wants to know (confirm or disagree) - dnsdist does not restart (but that would make sense and would result in "auto-recovery") - backend resolver (running on the same OS) is up/down flapping according to dnsdist - dumpStats() reports a fd-usage of "0" [...] dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'down' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'up' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'down' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'up' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'down' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'up' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'down' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'up' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'down' dnsdist[20176]: Problem sending carbon data: Too many open files dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'up' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'down' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'up' dnsdist[20176]: Marking downstream 127.0.0.1:53 as 'down' [...] We restarted it manually. ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] expected file descriptor usage
for the record: https://github.com/PowerDNS/pdns/issues/8039 ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] expected file descriptor usage
Remi Gacogne wrote: > No, it's not. Would you be able to issue a lsof -n -p dnsdist process>? Since dnsdist just restarted the output is not so useful at the moment but I'll get back to you once the fd-usage has grown again. > I wonder if that might be related to [1], which fixes an issue with > timeouts. Would you be willing to test it? We can provide packages if > needed. We are happy to test your debian packages if we can verify their signature using your usual signing keys. ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] expected file descriptor usage
Hi Cristoph, On 7/6/19 9:39 PM, Christoph wrote: > we run a small DoH server using dnsdist > and are observing an almost linear growth > of dnsdist's file descriptor usage > (initially about ~1k/day now down to ~500/day) > > Is this expected behavior? No, it's not. Would you be able to issue a lsof -n -p ? If you don't want to shared the whole output, for example because of the IP addresses listed in there, can you at least confirm that most of the file descriptors are incoming TCP sockets from clients to the DoH port? It could also be TCP connections from dnsdist to your backend but we do clean these up regularly. I wonder if that might be related to [1], which fixes an issue with timeouts. Would you be willing to test it? We can provide packages if needed. > What will happen when it hits the limit > of 16k? We might refuse new TCP connections, or if we fail to properly handle a case you might see dnsdist restarting after an unhandled exception. [1]: https://github.com/PowerDNS/pdns/pull/7927 Best regards, -- Remi Gacogne PowerDNS.COM BV - https://www.powerdns.com/ signature.asc Description: OpenPGP digital signature ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
