> -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Mr. Demeanour > Sent: Friday, November 20, 2009 5:10 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 4.4.2: server out-of-memory with gnutls > > Mr. Demeanour wrote: > > Mr. Demeanour wrote: > >> Hi, > >> > >> I'm running a central rsyslog server with a couple of remote WAN > >> (internet) clients and several remote LAN clients. Traffic is low - > >> of the order of 10,000 messages per day. Internet clients > >> communicate with the server using gnutls. LAN clients are currently > >> using UDP. The server writes client logs to mysql, and also writes > >> messages of local origin to disk. > > > > Further to this: > > > > I have been running 4.5.6 for about a week now, *without* gnutls > > enabled. No leaks. > > > > This evening I re-enabled gnutls, and almost immediately noted > > excessive memory usage, *and* 99% cpu. > > > > It seems that the high CPU usage occurs with hosts outside my local > > network; it may be that there is some misconfiguration of > NAT that is > > behind that problem. > > Not NAT. It seems that I had set up the server certificate with an > incorrect CN. > > I guess the client was trying repeatedly to make a connection that was > doomed to fail every time. That would explain the CPU spike. > If there is > also a memory leak in the gnutls server code concerning connection > setup, that would explain the memory consumption also. > > Perhaps rsyslog should give up trying to connect to a remote > server, or > at least back off, if the error it encounters is of a kind that most > likely requires human intervention? Such would generally be > the case if > a certificate is invalid.
The situation is a bit complex in such cases. There, a shortcoming in RFC5425 "works together" with a shortcoming of GnuTLS. The end result is that I am unable to detect a problem occurs. This happens: RFC5425 does not specifiy app-level acknowledgements. Consequently, there is no way to convey an error condition back to the client. GnuTLS does not permit to check the certificate *BEFORE* accepting a connection. Consequently, the connection must be accepted, then the certificate checked and then terminated if the cert if invalid. However, to the client this looks like a successfully established connection (beause it was connected) that just quickly went down (but the client does not know an error state as there is no app-layer vehicle to provide it back). This also disables the usual rsyslog logic to pause failed connection attempts - because the connection succeeds in the first place. The only solution I currently see is either not to use RFC5425 or use something else than GnuTLS (for example, openssl permits to check the certificate BEFORE accepting the connection). An other approach is to modify GnuTLS so that it provides an ability to do the check before accepting the certificate. I looked into that option, but it requires far too much effort to me. So the right thing to do would probably write an additional TLS driver for e.g. openssl. While this is actually on the todo list, I have to admit that I do not have time to do so at the moment, nor do I see a spot in the near future where I could tackle that beast. I still have the hope that we will receive a contributed driver for NSS, which would most probably also solve this issue. Or wait for GnuTLS to evolve... I guess you get the picture ;) Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
