This is what it says for me: " What's wrong with syslog? A lot, actually.
I recently had a post <https://www.myname.website/im-not-burned-out-im-pissed-off> make it to Hacker News <https://news.ycombinator.com/item?id=21645114>, lobste.rs <https://lobste.rs/s/a3hunb/i_m_not_burned_out_i_m_pissed_off>, Pinboard, and reddit <https://www.reddit.com/r/hackernews/comments/e2dkoe/im_not_burned_out_im_pissed_off/>. This was not my doing, I have no idea who submitted the article. Honestly I was a bit surprised that anyone even knows this blog exists. I write this as a therapy session for myself when I need to rant but no one needs to hear it. If I knew I was writing for an audience, I would have written a *lot* differently. The biggest thing I noticed from reading the comments especially on HN was people who don't understand what I'm talking about and think I'm just ranting because I hate everyone and everything. That may be true, but that's not exactly the point of the article. I can easily forgive the misunderstanding because, again, I wasn't writing for an audience I was writing for myself. So let me take the opportunity to clear some things up. This misconception was repeated a few times in various comments: “*syslog is a standard and if you don't like it you just don't like your job*”. So let me explain, from a security standpoint, what's wrong with syslog. Syslog for security logging Let's start by explaining what syslog is and what it does in a security context. The product I work with is a SIEM, which basically collect logs from around your environment and combines them together to find security problems. So we'll collect logs from firewalls, domain controllers, IDPS systems, Linux servers, AV systems, Tripwire, everything. Everything we can get. The majority of these send their logs in syslog format. The logs we get aren't necessarily sensitive on their own, but they do provide a lot of contextual data that attackers would love. If you want to see alerts around one account failing to authenticate, you necessarily need to send that account's username, as well as the source and destination of the traffic. Now any attacker who sees that log knows the address of your domain controller, a valid username, and what IP address they're connecting from. This is... problematic from a security standpoint. Now mainly my biggest beef is security vendors who want to ship syslog over the Internet. I've only encountered it a handful of times, but once is bad enough. But that's just an add-on to how outdated syslog is, especially for security logging. Syslog is a *clear text protocol*. There are methods of encrypting syslog traffic, but it's not standardized and it's up to the vendor to support it. Most don't. The best way is using a VPN tunnel, but then you have the overhead of a VPN tunnel and another point of failure. Again, we're talking security logs. If your logging goes down, so does your security. And if you don't want the overhead of syslog over TLS or stunnel or OpenVPN, that means you're sending clear text *security logs* over the wire. On your internal network that's bad enough, but *over the Internet*?! You'd have to be insane (or work for Cisco). No standard formatting So that's the security beef with syslog over the Internet. Let's move on to all the other reasons syslog is terrible. The next one is the lack of a logging standard. IBM's product uses LEEF format, that looks like this: LEEF:2.0|Trend Micro|Deep Security Manager||192|cat=System name=Alert Ended desc=Alert: CPU Warning Threshold Exceeded\nSubject: 10.201.114.164\nSeverity: Warning sev=3 src=10.201.114.164 usrName=System msg=Alert: CPU Warning Threshold Exceeded\nSubject: 10.201.114.164\nSeverity: Warning TrendMicroDsTenant=Primary HP/MicroFocus's product uses CEF, which looks like this: CEF:0|Trend Micro|Deep Security Manager||600|User Signed In|3|src=10.52.116.160 suser=admin target=admin msg=User signed in from 2001:db8::5 Linux syslog generates unstructured garbage which looks like this: Failed password for root from 192.168.50.65 port 34780 ssh2 Cisco's IOS products generate logs like this: 00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down 2 So now your syslog receiver (the SIEM in this case) has to parse these multitude of logging formats. CEF and LEEF are nice because it's at least *some* structure to key your searches/regex on. Other syslog formats... not so much. And in the world of security, parsing a message wrong means potentially labeling the victim as the source of the attack and the attacker as the destination. In a real SOC, this would be the difference between an external hacker breaching a machine and getting a foothold in your network versus malware on one of your workstations talking to a command and control server. That is a *massive* difference and working under the wrong assumption could cost hours or days of investigation. Meanwhile the hacker is still crawling around your network. Unreliable for security logs So now that we know syslog is insecure and difficult to parse, let's move on to its reliability. Syslog has no function built in to retry a message. If your network goes down or your syslog receiver (your SIEM) goes down, those messages are lost. The sender has no idea that no one is listening. Sure you can use TCP <https://bitbucket.org/blog/follow-up-on-our-downtime-last-week> but... I wouldn't recommend it. So in the real world, this means if you're under a DoS attack and your syslog receiver/SIEM goes over license, or over capacity of the hardware, or the network link drops, or the attacker changes your firewall rules or any number of ways the link between the sender and receiver can be severed during an active attack... all those logs are lost. Gone forever. You will never get them back. Which means once the dust has settled, chances are you still have no idea where the attack came from, what happened, or if the attacker is still inside your network. Because your devices were dutifully sending syslog blindly assuming the receiver was listening, and the system has no idea those logs were lost. Unless you're using TCP, in which case when the link gets severed for whatever reason, your entire network crashes. What would a robust logging solution look like? There's plenty of robust logging solutions. Sure lots of them have their own issues, but let's take it one step at a time. JDBC is a good one. It's a pull not a push, so as long as your SIEM can remember where it left off, it will always get the missing logs after the dust settles. Many Windows agents are good at this too (many are not... YMMV). Properly written API clients can do this as well, remembering where they left off and resuming when the network connection drops. With the added benefit of, if you're using HTTPS you're not sending clear text logs across the network or Internet. So, for the love of god, stop using syslog for security logging." On Mon, Dec 2, 2019 at 11:01 AM Ralph Goers <[email protected]> wrote: > I am unable to view that web site as it has certificate problems. > > Ralph > > > On Dec 2, 2019, at 8:48 AM, Matt Sicker <[email protected]> wrote: > > > > https://www.myname.website/whats-wrong-with-syslog-a-lot-actually > > > > This article seems fairly interesting. I know that we support TLS for > > syslog, though I didn't know that wasn't standard (or is that specific > > to RFC 5424?). In fact, I'm not even sure how much of his criticism > > apply to the updated standard and which are specific to the original > > syslog format. > > > > -- > > Matt Sicker <[email protected]> > > > > >
