All of this would apply to the original BSD syslog format. RFC 5424 solves most of his problems - presuming the SIEM supports it. Nowadays I am seeing more JSON logging. The latest updates to “Logging in the Cloud” in 2.13.0 will recommend using GELF.
Ralph > On Dec 2, 2019, at 9:02 AM, Gary Gregory <[email protected]> wrote: > > This is what it says for me: > " > What's wrong with syslog? A lot, actually. > > I recently had a post > <https://www.myname.website/im-not-burned-out-im-pissed-off> make it to Hacker > News <https://news.ycombinator.com/item?id=21645114>, lobste.rs > <https://lobste.rs/s/a3hunb/i_m_not_burned_out_i_m_pissed_off>, Pinboard, > and reddit > <https://www.reddit.com/r/hackernews/comments/e2dkoe/im_not_burned_out_im_pissed_off/>. > This was not my doing, I have no idea who submitted the article. Honestly I > was a bit surprised that anyone even knows this blog exists. I write this > as a therapy session for myself when I need to rant but no one needs to > hear it. If I knew I was writing for an audience, I would have written a > *lot* differently. > > The biggest thing I noticed from reading the comments especially on HN was > people who don't understand what I'm talking about and think I'm just > ranting because I hate everyone and everything. That may be true, but > that's not exactly the point of the article. I can easily forgive the > misunderstanding because, again, I wasn't writing for an audience I was > writing for myself. So let me take the opportunity to clear some things up. > > This misconception was repeated a few times in various comments: “*syslog > is a standard and if you don't like it you just don't like your job*”. So > let me explain, from a security standpoint, what's wrong with syslog. > Syslog for security logging > > Let's start by explaining what syslog is and what it does in a security > context. The product I work with is a SIEM, which basically collect logs > from around your environment and combines them together to find security > problems. So we'll collect logs from firewalls, domain controllers, IDPS > systems, Linux servers, AV systems, Tripwire, everything. Everything we can > get. The majority of these send their logs in syslog format. The logs we > get aren't necessarily sensitive on their own, but they do provide a lot of > contextual data that attackers would love. If you want to see alerts around > one account failing to authenticate, you necessarily need to send that > account's username, as well as the source and destination of the traffic. > Now any attacker who sees that log knows the address of your domain > controller, a valid username, and what IP address they're connecting from. > This is... problematic from a security standpoint. > > Now mainly my biggest beef is security vendors who want to ship syslog over > the Internet. I've only encountered it a handful of times, but once is bad > enough. But that's just an add-on to how outdated syslog is, especially for > security logging. Syslog is a *clear text protocol*. There are methods of > encrypting syslog traffic, but it's not standardized and it's up to the > vendor to support it. Most don't. The best way is using a VPN tunnel, but > then you have the overhead of a VPN tunnel and another point of failure. > Again, we're talking security logs. If your logging goes down, so does your > security. > > And if you don't want the overhead of syslog over TLS or stunnel or > OpenVPN, that means you're sending clear text *security logs* over the > wire. On your internal network that's bad enough, but *over the Internet*?! > You'd have to be insane (or work for Cisco). > No standard formatting > > So that's the security beef with syslog over the Internet. Let's move on to > all the other reasons syslog is terrible. The next one is the lack of a > logging standard. IBM's product uses LEEF format, that looks like this: > > LEEF:2.0|Trend Micro|Deep Security Manager||192|cat=System name=Alert Ended > desc=Alert: CPU Warning Threshold Exceeded\nSubject: > 10.201.114.164\nSeverity: Warning sev=3 src=10.201.114.164 usrName=System > msg=Alert: CPU Warning Threshold Exceeded\nSubject: > 10.201.114.164\nSeverity: Warning TrendMicroDsTenant=Primary > > HP/MicroFocus's product uses CEF, which looks like this: > > CEF:0|Trend Micro|Deep Security Manager||600|User Signed > In|3|src=10.52.116.160 suser=admin target=admin msg=User signed in from > 2001:db8::5 > > Linux syslog generates unstructured garbage which looks like this: > > Failed password for root from 192.168.50.65 port 34780 ssh2 > > Cisco's IOS products generate logs like this: > > 00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface > GigabitEthernet0/1, changed state to down 2 > > So now your syslog receiver (the SIEM in this case) has to parse these > multitude of logging formats. CEF and LEEF are nice because it's at least > *some* structure to key your searches/regex on. Other syslog formats... not > so much. And in the world of security, parsing a message wrong means > potentially labeling the victim as the source of the attack and the > attacker as the destination. In a real SOC, this would be the difference > between an external hacker breaching a machine and getting a foothold in > your network versus malware on one of your workstations talking to a > command and control server. That is a *massive* difference and working > under the wrong assumption could cost hours or days of investigation. > Meanwhile the hacker is still crawling around your network. > Unreliable for security logs > > So now that we know syslog is insecure and difficult to parse, let's move > on to its reliability. Syslog has no function built in to retry a message. > If your network goes down or your syslog receiver (your SIEM) goes down, > those messages are lost. The sender has no idea that no one is listening. > Sure you can use TCP > <https://bitbucket.org/blog/follow-up-on-our-downtime-last-week> but... I > wouldn't recommend it. > > So in the real world, this means if you're under a DoS attack and your > syslog receiver/SIEM goes over license, or over capacity of the hardware, > or the network link drops, or the attacker changes your firewall rules or > any number of ways the link between the sender and receiver can be severed > during an active attack... all those logs are lost. Gone forever. You will > never get them back. Which means once the dust has settled, chances are you > still have no idea where the attack came from, what happened, or if the > attacker is still inside your network. Because your devices were dutifully > sending syslog blindly assuming the receiver was listening, and the system > has no idea those logs were lost. Unless you're using TCP, in which case > when the link gets severed for whatever reason, your entire network crashes. > What would a robust logging solution look like? > > There's plenty of robust logging solutions. Sure lots of them have their > own issues, but let's take it one step at a time. JDBC is a good one. It's > a pull not a push, so as long as your SIEM can remember where it left off, > it will always get the missing logs after the dust settles. Many Windows > agents are good at this too (many are not... YMMV). Properly written API > clients can do this as well, remembering where they left off and resuming > when the network connection drops. With the added benefit of, if you're > using HTTPS you're not sending clear text logs across the network or > Internet. > > So, for the love of god, stop using syslog for security logging." > > On Mon, Dec 2, 2019 at 11:01 AM Ralph Goers <[email protected]> > wrote: > >> I am unable to view that web site as it has certificate problems. >> >> Ralph >> >>> On Dec 2, 2019, at 8:48 AM, Matt Sicker <[email protected]> wrote: >>> >>> https://www.myname.website/whats-wrong-with-syslog-a-lot-actually >>> >>> This article seems fairly interesting. I know that we support TLS for >>> syslog, though I didn't know that wasn't standard (or is that specific >>> to RFC 5424?). In fact, I'm not even sure how much of his criticism >>> apply to the updated standard and which are specific to the original >>> syslog format. >>> >>> -- >>> Matt Sicker <[email protected]> >>> >> >> >>
