On Mon, Dec 2, 2019 at 11:24 AM Ralph Goers <[email protected]> wrote:
> All of this would apply to the original BSD syslog format. RFC 5424 solves > most of his problems - presuming the SIEM supports it. Nowadays I am > seeing more JSON logging. The latest updates to “Logging in the Cloud” in > 2.13.0 will recommend using GELF. > When you do you we can get a 2.13.0? Gary > > Ralph > > > On Dec 2, 2019, at 9:02 AM, Gary Gregory <[email protected]> wrote: > > > > This is what it says for me: > > " > > What's wrong with syslog? A lot, actually. > > > > I recently had a post > > <https://www.myname.website/im-not-burned-out-im-pissed-off> make it to > Hacker > > News <https://news.ycombinator.com/item?id=21645114>, lobste.rs > > <https://lobste.rs/s/a3hunb/i_m_not_burned_out_i_m_pissed_off>, > Pinboard, > > and reddit > > < > https://www.reddit.com/r/hackernews/comments/e2dkoe/im_not_burned_out_im_pissed_off/ > >. > > This was not my doing, I have no idea who submitted the article. > Honestly I > > was a bit surprised that anyone even knows this blog exists. I write this > > as a therapy session for myself when I need to rant but no one needs to > > hear it. If I knew I was writing for an audience, I would have written a > > *lot* differently. > > > > The biggest thing I noticed from reading the comments especially on HN > was > > people who don't understand what I'm talking about and think I'm just > > ranting because I hate everyone and everything. That may be true, but > > that's not exactly the point of the article. I can easily forgive the > > misunderstanding because, again, I wasn't writing for an audience I was > > writing for myself. So let me take the opportunity to clear some things > up. > > > > This misconception was repeated a few times in various comments: “*syslog > > is a standard and if you don't like it you just don't like your job*”. So > > let me explain, from a security standpoint, what's wrong with syslog. > > Syslog for security logging > > > > Let's start by explaining what syslog is and what it does in a security > > context. The product I work with is a SIEM, which basically collect logs > > from around your environment and combines them together to find security > > problems. So we'll collect logs from firewalls, domain controllers, IDPS > > systems, Linux servers, AV systems, Tripwire, everything. Everything we > can > > get. The majority of these send their logs in syslog format. The logs we > > get aren't necessarily sensitive on their own, but they do provide a lot > of > > contextual data that attackers would love. If you want to see alerts > around > > one account failing to authenticate, you necessarily need to send that > > account's username, as well as the source and destination of the traffic. > > Now any attacker who sees that log knows the address of your domain > > controller, a valid username, and what IP address they're connecting > from. > > This is... problematic from a security standpoint. > > > > Now mainly my biggest beef is security vendors who want to ship syslog > over > > the Internet. I've only encountered it a handful of times, but once is > bad > > enough. But that's just an add-on to how outdated syslog is, especially > for > > security logging. Syslog is a *clear text protocol*. There are methods of > > encrypting syslog traffic, but it's not standardized and it's up to the > > vendor to support it. Most don't. The best way is using a VPN tunnel, but > > then you have the overhead of a VPN tunnel and another point of failure. > > Again, we're talking security logs. If your logging goes down, so does > your > > security. > > > > And if you don't want the overhead of syslog over TLS or stunnel or > > OpenVPN, that means you're sending clear text *security logs* over the > > wire. On your internal network that's bad enough, but *over the > Internet*?! > > You'd have to be insane (or work for Cisco). > > No standard formatting > > > > So that's the security beef with syslog over the Internet. Let's move on > to > > all the other reasons syslog is terrible. The next one is the lack of a > > logging standard. IBM's product uses LEEF format, that looks like this: > > > > LEEF:2.0|Trend Micro|Deep Security Manager||192|cat=System name=Alert > Ended > > desc=Alert: CPU Warning Threshold Exceeded\nSubject: > > 10.201.114.164\nSeverity: Warning sev=3 src=10.201.114.164 usrName=System > > msg=Alert: CPU Warning Threshold Exceeded\nSubject: > > 10.201.114.164\nSeverity: Warning TrendMicroDsTenant=Primary > > > > HP/MicroFocus's product uses CEF, which looks like this: > > > > CEF:0|Trend Micro|Deep Security Manager||600|User Signed > > In|3|src=10.52.116.160 suser=admin target=admin msg=User signed in from > > 2001:db8::5 > > > > Linux syslog generates unstructured garbage which looks like this: > > > > Failed password for root from 192.168.50.65 port 34780 ssh2 > > > > Cisco's IOS products generate logs like this: > > > > 00:00:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface > > GigabitEthernet0/1, changed state to down 2 > > > > So now your syslog receiver (the SIEM in this case) has to parse these > > multitude of logging formats. CEF and LEEF are nice because it's at least > > *some* structure to key your searches/regex on. Other syslog formats... > not > > so much. And in the world of security, parsing a message wrong means > > potentially labeling the victim as the source of the attack and the > > attacker as the destination. In a real SOC, this would be the difference > > between an external hacker breaching a machine and getting a foothold in > > your network versus malware on one of your workstations talking to a > > command and control server. That is a *massive* difference and working > > under the wrong assumption could cost hours or days of investigation. > > Meanwhile the hacker is still crawling around your network. > > Unreliable for security logs > > > > So now that we know syslog is insecure and difficult to parse, let's move > > on to its reliability. Syslog has no function built in to retry a > message. > > If your network goes down or your syslog receiver (your SIEM) goes down, > > those messages are lost. The sender has no idea that no one is listening. > > Sure you can use TCP > > <https://bitbucket.org/blog/follow-up-on-our-downtime-last-week> but... > I > > wouldn't recommend it. > > > > So in the real world, this means if you're under a DoS attack and your > > syslog receiver/SIEM goes over license, or over capacity of the hardware, > > or the network link drops, or the attacker changes your firewall rules or > > any number of ways the link between the sender and receiver can be > severed > > during an active attack... all those logs are lost. Gone forever. You > will > > never get them back. Which means once the dust has settled, chances are > you > > still have no idea where the attack came from, what happened, or if the > > attacker is still inside your network. Because your devices were > dutifully > > sending syslog blindly assuming the receiver was listening, and the > system > > has no idea those logs were lost. Unless you're using TCP, in which case > > when the link gets severed for whatever reason, your entire network > crashes. > > What would a robust logging solution look like? > > > > There's plenty of robust logging solutions. Sure lots of them have their > > own issues, but let's take it one step at a time. JDBC is a good one. > It's > > a pull not a push, so as long as your SIEM can remember where it left > off, > > it will always get the missing logs after the dust settles. Many Windows > > agents are good at this too (many are not... YMMV). Properly written API > > clients can do this as well, remembering where they left off and resuming > > when the network connection drops. With the added benefit of, if you're > > using HTTPS you're not sending clear text logs across the network or > > Internet. > > > > So, for the love of god, stop using syslog for security logging." > > > > On Mon, Dec 2, 2019 at 11:01 AM Ralph Goers <[email protected]> > > wrote: > > > >> I am unable to view that web site as it has certificate problems. > >> > >> Ralph > >> > >>> On Dec 2, 2019, at 8:48 AM, Matt Sicker <[email protected]> wrote: > >>> > >>> https://www.myname.website/whats-wrong-with-syslog-a-lot-actually > >>> > >>> This article seems fairly interesting. I know that we support TLS for > >>> syslog, though I didn't know that wasn't standard (or is that specific > >>> to RFC 5424?). In fact, I'm not even sure how much of his criticism > >>> apply to the updated standard and which are specific to the original > >>> syslog format. > >>> > >>> -- > >>> Matt Sicker <[email protected]> > >>> > >> > >> > >> > > >
