Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
On Fri, Mar 19, 2021 at 8:03 AM Kyle Evans wrote: > > On Mon, Mar 15, 2021 at 9:47 AM Jason A. Donenfeld wrote: > > > > Hi everybody, > > > > [... snip ...] > > > > The first step was assessing the current state of the code the previous > > developer had dumped into the tree. It was not pretty. I imagined > > strange Internet voices jeering, “this is what gives C a bad name!” > > This was a highly unnecessary jab. > > > There were random sleeps added to “fix” race conditions > > This is true. > > > validation functions that just returned true > > I looked back at the history here just now, and I count one, > wg_allowedip_valid, that pretty much got ripped out anyways. > > > catastrophic cryptographic vulnerabilities > > whole parts of the protocol unimplemented > > I'm not qualified to speak about these two, which are perhaps the > worst from the public's perspective. > > > kernel panics > > These are true. > > > security bypasses > > overflows > > I only recall one of each. > > > random printf statements deep in crypto code > > This is true. > > > the most spectacular buffer overflows > > English is a crappy language, but this sounds like you're advertising > more than one. There was one, and for the record to anyone watching to > dispel "the word on the street" that it's potentially an RCE: based on what's > happening I cannot imagine how you could usefully turn it into an RCE. > Local privileged execution, 100%, but looking at it further, you've got to be > pretty skilled to make it before killing the kernel elsewhere. > Ah, I have to clarify this one; I suspect if you're acting as a wg gateway, it's possible. It still doesn't look like an easy one to exploit. > > and the whole litany of awful things that go wrong when people aren’t > > careful when they write C. > > This is an exceedingly broad statement, and it would have been good to provide > some pointers to these. > > > [...] > > You know that I don't appreciate how you handled this initial communication, > as I've told you a number of times. Now that I look at the history again, I'm > even more disappointed in how you handled this because there are some > pretty broad statements in here and language that could go either way. > > I've additionally recommended, as a developer and not in any kind of official > capacity, that we can't include if_wg in any future version of base. We > cannot > have our users being put at the risk of this kind of publicity if we > can't get security > advisories issued in time. Ports is a fine place, where security issues can be > addressed expeditiously and more in line with how the WireGuard project > chooses > to handle them. > > Thanks, > > Kyle Evans
Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
On Mon, Mar 15, 2021 at 9:47 AM Jason A. Donenfeld wrote: > > Hi everybody, > > [... snip ...] > > The first step was assessing the current state of the code the previous > developer had dumped into the tree. It was not pretty. I imagined > strange Internet voices jeering, “this is what gives C a bad name!” This was a highly unnecessary jab. > There were random sleeps added to “fix” race conditions This is true. > validation functions that just returned true I looked back at the history here just now, and I count one, wg_allowedip_valid, that pretty much got ripped out anyways. > catastrophic cryptographic vulnerabilities > whole parts of the protocol unimplemented I'm not qualified to speak about these two, which are perhaps the worst from the public's perspective. > kernel panics These are true. > security bypasses > overflows I only recall one of each. > random printf statements deep in crypto code This is true. > the most spectacular buffer overflows English is a crappy language, but this sounds like you're advertising more than one. There was one, and for the record to anyone watching to dispel "the word on the street" that it's potentially an RCE: based on what's happening I cannot imagine how you could usefully turn it into an RCE. Local privileged execution, 100%, but looking at it further, you've got to be pretty skilled to make it before killing the kernel elsewhere. > and the whole litany of awful things that go wrong when people aren’t > careful when they write C. This is an exceedingly broad statement, and it would have been good to provide some pointers to these. > [...] You know that I don't appreciate how you handled this initial communication, as I've told you a number of times. Now that I look at the history again, I'm even more disappointed in how you handled this because there are some pretty broad statements in here and language that could go either way. I've additionally recommended, as a developer and not in any kind of official capacity, that we can't include if_wg in any future version of base. We cannot have our users being put at the risk of this kind of publicity if we can't get security advisories issued in time. Ports is a fine place, where security issues can be addressed expeditiously and more in line with how the WireGuard project chooses to handle them. Thanks, Kyle Evans
Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
Hi Scott, It's certainly disappointing to receive your email like that. I see that you're really upset, and I think we have a shared interest in deescalating this. You've mentioned the desire to talk about this in public, so I'm responding in the announcement thread to you, as this seems like the appropriate venue for discussion. I've also CC'd the FreeBSD Security Officer. I've responded to your email and its threats in line mailing-list style below: On Mon, Mar 15, 2021 at 6:08 PM Scott Long wrote: > What you and Kyle did was tell the world that there are a number of > zero-day exploits in the code. You gave us no details until after the > fact, gave us no time to mitigate, correct, and publish before your > announcement and Kyle's code drop, and used the opportunity to > bash the code, and by extension us, for your own self-gain. I have no idea how you possibly have arrived at this here. FreeBSD 13 is currently at RC2 or RC3, if I recall correctly. Our whole motivation for working on this so tirelessly and with so much energy and devotion over the week was to get things fixed up before 13 was actually released. I dropped everything else I was working on to focus on this. I'm actually with my folks in Colorado and should be spending more time with them but the idea of FreeBSD 13 shipping with all those bugs seemed like a really important thing to address. Here's what we did, step by step: 1. We made a clone of the freebsd git repo where we could all push changes to master, to git.zx2c4.com like everything else. 2. We took turns taking big passes at the codebase, reading it, auditing it, fixing things. We did a lot of syncing and comparison with the OpenBSD code base. For example, in one step, we reordered the functions to match the flow of OpenBSD so we could see it more easily side by side. 3. We kept pushing commits like this in that manner. 4. We didn't know where we'd end up, but after a few days of this, the number of changes we had accumulated and fishy business we had observed was massive. 5. When we thought we reached a good resting point -- and when it became clear that our rapid fire changes were becoming too numerous -- we squashed that down to a summary commit, and Kyle committed it. 6. Seeing that the crypto needed to be re-accelerated, I published a call for help here -- https://lists.freebsd.org/pipermail/freebsd-hackers/2021-March/057076.html 7. I published wireguard-tools with the improved wg(8) command here -- https://lists.zx2c4.com/pipermail/wireguard/2021-March/006493.html 8. I sent a summary of everything to the mailing list here -- https://lists.zx2c4.com/pipermail/wireguard/2021-March/006494.html I also made sure to email a FreeBSD Security Officer, Colin, to keep him posted about the state of this, so he was kept apprised. Again, the whole idea was: two weeks until release! If I suggest the code is just disabled, people will be really upset, so instead we'll try to fix as much as fast as possible before the release. And actually, really focused sprints like that are fun and stimulating. We were all really enjoying writing code until the torrents of anger started coming from you about our efforts. (And I also learned that this isn't the first time you guys tried to intimidate an open source project -- see this summary of the opnSense defamatory website Netgate created -- https://opnsense.org/opnsense-com/ .) And in the process, too, I've tried to be in contact with you and Jim and let you know what our intentions are and to diffuse tensions. I spent time on a video call trying to describe to you some of the security things we found, in case it wasn't possible for you to use the new code right away. I've also made it abundantly clear to you how much I want to work WITH Netgate. When that Reddit thread cropped up, I offered to you multiple times to send a message to it telling people that we've spoken and it looks like you have a good plan and every things going to be okay, but you didn't respond to my offer. On Mon, Mar 15, 2021 at 6:08 PM Scott Long wrote: > None > of these actions reflect good-faith collaboration, and your statements > that you think that our work is "really cool" ring hollow. Maybe you > and Kyle are really naive and thought that this was how people > normally collaborate in the security and business worlds, and that > everyone would rush to praise you and shower you with contracts > and funding. And hey, maybe that’ll happen, you certainly have > made a splash. That of course, comes at our expense, and it’s > likely to be a pretty damaging one. Now with the Ars article, I’m > starting to wonder if this was a coordinated smear campaign. > Normally we don’t get this kind of attention. It’s going to be painful > to navigate our way out. I'm not sure how to address these types of... theories. I certainly AM acting in good faith. I can't possibly see how it benefits anybody to have WireGuard and
Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
That's awesome! Thanks a lot for your hard work and dedication on wireguard. I really appreciate what you do including your willingness to work with non Linux OSes. Cheers, Reto
[ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
Hi everybody, I’m pleased to announce that WireGuard now runs inside the FreeBSD kernel, with a driver called if_wg. It has full support of wg(8) and wg-quick(8) [5], as well as general integration into FreeBSD userland. Performance should be decent. The implementation in FreeBSD’s main branch should pretty much work, though it’s something of a so-so work in progress. To learn what I mean there, read on… Sometime ago, a popular firewall vendor tasked a developer with writing a WireGuard implementation for FreeBSD. They didn’t bother reaching out to the project. That’s okay, I figured, I’ll reach out and see if I can help and coordinate. What followed over the next year was a series of poor communications – messages unanswered, code reviews ignored, that kind of thing. Usually collaborations I’ve had with others have been full of excitement, but it just didn’t work out here. In the few discussions we were able to have, I did get across some key points, like, “you’ll save a bunch of time if you use the OpenBSD code as a starting point.” But mostly it seemed like a stop-and-go effort that the WireGuard project didn’t have much to do with. Then, at some point, whatever code laying around got merged into the FreeBSD tree and the developer tasked with writing it moved on. Fortunately, two weeks before FreeBSD 13.0 was due to be released, FreeBSD core developer Kyle Evans emailed the list about integrating wireguard-tools (wg(8) and such). In the ensuing discussion I mentioned that we really need to get the actual if_wg kernel implementation up to snuff. We took the conversation to IRC, and agreed that we should work on figuring out what to do before the release date. At the same time, Matt Dunwoodie, who worked on the OpenBSD implementation, also took a look at what had become of that implementation in FreeBSD. Over the next week, the three of us dug in and completely reworked the implementation from top to bottom, each one of us pushing commits and taking passes through the code to ensure correctness. The result was [6]. It was an incredible effort. The collaboration was very fast paced and exciting. Matt and Kyle are terrific programmers and fun to work with too. The first step was assessing the current state of the code the previous developer had dumped into the tree. It was not pretty. I imagined strange Internet voices jeering, “this is what gives C a bad name!” There were random sleeps added to “fix” race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code, the most spectacular buffer overflows, and the whole litany of awful things that go wrong when people aren’t careful when they write C. Or, more simply, it seems typical of what happens when code ships that wasn’t meant to. It was essentially an incomplete half-baked implementation – nothing close to something anybody would want on a production machine. Matt had to talk me out of just insisting they pull the code entirely, and rework it more slowly and carefully for the next release cycle. And he was right: nobody would have agreed to do that, and it would only have fostered frustration from folks genuinely enthusiastic about if_wg. So our one and only option was to iteratively improve it as fast as we could during the two weeks before release, and try to make it as simple and close as possible to OpenBSD so that we could benefit from the previous analysis done there. With that as our mission, we set out auditing and rewriting code. One curious thing of note is that there were 40,000 lines of optimized crypto implementations pulled out of the Linux kernel compat module but not really wired up correctly, and mangled beyond repair with mazes of Linux→FreeBSD ifdefs. I wound up replacing this with an 1,800 line file, crypto.c [1], containing all of the cryptographic primitives needed to implement WireGuard. Aside from its place in the FreeBSD story, this is kind of neat in its own right: these are simple, but fast enough, reference implementations. It’s not deliberately tiny or obfuscated like TweetNaCl is, yet is still just a single file, and the Curve25519 field arithmetic in it is formally verified. Maybe other projects will find use for it. Future releases will hopefully get rid of crypto.c and hook into FreeBSD’s already existing optimized implementations [4], which should give a nice performance boost, but given the time crunch, having something boring, safe, and simple seemed like the way to go. We reduced the project structure down to four C files – the aforementioned crypto.c, two files copied verbatim from OpenBSD – wg_noise.c and wg_cookie.c – and if_wg.c, the actual interface device driver implementation and protocol logic. The IPC interface was reworked as well, and wg(8) in the wireguard-tools package grew support for it (also rewritten from the original
