It might be sufficient for now, that's true, but as far I understand the
issue, it gets harder and harder to maintain with each version. Drivers are
inside the kernel right now and having one huge program is always more
difficult, than having many smaller ones, that interact through some sort of
interface.
I know very well, that at this point Linux-libre is very stable and performs
very well and what not. But imagine what will happen in, say, five years,
when we will have much more drivers in the kernel. It will be like a fragile
house of playing cards, if one small thing goes wrong somewhere, you're in
very big trouble.
I believe Linus Torvalds made a huge contribution by starting Linux, but he
wasn't a professional programmer at that point in time! He was a very
talented student, with lots of time at his hands. No wonder he made some
design choices that aren't too perfect.