Trust me- you're not the first to have ever been concerned about SELinux. People have had a look at it before, and there really is nothing to worry about. If there was (since it's GPL) there would have been an uproar, and it would have been modified or rejected by the GNU/Linux community.
Sure, it comes from the NSA. But it's also under the GPL. There is nothing to fear.
