Thanks for the thoughtful feedback. Actually, no, I'll say that this is the
best feedback I've gotten, so mega thanks. I'd heartily expand my
appreciation to Albert, Andy, and others. And let me assure you that my reply
isn't about anyone in particular, but the situation as I see it. I will say,
however, that your mindset - that of an expert who knows his subject cold and
will defend it with precision and passion - is exactly why I wrote the paper
and have my current position. Please allow me to try and explain.
On Feb 19, 2013, at 10:16 AM, Jarrod B Johnson <jbjoh...@us.ibm.com> wrote:
> Overall, most of the concerns are less about the fundamentals of IPMI and
> more about the general state of service processors in general.
>
Well, in part. As I wrote in the paper it's a confluence between the spec, how
it's implemented and expanded by the various vendors, and how it's actually
used.
If by service processor you mean BMC, yes, that's the most egregious offender.
However, it wouldn't exist if it weren't for IPMI, and the power it has to have
to implement the spec, as well as some of the design decision of the spec
drives decisions that harm the security of the BMC and everything around it.
In my mind there's almost nothing that has the power to be misused in the
computer than the BMC (I'm sure now someone will now inform me that there's
actually another tinier processor on the BMC that drives it and… ;))
But then we have to look at how it's actually used… more in a sec.
> [unfair to RAKP and...]
>
> "The reusable passwords used for IPMI authentication are saved in clear text
> in flash memory" True, but it is a shared secret mechanism that permits the
> client to never have to transmit the password on the wire (which might be
> debated to be a more vulnerable place). It also provides for implicit mutual
> authentication if the client is strict enough in what it will accept from an
> IPMI device. One-way hashed passwords have their own problems, notably that
> a spoofed server has a pretty high chance of acquiring secrets through MITM.
> Ideally, you correctly have everything private-public key based (which
> admittedly has no consideration in the IPMI spec), the gap in practice is
> that the host public key is frequently taken at face value. Shared secret at
> least makes the admin unwittingly set up mutual authentication. The shared
> secret scenario is troublesome as it limits the security for those that know
> and care about security, but has some tendency to instill better practices
> for those that care or know less, but do at least care enough to 'set a
> password'. My desire in this front would be an additional public/private key
> based auth mechanism to allow diligent users and software to implement
> something more unambiguously secure. However, such diligent users can limit
> shared secret scope already through unique key data everywhere (though no one
> seems willing).
>
Overall in theory you're correct (I might quibble a bit about the details, but
it's 6 of one vs. a half-dozen of another.) My argument, however, is that this
isn't how people use it in the real world. When you have over 100K servers all
with the same password over a period of years you might well call them foolish,
but this isn't uncommon at all. People are selling servers on eBay with no
notion that it might be a problem.
I'll revisit how I was unfair to RAKP - but while I think you know I agree
about the xCAT and other implementations that offer this feature, I've never
heard of a single place ever using it (I'm sure it's out there somewhere, but
it certainly doesn't appear to be popular.) I'd love to simply say go that
way, but as you've said in the past, people want to cling to their
preconceptions and want to hold onto those passwords. I just checked: 32 hits
on "xCAT" & "RAKP" on google, and it looks like 5 of those links are actually
valid. Adding security trims the list to 2 links. ("kq" "rakp" "security"
"ipmi" gets zero hits!) There are no implementation guides or usage stories or
questions. How do you expect people to know what to do? (I hope you know that
this isn't an attack on you or xCAT in particular; I hold both in high esteem,
I'm simply trying to point to the current state of affairs.)
I would say the major issue I have with the IPMI community in general is that
there seemingly has been no effort to communicate to people what to avoid, or
give out some best practices on how to use the stuff wisely. I've repeatedly
said I've never even (consciously) heard of IPMI prior to a few months ago. I
was the security architect of the largest security company in the world and I
knew only vaguely of the management network, and the various iDRACs, IMM's,
etc. of the world, but mostly it was one of those "ops-things". You could
rightfully say I was ignorant, and I was, but let me assure you that I wasn't
alone. No one knows anything about this stuff WRT security.
That no one in the community has ever tried to even write up a list of best
security practices, or how to dispose of a motherboard that has a BMC (or tell
people it's an issue), let alone jot down a simple scanner to check for such
issues, is simply astounding to me, and it points to the serious ignorance of
the non-IPMI community. I wrote the first audit tool ever put on the net, and
coauthored first security scanner anyone had heard of, and I thought my little
configuration checking days were long past, but it seems like we keep doing the
same things over and again. I've no illusions that my little effort will make
a big difference, but I am trying to spread some education (hopefully not
scaremongering or being inaccurate while doing so) so people can make more
informed decisions.
> "it has to ask a potentially compromised BMC to answer truthfully." This can
> be controlled vendor to vendor. E.g. an IBM IMMv2 has the entire boot
> process and filesystem signed in such a way that malicious attempts to
> manipulate it are thwarted. This exacerbates the 'precludes customer for
> applying fixes prior to the vendor' issue, but it does mean that commands
> shall work as intended by the manufacturer. The key here being to pay
> attention to how the vendor protects the integrity of their implementation.
>
It could be that IBM does a good job, and you alluded to that earlier (and IBM
has wonderful documentation as well). That said… there are lots of things that
are "work as intended" by vendors that are really freakin' aggravating ;) And
look: IBM has a list of features several pages long. Don't you think it's
possible that one of those might be a security problem? And if so, what do I
as a customer do? I can't patch it, I can audit the thing, I can't safeguard
it, I can't log it, I can't remove the feature (it can be simply turned back
on.) I'm just hosed.
And if I really can't subvert the boot process (for sake of argument; no one
has seen the stuff and its undocumented, so I can't comment), that's fine - I
don't care about the boot process, that's only a single vector. BMCs typically
have very long uptimes - I'll just put something ephemerally that subverts
everything, and renew it if a boot ever happens. IBM could have carved the OS
in rock, but I don't think think they're immune from a bazillion security
issues in the OS and applications. This is not a solved problem. And the BMC
- at least with some IBM things that I've seen it's made by Renesas, which has
a semiconductor design center and manufacturing things in China. Do you really
think you can completely trust that it only does what its spec'd for?
May I remind you of the paper where IBM's undocumented feature to generate
SMI's from the BMC was used to have absolutely control over memory and virtual
machines? This feature, I was informed, was gotten under an NDA by the
researchers. What other backdoors are there? A couple of months ago I found a
backdoor to login to the BMC from one of the largest server vendors in the
world (name elided since I'm working with them on removing and warning
customers about it); did they put it in? Did the OEM'ers? Who knows? But
hearing a vendor say "trust us, your security is safe with us", when it's been
demonstrably shown that they can't be trusted, doesn't mollify me.
By making the BMC a black box with immense power the vendors are begging or
nearly designing the BMCs for abuse.
> In general, my recommendation would be for environments worried about
> security to stop caring so much about direct human interaction and let higher
> order software manage unique keys everywhere there is a desire to use
> 'passwords' today. Someone recovers the BMC password off an ebay discarded
> box? No big deal, the scope of that key was unique. Someone tricks the
> security framework responsible for key coordination into divulging a new key?
> No heightened privilege, but perhaps a way to remove all access to a BMC
> which is less bad.
>
The problem isn't with IPMI, the problem isn't with the vendors who make the
BMCs and add on features, the problem is not how its used - you can't untangle
any of these things and claim "but it's not my fault!"
I've a different proposal - open up the BMC. Let people inside. Let people
understand what's going on, and allow them to take matters into their own
hands, and perhaps even make things better. I'm not an open source zealot, but
come on, give people a chance, at least (not you, talking to vendors here!)
I'll close with a historical bit - there is a long history of systems that use
closed source and design, are widely used, and it's not a pretty sight
(security speaking. Check out this paper on automobile security!)
dan------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Ipmitool-devel mailing list
Ipmitool-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ipmitool-devel
