Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting
On 2013-05-27 11:29:31 +1200 (+1200), Robert Collins wrote: On 27 May 2013 11:02, Chris Bartels ch...@christopherbartels.com wrote: [...] Couldn't I re-flash the BIOS between each tenant to be sure there isn't any problem with it? Unless you flash the BIOS with separate hardware (not by running the flasher on the potentially compromised hardware itself), no. And even then you'll need to be sure you flash every single EEPROM, not just the system board BIOS, and you'll need to make sure you catch any that have been toggled into readonly mode by an attacker and pull and replace them. Note that a simple examination of device drivers / system firmware won't necessarily cover every power on EEPROM in the system :). [...] Note that this is a not-often-talked-about security risk throughout the industry, it's not just an OpenStack baremetal issue. Many (most? all?) data center hosting companies reuse servers between short-term dedicated hardware tenants without doing much more than a disk wipe and typical BIOS upgrade. For that matter, there's a similar risk when purchasing used or refurbished hardware... or even new hardware, depending on how much you trust the procurement chain (but in that case there's at least readily available legal recourse if you find out the manufacturer/distributor/carrier intentionally engaged in compromising the hardware). Some companies are aware of these possibilities and may have simply decided their risk analysis shows it's not worth mitigating in their situations, but many are not aware that this attack surface even exists to begin with. Now, whether can you trust that the computer manufacturing and software industries can solve this problem (Trusted Computing and so on) is another question entirely. -- Jeremy Stanley ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting
On 28 May 2013 01:23, Jeremy Stanley fu...@yuggoth.org wrote: Note that this is a not-often-talked-about security risk throughout the industry, it's not just an OpenStack baremetal issue. Indeed! However while it was obscure, esoteric and largely unknown 20 years ago, it's now part of the standard risk profile from a security perspective - it's precisely what UEFI secure boot targets... The current bleeding edge of attacks is factory compromised bus devices, with stock firmware having a hostile mode that isn't even compromised, but is built-in. *That* I'm willing to ignore for now:). Well, other than buying good hardware :). Many (most? all?) data center hosting companies reuse servers between short-term dedicated hardware tenants without doing much more than a disk wipe and typical BIOS upgrade. For that matter, there's a similar risk when purchasing used or refurbished hardware... or even new hardware, depending on how much you trust the procurement chain (but in that case there's at least readily available legal recourse if you find out the manufacturer/distributor/carrier intentionally engaged in compromising the hardware). Yup :). Some companies are aware of these possibilities and may have simply decided their risk analysis shows it's not worth mitigating in their situations, but many are not aware that this attack surface even exists to begin with. Now, whether can you trust that the computer manufacturing and software industries can solve this problem (Trusted Computing and so on) is another question entirely. Yeah :(. It's not clear that adding a whole new OS to the boot process is the right answer, but it's the only one with widespread adoption so far. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Cloud Services ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting
I'll just use full server sized VMs made of KVM disclose in my product detail page that the dedicated servers are comprised of this design to mitigate the attack vector we're speaking of. -Original Message- From: Openstack [mailto:openstack-bounces+chris=christopherbartels@lists.launchpad.net] On Behalf Of Robert Collins Sent: Monday, May 27, 2013 2:37 PM To: Jeremy Stanley Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting On 28 May 2013 01:23, Jeremy Stanley fu...@yuggoth.org wrote: Note that this is a not-often-talked-about security risk throughout the industry, it's not just an OpenStack baremetal issue. Indeed! However while it was obscure, esoteric and largely unknown 20 years ago, it's now part of the standard risk profile from a security perspective - it's precisely what UEFI secure boot targets... The current bleeding edge of attacks is factory compromised bus devices, with stock firmware having a hostile mode that isn't even compromised, but is built-in. *That* I'm willing to ignore for now:). Well, other than buying good hardware :). Many (most? all?) data center hosting companies reuse servers between short-term dedicated hardware tenants without doing much more than a disk wipe and typical BIOS upgrade. For that matter, there's a similar risk when purchasing used or refurbished hardware... or even new hardware, depending on how much you trust the procurement chain (but in that case there's at least readily available legal recourse if you find out the manufacturer/distributor/carrier intentionally engaged in compromising the hardware). Yup :). Some companies are aware of these possibilities and may have simply decided their risk analysis shows it's not worth mitigating in their situations, but many are not aware that this attack surface even exists to begin with. Now, whether can you trust that the computer manufacturing and software industries can solve this problem (Trusted Computing and so on) is another question entirely. Yeah :(. It's not clear that adding a whole new OS to the boot process is the right answer, but it's the only one with widespread adoption so far. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Cloud Services ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Using openstack to manage dedicated servers in a service provider setting
Hi, I'm working on a startup that aims to rent dedicated servers to tech startups, and I would like to use OpenStack to manage the servers I rent out. I saw on the OpenStack Foundation YouTube channel there was a video there about using OpenStack to manage bare metal, but the presenter in the video had such a strong accent that I couldn't understand anything they were saying didn't learn a thing from the video. I'm interested in learning the basics about what OpenStack can do with bare metal, and what its limitations are when deployed in this manner. Would someone kindly direct me to resources that would explain this to me please? If anyone else has experience in this use case I'd love to hear from you to learn from your experience. Thank you. Regards, Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting
Hi, Look at these pages. I dont know if there is any difference and I didnt have done bare metal deployment. https://wiki.openstack.org/wiki/Baremetal https://wiki.openstack.org/wiki/GeneralBareMetalProvisioningFramework Rusty On Sun, May 26, 2013 at 9:01 PM, Chris Bartels ch...@christopherbartels.com wrote: Hi, ** ** I’m working on a startup that aims to rent dedicated servers to tech startups, and I would like to use OpenStack to manage the servers I rent out. ** ** I saw on the OpenStack Foundation YouTube channel there was a video there about using OpenStack to manage bare metal, but the presenter in the video had such a strong accent that I couldn’t understand anything they were saying didn’t learn a thing from the video. ** ** I’m interested in learning the basics about what OpenStack can do with bare metal, and what its limitations are when deployed in this manner. ** ** Would someone kindly direct me to resources that would explain this to me please? ** ** If anyone else has experience in this use case I’d love to hear from you to learn from your experience. ** ** Thank you. ** ** Regards, Chris ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting
On 27 May 2013 07:01, Chris Bartels ch...@christopherbartels.com wrote: Hi, I’m working on a startup that aims to rent dedicated servers to tech startups, and I would like to use OpenStack to manage the servers I rent out. I saw on the OpenStack Foundation YouTube channel there was a video there about using OpenStack to manage bare metal, but the presenter in the video had such a strong accent that I couldn’t understand anything they were saying didn’t learn a thing from the video. Which video in particular ? There are a number of groups who have been taped presenting on bare metal things. I'm part of a team working on using OpenStack [baremetal] to deploy OpenStack [virtual]. So I'm happy to answer any questions. http://www.openstack.org/summit/portland-2013/session-videos/presentation/provisioning-bare-metal-with-openstack is a good video on the baremetal layer, which is what will interest you I think. That said, there are huge security issues with repurposing baremetal from one tenant to another: in the absence of UEFI secure boot it is possible for the prior tenant to inject hostile boot-time firmware into physical devices that have software flashable EEPROMS. Currently in OpenStack we have no mitigation for this at all: so I would very strongly advise against using OpenStack baremetal to provide dedicated machines. What I suggest you do instead is provide KVM instances where the KVM flavor size exactly matches the physical machines - so youre tenants have the full capacity of the machine, and only the [low] overhead of the KVM layer. This has a -much- better security story. You could use TripleO - OpenStack on OpenStack - to manage this setup. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Cloud Services ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting
I had originally wanted to deploy full server sized KVM instances and rent VPS' that way, but it was brought to my attention that a certain market segment which I'm targeting- tech startups, who are testing apps on these rentals, are unable to get reliable metrics because of the software between their app the hardware. So I've shifted gears to offering dedicated servers instead, to remove that layer of interference. Couldn't I re-flash the BIOS between each tenant to be sure there isn't any problem with it? -Original Message- From: Robert Collins [mailto:robe...@robertcollins.net] Sent: Sunday, May 26, 2013 4:56 PM To: ch...@christopherbartels.com Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting On 27 May 2013 07:01, Chris Bartels ch...@christopherbartels.com wrote: Hi, I'm working on a startup that aims to rent dedicated servers to tech startups, and I would like to use OpenStack to manage the servers I rent out. I saw on the OpenStack Foundation YouTube channel there was a video there about using OpenStack to manage bare metal, but the presenter in the video had such a strong accent that I couldn't understand anything they were saying didn't learn a thing from the video. Which video in particular ? There are a number of groups who have been taped presenting on bare metal things. I'm part of a team working on using OpenStack [baremetal] to deploy OpenStack [virtual]. So I'm happy to answer any questions. http://www.openstack.org/summit/portland-2013/session-videos/presentation/pr ovisioning-bare-metal-with-openstack is a good video on the baremetal layer, which is what will interest you I think. That said, there are huge security issues with repurposing baremetal from one tenant to another: in the absence of UEFI secure boot it is possible for the prior tenant to inject hostile boot-time firmware into physical devices that have software flashable EEPROMS. Currently in OpenStack we have no mitigation for this at all: so I would very strongly advise against using OpenStack baremetal to provide dedicated machines. What I suggest you do instead is provide KVM instances where the KVM flavor size exactly matches the physical machines - so youre tenants have the full capacity of the machine, and only the [low] overhead of the KVM layer. This has a -much- better security story. You could use TripleO - OpenStack on OpenStack - to manage this setup. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Cloud Services ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Using openstack to manage dedicated servers in a service provider setting
On 27 May 2013 11:02, Chris Bartels ch...@christopherbartels.com wrote: I had originally wanted to deploy full server sized KVM instances and rent VPS' that way, but it was brought to my attention that a certain market segment which I'm targeting- tech startups, who are testing apps on these rentals, are unable to get reliable metrics because of the software between their app the hardware. So I've shifted gears to offering dedicated servers instead, to remove that layer of interference. Couldn't I re-flash the BIOS between each tenant to be sure there isn't any problem with it? Unless you flash the BIOS with separate hardware (not by running the flasher on the potentially compromised hardware itself), no. And even then you'll need to be sure you flash every single EEPROM, not just the system board BIOS, and you'll need to make sure you catch any that have been toggled into readonly mode by an attacker and pull and replace them. Note that a simple examination of device drivers / system firmware won't necessarily cover every power on EEPROM in the system :). As for your tech startups, unless they are going to be running on bare metal - e.g. their competitive advantage is going to be datacentre operations efficiency - they are most likely going to be deploying on a virtual substrate themselves. I would validate the proported inability to get good metrics : give them a kvm instance with a reserved machine, and the only noise will be kvm platform management (vs other tenants). That should be able to deliver very robust (within a few %) estimates of capacity and performance for nearly any workload. The cases where it cannot - well, find those cases. To do such a validation, I would pick a metric you think would be distorted - e.g. IOPS - and find or write a bench test for it, then use that from within the KVM instance on a machine (running with the full machine, raw backing devices, etc) and then again from within the machine with no KVM layer. For the metrics are invalid, you'll need to obtain not just different results, but non-predictably different results. E.g. consistently 30% would be a nuisance but still allow prediction for behaviour on bare metal. But sometimes 1% slower and sometimes 40% slower would make it much harder to use. HTH, Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Cloud Services ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
