Dear Alejandro, every time we run a script to prune the _base directory in our enviroments we raise a pray to Santa Bárbara ( http://es.wikipedia.org/wiki/B%C3%A1rbara_(m%C3%A1rtir)#Patronazgo). Yes, _base folder with shared storage is a risky business.
Have you considered not using _base backed images? This article can help: http://www.pixelbeat.org/docs/openstack_libvirt_images/ If I'm right and you are using some kind of NFS device with deduplication, may be it can be an option. Cheers Diego -- Diego Parrilla <http://www.stackops.com/>*CEO* *www.stackops.com <http://www.stackops.com/> | * [email protected] | +34 91 005-2164 | skype:diegoparrilla On Wed, Apr 2, 2014 at 8:04 PM, Alejandro Comisario < [email protected]> wrote: > Hi guys ... > We have a pretty big openstack environment and we use a shared NFS to > populate backing file directory ( the famous _base directory located > on /var/lib/nova/instances/_base ) due to a human error, the backing > file used by thousands of guests was deleted, causing this guests to > go read-only filesystem in a second. > > Till that moment we were convinced to use the _base directory as a > shared NFS because: > > * spawning a new ami gives total visibility to the whole cloud making > instances take nothing to boot despite the nova region > * ease glance workload > * easiest management no having to replicate files constantly not > pushing bandwidth usage internally > > But after this really big issue, and after what took us to recover > from this, we were thinking about how to protect against this kind of > "single point of failure". > Our first aproach this days was to put Read Only the NFS share, making > impossible for computes ( and humans ) to write to that directory, > giving permision to just one compute whos the one responsible to spawn > an instance from a new ami and write the file to the directory, still > ... the storage keeps being the SPOF. > > So, we are handling the possibility of having the used backing files > LOCAL on every compute ( +1K hosts ) and reduce the failure chances to > the minimum, obviously, with a pararell talk about what technology to > use to keep data replicated among computes when a new ami is launched, > launching times, performance matters on compute nodes having to store > backing files locally, etc. > > This make me realize, i have a huge comminity behind openstack, so > wanted to ear from it: > > * what are your thoughts about what happened / what we are thinking right > now ? > * how does other users manage the backing file ( _base ) directory > having all this considerations on big openstack deployments ? > > I will be thrilled to read from other users, experiences and thoughts. > > As allways, best. > Alejandro > > _______________________________________________ > OpenStack-operators mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
