Hi, thank you both for such a high quality and educational thread. I wasn't aware of the possibility of configuring the LVM metadata in a host as read-only, but it fits beautifully with OpenNebula.
Mihály, would you like to share your experiences somewhere in the OpenNebula community? wiki.opennebula.org or blog.opennebula.org? I think many people can benefit from your setup. cheers, Jaime On Mon, Jan 21, 2013 at 6:58 PM, Miloš Kozák <[email protected]> wrote: > Oh snap, that sounds great I didn't know about that.. it makes all easier. > In this scenario only frontend can work with LVM, so no issues of > concurrent change. Only one last think to make it really safe against that. > Is there any way to suppress LVM changes from hosts, make it read only? And > let it RW at frontend? > > Thanks > > > Dne 21.1.2013 18:50, Mihály Héder napsal(a): > >> Hi, >> >> no, you don't have to do any of that. Also, nebula doesn't have to >> care about LVM metadata at all and therefore there is no corresponding >> function in it. At /etc/lvm there is no metadata, only configuration >> files. >> >> Lvm metadata simply sits somewhere at the beginning of your >> iscsi-shared disk, like a partition table. So it is on the storage >> that is accessed by all your hosts, and no distribution is necessary. >> Nebula frontend simply issues lvcreate, lvchange, etc, on this shared >> disk and those commands will manipulate the metadata. >> >> It is really LVM's internal business, many layers below opennebula. >> All you have to make sure that you don't run these commands >> concurrently from multiple hosts on the same iscsi-attached disk, >> because then they could interfere with each other. This setting is >> what you have to indicate in /etc/lvm on the server hosts. >> >> Cheers >> Mihály >> >> >> On 21 January 2013 18:37, Miloš Kozák <[email protected]> wrote: >> >>> Thank you. does it mean, that I can distribute metadata files located in >>> /etc/lvm on frontend onto other hosts and these hosts will see my logical >>> volumes? Is there any code in nebula which would provide it? Or I need to >>> update DS scripts to update/distribute LVM metadata among servers? >>> >>> Thanks, Milos >>> >>> Dne 21.1.2013 18:29, Mihály Héder napsal(a): >>> >>> Hi, >>>> >>>> lvm metadata[1] is simply stored on the disk. In the setup we are >>>> discussing this happens to be a shared virtual disk on the storage, >>>> so any other hosts that are attaching the same virtual disk should see >>>> the changes as they happen, provided that they re-read the disk. This >>>> re-reading step is what you can trigger with lvscan, but nowadays that >>>> seems to be unnecessary. For us it works with Centos 6.3 so I guess Sc >>>> Linux should be fine as well. >>>> >>>> Cheers >>>> Mihály >>>> >>>> >>>> [1] >>>> http://www.centos.org/docs/5/**html/Cluster_Logical_Volume_** >>>> Manager/lvm_metadata.html<http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html> >>>> >>>> On 21 January 2013 12:53, Miloš Kozák <[email protected]> wrote: >>>> >>>>> Hi, >>>>> thank you for great answer. As I wrote my objective is to avoid as much >>>>> of >>>>> clustering sw (pacemaker,..) as possible, so clvm is one of these >>>>> things >>>>> I >>>>> feel bad about them in my configuration.. Therefore I would rather let >>>>> nebula manage LVM metadata in the first place as I you wrote. Only one >>>>> last >>>>> thing I dont understand is a way nebula distributes LVM metadata? >>>>> >>>>> Is kernel in Scientific Linux 6.3 new enought to LVM issue you >>>>> mentioned? >>>>> >>>>> Thanks Milos >>>>> >>>>> >>>>> >>>>> >>>>> Dne 21.1.2013 12:34, Mihály Héder napsal(a): >>>>> >>>>> Hi! >>>>>> >>>>>> Last time we could test an Equalogic it did not have option for >>>>>> create/configure Virtual Disks inside in it by an API, so I think the >>>>>> iSCSI driver is not an alternative, as it would require a >>>>>> configuration step per virtual machine on the storage. >>>>>> >>>>>> However, you can use your storage just fine in a shared LVM scenario. >>>>>> You need to consider two different things: >>>>>> -the LVM metadata, and the actual VM data on the partitions. It is >>>>>> true, that the concurrent modification of the metadata should be >>>>>> avoided as in theory it can damage the whole virtual group. You could >>>>>> use clvm which avoids that by clustered locking, and then every >>>>>> participating machine can safely create/modify/delete LV-s. However, >>>>>> in a nebula setup this is not necessary in every case: you can make >>>>>> the LVM metadata read only on your host servers, and let only the >>>>>> frontend modify it. Then it can use local locking that does not >>>>>> require clvm. >>>>>> -of course the host servers can write the data inside the partitions >>>>>> regardless that the metadata is read-only for them. It should work >>>>>> just fine as long as you don't start two VMs for one partition. >>>>>> >>>>>> We are running this setup with a dual controller Dell MD3600 storage >>>>>> without issues so far. Before that, we used to do the same with XEN >>>>>> machines for years on an older EMC (that was before nebula). Now with >>>>>> nebula we have been using a home-grown module for doing that, which I >>>>>> can send you any time - we plan to submit that as a feature >>>>>> enhancement anyway. Also, there seems to be a similar shared LVM >>>>>> module in the nebula upstream which we could not get to work yet, but >>>>>> did not try much. >>>>>> >>>>>> The plus side of this setup is that you can make live migration work >>>>>> nicely. There are two points to consider however: once you set the LVM >>>>>> metadata read-only you wont be able to modify the local LVMs in your >>>>>> servers, if there are any. Also, in older kernels, when you modified >>>>>> the LVM on one machine the others did not get notified about the >>>>>> changes, so you had to issue an lvs command. However in new kernels >>>>>> this issue seems to be solved, the LVs get instantly updated. I don't >>>>>> know when and what exactly changed though. >>>>>> >>>>>> Cheers >>>>>> Mihály Héder >>>>>> MTA SZTAKI ITAK >>>>>> >>>>>> On 18 January 2013 08:57, Miloš Kozák <[email protected]> wrote: >>>>>> >>>>>>> Hi, I am setting up a small installation of opennebula with >>>>>>> sharedstorage >>>>>>> using iSCSI. THe storage is Equilogic EMC with two controllers. >>>>>>> Nowadays >>>>>>> we >>>>>>> have only two host servers so we use backed direct connection between >>>>>>> storage and each server, see attachment. For this purpose we set up >>>>>>> dm-multipath. Cause in the future we want to add other servers and >>>>>>> some >>>>>>> other technology will be necessary in the network segment. Thesedays >>>>>>> we >>>>>>> try >>>>>>> to make it as same as possible with future topology from protocols >>>>>>> point >>>>>>> of >>>>>>> view. >>>>>>> >>>>>>> My question is related to the way how to define datastore, which >>>>>>> driver >>>>>>> and >>>>>>> TM is the best and which? >>>>>>> >>>>>>> My primal objective is to avoid GFS2 or any other cluster filesystem >>>>>>> I >>>>>>> would >>>>>>> prefer to keep datastore as block devices. Only option I see is to >>>>>>> use >>>>>>> LVM >>>>>>> but I worry about concurent writes isn't it a problem? I was >>>>>>> googling a >>>>>>> bit >>>>>>> and I found I would need to set up clvm - is it really necessary? >>>>>>> >>>>>>> Or is better to use iSCSI driver, drop the dm-multipath and hope? >>>>>>> >>>>>>> Thanks, Milos >>>>>>> >>>>>>> ______________________________**_________________ >>>>>>> Users mailing list >>>>>>> [email protected] >>>>>>> http://lists.opennebula.org/**listinfo.cgi/users-opennebula.**org<http://lists.opennebula.org/listinfo.cgi/users-opennebula.org> >>>>>>> >>>>>>> ______________________________**_________________ >>> Users mailing list >>> [email protected] >>> http://lists.opennebula.org/**listinfo.cgi/users-opennebula.**org<http://lists.opennebula.org/listinfo.cgi/users-opennebula.org> >>> >> > ______________________________**_________________ > Users mailing list > [email protected] > http://lists.opennebula.org/**listinfo.cgi/users-opennebula.**org<http://lists.opennebula.org/listinfo.cgi/users-opennebula.org> > -- Jaime Melis Project Engineer OpenNebula - The Open Source Toolkit for Cloud Computing www.OpenNebula.org | [email protected]
_______________________________________________ Users mailing list [email protected] http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
