I'd like to keep all the issues on the subject in the single email thread, so here is what I copy-pasted from A.Korolev: > http://tracker.ceph.com/issues/6693
Also, I don't see any reason for keeping this conversation private, so I'm adding fuel-dev. Dmitry - any successes so far in your research? On Tue, Nov 19, 2013 at 1:53 AM, Dmitry Borodaenko <[email protected] > wrote: > The reason it's not a limitation for a volume backed instance is this > misguided conditional: > > https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3922 > > It assumes that only a volume-backed instance without ephemeral disks > can be live-migrated without shared storage. I also found many other > places in live migration code in Nova making the same assumption. What > I did not find so far is any real reason for shared storage to be > required for anything other than backing the instance's boot drive, > which is no longer a concern with the Ephemeral RBD patch. I'll try to > disable this and other similar checks and see if that makes live > migration work for an instance backed by RBD. > > If that's the case and there are no other blockers in nova, libvirt or > qemu, fixing this in Nova will indeed be relatively straightforward. > > -Dmitry > > On Mon, Nov 18, 2013 at 9:37 AM, Mike Scherbakov > <[email protected]> wrote: > > If instance boots from volume, Nova should not have such a limitation. > So if > > it has, it might be easier to fix Nova instead. > > > > > > On Mon, Nov 18, 2013 at 8:56 PM, Dmitry Borodaenko > > <[email protected]> wrote: > >> > >> I used patched packages built by OSCI team per Jira OSCI-773, there are > >> two more patches on the branch mentioned in the thread on ceph-users, I > >> still need to review and test those. > >> > >> We have seen the same error reported on this thread about shared > storage, > >> Nova requires /var/lib/nova to be shared between all compute nodes for > live > >> migrations, I am still waiting for Haomai to confirm whether he was > able to > >> overcome this limitation. If not, we will have to add glusterfs or > cephfs, > >> which is too much work for 4.0 timeframe. > >> > >> On Nov 18, 2013 1:32 AM, "Mike Scherbakov" <[email protected]> > >> wrote: > >>> > >>> Dmitry - sorry for late response. > >>> It is good news - I remember time when we were experimenting with DRBD, > >>> and now we will have Ceph, which should be a way better for the > purposes we > >>> need it for. > >>> > >>> > works with the patched Nova packages > >>> What patches did you apply? OSCI team already aware? > >>> > >>> As we merged havana into master, what are your estimates on enabling > all > >>> of this? We had meeting w/Roman, David, and we really want to have live > >>> migration enabled in 4.0 (see #6 here: > >>> > https://mirantis.jira.com/wiki/display/PRD/4.0+-+Mirantis+OpenStack+release+home+page > ) > >>> > >>> Thanks, > >>> > >>> > >>> On Wed, Nov 13, 2013 at 12:39 AM, Dmitry Borodaenko > >>> <[email protected]> wrote: > >>>> > >>>> Ephemeral storage in Ceph works with the patched Nova packages, we can > >>>> start updating our Ceph manifests as soon as we have havana branch > >>>> merged into fuel master! > >>>> > >>>> ---------- Forwarded message ---------- > >>>> From: Dmitry Borodaenko <[email protected]> > >>>> Date: Tue, Nov 12, 2013 at 12:38 PM > >>>> Subject: Re: Ephemeral RBD with Havana and Dumpling > >>>> To: [email protected] > >>>> > >>>> > >>>> And to answer my own question, I was missing a meaningful error > >>>> message: what the ObjectNotFound exception I got from librados didn't > >>>> tell me was that I didn't have the images keyring file in /etc/ceph/ > >>>> on my compute node. After 'ceph auth get-or-create client.images > > >>>> /etc/ceph/ceph.client.images.keyring' and reverting images caps back > >>>> to original state, it all works! > >>>> > >>>> On Tue, Nov 12, 2013 at 12:19 PM, Dmitry Borodaenko > >>>> <[email protected]> wrote: > >>>> > I can get ephemeral storage for Nova to work with RBD backend, but I > >>>> > don't understand why it only works with the admin cephx user? With a > >>>> > different user starting a VM fails, even if I set its caps to 'allow > >>>> > *'. > >>>> > > >>>> > Here's what I have in nova.conf: > >>>> > libvirt_images_type=rbd > >>>> > libvirt_images_rbd_pool=images > >>>> > rbd_secret_uuid=fd9a11cc-6995-10d7-feb4-d338d73a4399 > >>>> > rbd_user=images > >>>> > > >>>> > The secret UUID is defined following the same steps as for Cinder > and > >>>> > Glance: > >>>> > http://ceph.com/docs/master/rbd/libvirt/ > >>>> > > >>>> > BTW rbd_user option doesn't seem to be documented anywhere, is that > a > >>>> > documentation bug? > >>>> > > >>>> > And here's what 'ceph auth list' tells me about my cephx users: > >>>> > > >>>> > client.admin > >>>> > key: AQCoSX1SmIo0AxAAnz3NffHCMZxyvpz65vgRDg== > >>>> > caps: [mds] allow > >>>> > caps: [mon] allow * > >>>> > caps: [osd] allow * > >>>> > client.images > >>>> > key: AQC1hYJS0LQhDhAAn51jxI2XhMaLDSmssKjK+g== > >>>> > caps: [mds] allow > >>>> > caps: [mon] allow * > >>>> > caps: [osd] allow * > >>>> > client.volumes > >>>> > key: AQALSn1ScKruMhAAeSETeatPLxTOVdMIt10uRg== > >>>> > caps: [mon] allow r > >>>> > caps: [osd] allow class-read object_prefix rbd_children, > allow > >>>> > rwx pool=volumes, allow rx pool=images > >>>> > > >>>> > Setting rbd_user to images or volumes doesn't work. > >>>> > > >>>> > What am I missing? > >>>> > > >>>> > Thanks, > >>>> > > >>>> > -- > >>>> > Dmitry Borodaenko > >>>> > >>>> > >>>> > >>>> -- > >>>> Dmitry Borodaenko > >>>> > >>>> > >>>> -- > >>>> Dmitry Borodaenko > >>> > >>> > >>> > >>> > >>> -- > >>> Mike Scherbakov > > > > > > > > > > -- > > Mike Scherbakov > > > > -- > Dmitry Borodaenko > -- Mike Scherbakov
-- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
