Thanks Ricky, I've added reviewing this to my todo list
Dan ----- Original Message ----- > From: "Ricky Hopper" <[email protected]> > To: "Dan Yasny" <[email protected]> > Cc: [email protected] > Sent: Thursday, 2 August, 2012 5:37:10 PM > Subject: Re: [Engine-devel] Domain rescan action question > > In the interest of good discussion, we've put up a feature page for > this > feature (http://wiki.ovirt.org/wiki/Features/Domain_Scan), which > links to > a talk page where modifications can be proposed to how I've laid out > the > feature. So far, it covers how the query works and which commands > will > come about to implement it. I'd appreciate it if anyone concerned > could > check this out and make any changes as they see fit so we can get > going > with the coding. > > - Ricky > > On 8/1/12 2:35 PM, "Dan Yasny" <[email protected]> wrote: > > > > > > >----- Original Message ----- > >> From: "Ricky Hopper" <[email protected]> > >> To: "Dan Yasny" <[email protected]> > >> Cc: [email protected], "Itamar Heim" <[email protected]>, > >> "Andrew > >>Cathrow" <[email protected]> > >> Sent: Wednesday, 1 August, 2012 8:36:09 PM > >> Subject: Re: [Engine-devel] Domain rescan action question > >> > >> > >> > >> On 8/1/12 10:05 AM, "Dan Yasny" <[email protected]> wrote: > >> > >> > > >> > > >> >----- Original Message ----- > >> >> From: "Ricky Hopper" <[email protected]> > >> >> To: "Dan Yasny" <[email protected]> > >> >> Cc: [email protected], "Itamar Heim" <[email protected]>, > >> >> "Andrew > >> >>Cathrow" <[email protected]> > >> >> Sent: Wednesday, 1 August, 2012 4:56:45 PM > >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> > >> >> > >> >> > >> >> On 8/1/12 9:42 AM, "Dan Yasny" <[email protected]> wrote: > >> >> > >> >> > > >> >> > > >> >> >----- Original Message ----- > >> >> >> From: "Ricky Hopper" <[email protected]> > >> >> >> To: "Dan Yasny" <[email protected]>, "Andrew Cathrow" > >> >> >><[email protected]> > >> >> >> Cc: [email protected], "Itamar Heim" > >> >> >> <[email protected]>, > >> >> >> "Ricky > >> >> >>Hopper" <[email protected]> > >> >> >> Sent: Wednesday, 1 August, 2012 4:34:53 PM > >> >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> > >> >> >> > >> >> >> > >> >> >> On 8/1/12 5:59 AM, "Dan Yasny" <[email protected]> wrote: > >> >> >> > >> >> >> > > >> >> >> > > >> >> >> >----- Original Message ----- > >> >> >> >> From: "Andrew Cathrow" <[email protected]> > >> >> >> >> To: "Itamar Heim" <[email protected]>, "Dan Yasny" > >> >> >> >> <[email protected]>, > >> >> >> >>"Ricky Hopper" <[email protected]> > >> >> >> >> Cc: [email protected] > >> >> >> >> Sent: Wednesday, 1 August, 2012 12:24:42 AM > >> >> >> >> Subject: Re: [Engine-devel] Domain rescan action question > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> ----- Original Message ----- > >> >> >> >> > From: "Itamar Heim" <[email protected]> > >> >> >> >> > To: "Ricky Hopper" <[email protected]> > >> >> >> >> > Cc: [email protected] > >> >> >> >> > Sent: Tuesday, July 31, 2012 4:44:34 PM > >> >> >> >> > Subject: Re: [Engine-devel] Domain rescan action > >> >> >> >> > question > >> >> >> >> > > >> >> >> >> > On 07/31/2012 11:30 PM, Hopper, Ricky wrote: > >> >> >> >> > > Hey all, > >> >> >> >> > > > >> >> >> >> > > As I'm making progress with the domain rescan > >> >> >> >> > > functionality, > >> >> >> >> > > I've > >> >> >> >> > > realized that I'm unsure what to do with any disks > >> >> >> >> > > that > >> >> >> >> > > are > >> >> >> >> > > detected on > >> >> >> >> > > the domain. Should I add them back into the database > >> >> >> >> > > to > >> >> >> >> > > be > >> >> >> >> > > listed > >> >> >> >> > > as > >> >> >> >> > > floating disks, or should I just return a list of > >> >> >> >> > > disk > >> >> >> >> > > images > >> >> >> >> > > to > >> >> >> >> > > be > >> >> >> >> > > attached to whatever the caller of the query needs? > >> >> >> >> > > > >> >> >> >> > > - Ricky > >> >> >> >> > > >> >> >> >> > i'm not sure they should be added automatically. > >> >> >> >> > I think a dialog[1] showing orphan disks/images on the > >> >> >> >> > storage > >> >> >> >> > domain > >> >> >> >> > for user to choose which to import as 'floating' disks > >> >> >> >> > would > >> >> >> >> > be > >> >> >> >> > better > >> >> >> >> > than auto importing them. > >> >> >> >> > > >> >> >> >> > there is also the reverse of flagging existing disks as > >> >> >> >> > 'missing' > >> >> >> >> > in > >> >> >> >> > storage? > >> >> >> >> > > >> >> >> >> > >> >> >> >> Perhaps we should start a feature page to discuss and > >> >> >> >> better > >> >> >> >> scope > >> >> >> >> it. > >> >> >> >> There is a feature page that we could expand, it doesn't > >> >> >> >> discuss > >> >> >> >> the > >> >> >> >> notion of importing those disks which is certainly > >> >> >> >> something > >> >> >> >> we > >> >> >> >> need > >> >> >> >> to address. > >> >> >> >> > >> >> >> >> > >> >> >> >> http://wiki.ovirt.org/wiki/Features/Orphaned_Images > >> >> >> > > >> >> >> >The original idea was to scan the storage domains and > >> >> >> >compare > >> >> >> >the > >> >> >> >images > >> >> >> >lists to the database, thus getting a list of images no > >> >> >> >longer > >> >> >> >relevant > >> >> >> >and scrubbing the storage. This will actually be addressed > >> >> >> >properly > >> >> >> >in > >> >> >> >the future (Ayal can elaborate on that) but for now this is > >> >> >> >needed > >> >> >> >at > >> >> >> >least for that use case. > >> >> >> > > >> >> >> > > >> >> >> >As I understand, the conversation here is about trying to > >> >> >> >take > >> >> >> >an > >> >> >> >already > >> >> >> >populated SD (from another setup I suppose), scanning it > >> >> >> >and > >> >> >> >putting > >> >> >> >it > >> >> >> >into RHEV? > >> >> >> > >> >> >> As I understood it, the purpose of this functionality wasn't > >> >> >> to > >> >> >> find > >> >> >> images which should be removed from storage, but to find > >> >> >> images > >> >> >> on > >> >> >> the > >> >> >> domain that oVirt was unaware of and importing them for use > >> >> >> (for > >> >> >> instance, > >> >> >> if a disk was created outside of oVirt on the domain). If > >> >> >> one > >> >> >> of > >> >> >> the > >> >> >> use > >> >> >> cases for this feature is also the orphaned images mentioned > >> >> >> on > >> >> >> the > >> >> >> feature page, that may expand the functionality into a > >> >> >> separate > >> >> >> domain > >> >> >> scrub and storage import, both of which would call the > >> >> >> rescan > >> >> >> (meaning the > >> >> >> rescan would not actually add to the database, but instead > >> >> >> return > >> >> >> a > >> >> >> list > >> >> >> of "orphaned" disk images). > >> >> >> > >> >> >> Another solution would be to import all disk images into the > >> >> >> database > >> >> >> either way, and let the user delete any orphaned images from > >> >> >> the > >> >> >> GUI. > >> >> > > >> >> >I think are nice to have, but the problem with the scanning is > >> >> >that > >> >> >if > >> >> >we're not scanning a master domain or an export domain, all we > >> >> >will > >> >> >see > >> >> >is a bunch of images with no context or even hints as to where > >> >> >they > >> >> >belong. The data that makes it all usable is in the engine > >> >> >database > >> >> >and > >> >> >in the ovf files on the master domain. > >> >> > > >> >> >This is why I stopped at the orphaned images part of the > >> >> >feature > >> >> >- > >> >> >because there it's feasible, I would rely on the engine > >> >> >database > >> >> >for > >> >> >image ID comparisons. > >> >> > > >> >> >If we present a user with a list of nameless disks, I doubt it > >> >> >will > >> >> >be of > >> >> >any use. > >> >> > >> >> The way this would work is by comparing a list of disk images > >> >> from > >> >> vdsm > >> >> and from oVirt's database, finding the ones vdsm returns that > >> >> oVirt > >> >> doesn't have, and then either adding or returning those images. > >> >> So > >> >> oVirt's > >> >> db will be used in the comparison. > >> > > >> >This will work only when scanning storage domains already > >> >attached > >> >and in > >> >use by the current oVirt setup. What I am talking about is what > >> >will > >> >happen if a LUN that used to be a SD in another oVirt setup is > >> >discovered > >> >and scanned, with no engine db to compare with. If we don't > >> >consider > >> >such > >> >a use case, life is definitely quite easy, and we're basically > >> >within the > >> >scope of the orphaned images feature > >> > >> This use case should definitely be considered, maybe have a > >> separate > >> case > >> where the rescan would return all "compatible" disks (i.e. disks > >> that > >> aren't just partial snapshots and the like) if the domain has not > >> yet > >> been > >> mounted. Essentially, it would run the same comparison, but > >> compare > >> against an empty list rather than a list of disks. There's no way > >> it's as > >> simple as that (I'm unsure of the methods oVirt uses to mount a > >> domain), > >> but it's a good starting point. > > > >There is no complex method there. For file storage it's just a mount > >command, and for block it's LVM (plus iscsi session establishment, > >if > >needed) > > > >> > > >> >> > >> >> As far as presenting the user with nameless disks, that's a > >> >> point > >> >> I > >> >> hadn't > >> >> considered; we could generate some sort of placeholder metadata > >> >> upon > >> >> addition to show the user that these are new/orphaned disks > >> >> that > >> >> were > >> >> found on the storage domain. Is it safe to assume that the > >> >> disks > >> >> discovered by this feature won't be attached to anything? > >> > > >> >The oVirt paradigm says "if it isn't in the engine db, it's not > >> >ours", so > >> >any LV or image we discover that is missing from the DB or the > >> >snapshot > >> >chain of the image in the DB, is nameless, and orphaned. > >> > > >> >Such an image on a current SD, belonging to a working oVirt setup > >> >is > >> >definitely an orphaned image. Attaching these to VMs is usually > >> >also > >> >useless, because they are more often than not discarded snapshots > >> >that > >> >didn't get discarded cleanly for some reason. > >> > > >> > > >> >Now, if we want to make this usable, we might want to actually > >> >check > >> >the > >> >qcow2 metadata of the image to see whether it's a mid-chain > >> >snapshot > >> >(and > >> >if so it's probably just a candidate for cleanup), or a > >> >standalone > >> >qcow2 > >> >or raw image, and then we can move on with the virt-* tools, to > >> >find > >> >out > >> >the image size and the filesystems it contains. This will at > >> >least > >> >provide the user with some usable information about the detected > >> >image. > >> >If we're talking about scanning an SD that doesn't presently > >> >belong > >> >to > >> >the current oVirt setup, then this is even more relevant, because > >> >all of > >> >the images will have no VM-related context. > >> > >> We're currently working on having disks created outside of the > >> oVirt > >> environment, so not all orphaned disks on the existing storage > >> domain > >> will > >> be artifacts of supposedly-deleted data. > > > >Do you mean like rhevm-image-upload, or something different? > > > >> For our use case, disk > >> images > >> created by us will be able to be imported into oVirt and attached > >> to > >> a VM > >> created through the engine. Because of this, saying "if it isn't > >> in > >> the > >> engine db, it's not ours" wouldn't necessarily be true. > >> > >> When you talk about checking the metadata, does either oVirt or > >> vdsm > >> have > >> a simple way to do this? A query of some sort would be ideal for > >> this, as > >> it could be run for each image as a qualifier for import. > > > >qemu-img info and libguestfs commands should do. > >Besides, our images do come with some metadata (in the LVM tags or a > >.meta file) > > > >> > >> Also, as far as writing the functionality itself, I'm gathering > >> that > >> it > >> should be structured as a query to return these orphaned images, > >> which can > >> then be acted upon/added to the database through a separate > >> command > >> after > >> checking the validity of each image? > > > >Yes, a simple way to say "import this one to DB, attach to VM X or > >make > >floating", "delete that one", "skip" > > > > > >> > > >> >> > > >> >> > > >> >> >> > > >> >> >> >> > >> >> >> >> > > >> >> >> >> > [1] or a subtab on the storage domain. > >> >> >> >> > > >> >> >> >> > _______________________________________________ > >> >> >> >> > Engine-devel mailing list > >> >> >> >> > [email protected] > >> >> >> >> > http://lists.ovirt.org/mailman/listinfo/engine-devel > >> >> >> >> > > >> >> >> >> > >> >> >> > > >> >> >> >-- > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> >Regards, > >> >> >> > > >> >> >> >Dan Yasny > >> >> >> >Red Hat Israel > >> >> >> >+972 9769 2280 > >> >> >> > >> >> >> > >> >> > > >> >> >-- > >> >> > > >> >> > > >> >> > > >> >> >Regards, > >> >> > > >> >> >Dan Yasny > >> >> >Red Hat Israel > >> >> >+972 9769 2280 > >> >> > >> >> > >> > > >> >-- > >> > > >> > > >> > > >> >Regards, > >> > > >> >Dan Yasny > >> >Red Hat Israel > >> >+972 9769 2280 > >> > >> > > > >-- > > > > > > > >Regards, > > > >Dan Yasny > >Red Hat Israel > >+972 9769 2280 > >_______________________________________________ > >Engine-devel mailing list > >[email protected] > >http://lists.ovirt.org/mailman/listinfo/engine-devel > > -- Regards, Dan Yasny Red Hat Israel +972 9769 2280 _______________________________________________ Engine-devel mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/engine-devel
