On 06/19/2012 01:45 AM, Saggi Mizrahi wrote:
First of all I'd like to suggest not using the LSM acronym as it can also mean 
live-storage-migration and maybe other things.

Sure, what do you suggest ? libSM ?

Secondly I would like to avoid talking about what needs to be changed in VDSM 
before we figure out what exactly we want to accomplish.

Also, there is no mention on credentials in any part of the process.
How does VDSM or the host get access to actually modify the storage array?
Who holds the creds for that and how?
How does the user set this up?

Per my original discussion on this with Ayal, this is what he had suggested... "In addition, I'm assuming we will either need a new 'storage array' entity in engine to keep credentials, or, in case of storage array as storage domain, just keep this info as part of the domain at engine level."

Either we can have the libstoragemgmt cred stored in the engine as part of engine-setup or have the user input them as part of Storage Prov and user clicks on "remember cred" button, so engine saves it and passes it to VDSM as needed ? In any way, the cred should come from the user/admin, no other way correct ?

In the array as domain case. How are the luns being mapped to initiators. What 
about setting discovery credentials.
In the array set up case. How will the hosts be represented in regards to 
How will the different schemes and capabilities in regard to authentication 
methods will be expressed.

Not clear on what the concern here is. Can you pls provide more clarity on the problem here ?
Maybe providing some examples will help.

Rest of the comments inline

----- Original Message -----
From: "Deepak C Shetty"<deepa...@linux.vnet.ibm.com>
To: "VDSM Project Development"<vdsm-devel@lists.fedorahosted.org>
Cc: libstoragemgmt-de...@lists.sourceforge.net, engine-de...@ovirt.org
Sent: Wednesday, May 30, 2012 5:38:46 AM
Subject: [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration

Hello All,

      I have a draft write-up on the VDSM-libstoragemgmt integration.
I wanted to run this thru' the mailing list(s) to help tune and
crystallize it, before putting it on the ovirt wiki.
I have run this once thru Ayal and Tony, so have some of their

I still have few doubts/questions, which I have posted below with
ending with '?'

Comments / Suggestions are welcome&  appreciated.


[Ccing engine-devel and libstoragemgmt lists as this stuff is
to them too]


1) Background:

VDSM provides high level API for node virtualization management. It
in response to the requests sent by oVirt Engine, which uses VDSM to
all node virtualization related tasks, including but not limited to
storage management.

libstoragemgmt aims to provide vendor agnostic API for managing
storage array. It should help system administrators utilizing open
source solutions have a way to programmatically manage their storage
hardware in a vendor neutral way. It also aims to facilitate
automation, ease of use and take advantage of storage vendor
features which improve storage performance and space utilization.

Home Page: http://sourceforge.net/apps/trac/libstoragemgmt/

libstoragemgmt (LSM) today supports C and python plugins for talking
external storage array using SMI-S as well as native interfaces (eg:
netapp plugin )
Plan is to grow the SMI-S interface as needed over time and add more
vendor specific plugins for exploiting features not possible via
or have better alternatives than using SMI-S.
For eg: Many of the copy offload features require to use vendor
commands, which justifies the need for a vendor specific plugin.

2) Goals:

      2a) Ability to plugin external storage array into oVirt/VDSM
virtualization stack, in a vendor neutral way.

      2b) Ability to list features/capabilities and other statistical
info of the array

      2c) Ability to utilize the storage array offload capabilities

3) Details:

LSM will sit as a new repository engine in VDSM.
VDSM Repository Engine WIP @ http://gerrit.ovirt.org/#change,192

Current plan is to have LSM co-exist with VDSM on the virtualization

*Note : 'storage' used below is generic. It can be a file/nfs-export
NAS targets and LUN/logical-drive for SAN targets.

VDSM can use LSM and do the following...
      - Provision storage
      - Consume storage

3.1) Provisioning Storage using LSM

Typically this will be done by a Storage administrator.

oVirt/VDSM should provide storage admin the
      - ability to list the different storage arrays along with their
types (NAS/SAN), capabilities, free/used space.
      - ability to provision storage using any of the array
(eg: thin provisioned lun or new NFS export )
      - ability to manage the provisioned storage (eg: resize/delete

Once the storage is provisioned by the storage admin, VDSM will have
refresh the host(s) for them to be able to see the newly provisioned
[SM] What about the clustered case, The management or the mailbox will have to 
be involved. Pros\Cons? Is there a capability for the storage to announce a 
change in topology? Can libstoragemgmt consume it? Does it even make sense?

A change in storage topology can only happen via the storage admin provisioning new LUNs, so why not have 'refresh capability' as a verb in the 'storage array' entity which causes VDSM to refresh the hosts via getDeviceList as put in 3.1.1 below ? Refresh capability can be invoked manually by the admin or can be setup periodically to happen.

3.1.1) Potential flows:

Mgmt ->  vdsm ->  lsm: create LUN + LUN Mapping / Zoning / whatever is
needed to make LUN available to list of hosts passed by mgmt
Mgmt ->  vdsm: getDeviceList (refreshes host and gets list of devices)
   Repeat above for all relevant hosts (depending on list passed
mostly relevant when extending an existing VG)
Mgmt ->  use LUN in normal flows.
[SM] This is all a bit vague in my opinion, concrete cases might prove more 

Can you provide your point of view here ?

3.1.2) How oVirt Engine will know which LSM to use ?

Normally the way this works today is that user can choose the host to
use (default today is SPM), however there are a few flows where mgmt
will know which host to use:
1. extend storage domain (add LUN to existing VG) - Use SPM and make
sure *all* hosts that need access to this SD can see the new LUN
2. attach new LUN to a VM which is pinned to a specific host - use
this host
3. attach new LUN to a VM which is not pinned - use a host from the
cluster the VM belongs to and make sure all nodes in cluster can see
new LUN

Flows for which there is no clear candidate (Maybe we can use the SPM
host itself which is the default ?)
1. create a new disk without attaching it to any VM
2. create a LUN for a new storage domain
[SM] Maybe the engine should do the work? What about permission? Will all hosts 
have the credentials to mess with the storage? Will they be passed on a per 
call basis to prevent other users from having access to the storage?

VDSM will get the creds via the engine or from the domain metadata, whatever is chosen as the right way. Once it has the creds, what perm issues do you see ? VDSM just uses the creds to talk with the libstoragemgmt.

3.2) Consuming storage using LSM

Typically this will be done by a virtualization administrator

oVirt/VDSM should allow virtualization admin to
      - Create a new storage domain using the storage on the array.
      - Be able to specify whether VDSM should use the storage offload
capability (default) or override it to use its own internal logic.

4) VDSM potential changes:

4.1) How to represent a VM disk, 1 LUN = 1 VMdisk or 1 LV = 1 VMdisk
I find this hard to understand. Maybe a different notation?

Like what ?

In ant case there is an abstracted case ie. storage domain. And there is a 
direct case ie. user provision luns to be used by VDSM and others as well.
The will both have different ways of representing the same underlying objects.

Do you see the storage repository engine fitting the bill here ? Does implementing a storage repo engine for each of the above case make sense here ?

Also, I think that credentials might be tricky to represent as different arrays 
use different schemes to allocated users\hosts to luns\targets.

Maybe Tony (from libstoragemgmt) can provide more insight here on how the creds can be passed to cover different scenarios. I am not very aware on the diff. types of schemes possible.

which bring another question...1 array == 1 storage domain OR 1
LUN/nfs-export on the array == 1 storage domain ?

Pros&  Cons of each...

1 array == 1 storage domain
      - Each new vmdisk (aka volume) will be a new lun/file on the
      - Easier to exploit offload capabilities, as they are available
the LUN/File granularity
      - Will there be any issues where there will be too many
... any maxluns limit on linux hosts that we might hit ?
          -- VDSM has been tested with 1K LUNs and it worked fine -
      - Storage array limitations on the number of LUNs can be a
      - Would it be ok to share the array for hosting another storage
domain if need be ?
          -- Provided the existing domain is not utilising all of the
free space
          -- We can create new LUNs and hand it over to anyone needed
      -- Changes needed in VDSM to work with raw LUNs, today it only
support for consuming LUNs via VG/LV.

1 LUN/nfs-export on the array == 1 storage domain
      - How to represent a new vmdisk (aka vdsm volume) if its a LUN
provisioned using SAN target ?
          -- Will it be VG/LV as is done today for block domains ?
          -- If yes, then it will be difficult to exploit offload
capabilities, as they are at LUN level, not at LV level.
      - Each new vmdisk will be a new file on the nfs-export, assuming
offload capability is available at the file level, so this should
for NAS targets ?
      - Can use the storage array for hosting multiple storage
          -- Provision one more LUN and use it for another storage
if need be.
      - VDSM already supports this today, as part of block storage
domains for LUNs case.

Note that we will allow user to do either one of the two options
depending on need.

4.2) Storage domain metadata will also include the
of the storage array as reported by LSM.
          - Capabilities (taken via LSM) will be stored in the domain
metadata during storage domain create flow.
          - Need changes in oVirt engine as well ( see 'oVirt Engine
potential changes' section below )

4.3) VDSM to poll LSM for array capabilities on a regular basis ?
Per ayal:
      - If we have a 'storage array' entity in oVirt Engine (see
Engine potential changes' section below ) then we can have a 'refresh
capabilities' button/verb.
      - We can periodically query the storage array.
      - Query LSM before running operations (sounds redundant to me,
if it's cheap enough it could be simplest).

      Probably need a combination of 1+2 (query at very low frequency
1/hour or 1/day + refresh button)

5) oVirt Engine potential changes - as described by ayal :

      - We will either need a new 'storage array' entity in engine to
keep credentials, or, in case of storage array as storage domain,
keep this info as part of the domain at engine level.
      - Have a 'storage array' entity in oVirt Engine to support
'refresh capabilities' as a button/verb.
      - When user during storage provisioning, selects a LUN exported
from a storage array (via LSM), the oVirt Engine would know from then
onwards that this LUN is being served via LSM.
        It would then  be able to query the  capabilities of the LUN
show it to the virt admin during storage consumption flow.

6) Potential flows:
      - Create snapshot flow
          -- VDSM will check the snapshot offload capability in the
domain metadata
          -- If available, and override is not configured, it will use
LSM to offload LUN/File snapshot
          -- If override is configured or capability is not available,
will use its internal logic to create
             snapshot (qcow2).

      - Copy/Clone vmdisk flow
          -- VDSM will check the copy offload capability in the domain
          -- If available, and override is not configured, it will use
LSM to offload LUN/File copy
          -- If override is configured or capability is not available,
will use its internal logic to create
             snapshot (eg: dd cmd in case of LUN).

7) LSM potential changes:

      - list features/capabilities of the array. Eg: copy offload,
prov. etc.
      - list containers (aka pools) (present in LSM today)
      - Ability to list different types of arrays being managed, their
capabilities and used/free space
      - Ability to create/list/delete/resize volumes ( LUN or exports,
available in LSM as of today)
      - Get monitoring info with object (LUN/snapshot/volume) as
parameter for specific info. eg: container/pool free/used space, raid
type etc.

Need to make sure above info is listed in a coherent way across
(number of LUNs, raid type used? free/total per container/pool, per
LUN?. Also need I/O statistics wherever possible.

Engine-devel mailing list

vdsm-devel mailing list

Reply via email to