Re: [vdsm] How to figure out the transport type of Gluster volume from VDSM host (which is not a gluster peer) ?

2013-05-06 Thread Shu Ming

2013-5-6 22:33, Deepak C Shetty:

Hi Lists,
I am looking at options to figure the transport type of a gluster 
volume (given the volfileserver:volname) from a host that is *not* 
part of gluster volume (aka not a gluster peer).


The context here is GlusterFS as a storage domain in oVirt/VDSM, which 
is currently available in upstream oVirt.
This features exploits the QEMU-GlusterFS native integration, where 
the VM disk is specified using gluster+://... protocol.


For eg. if transport is TCP.. the URI looks liek 
gluster+tcp://..., otherwise gluster+rdma://...


Thus, to generate the gluster QEMU URI in VDSM, i need to know the 
Gluster volume's transport type and the only inputs that oVirt gets 
for GlusterFS storage domain are...

a) volfileserver (the host running glusterd)
b) volname (the name of the volume)

Currently i use VDSM's gluster plugin to do the eq. of "gluster volume 
info " to determine Gluster volume's transport type, but this 
won't work if the VDSM host is not a gluster peer, 
What do you mean by using "gluster peer"?  Does "gluster peer" mean the 
host is running glusterd?


which is a constraint! ... and I would like to fix/remove this 
constraint.


So i discussed a bit on #glsuter-dev IRC and want to put down the 
options here for the community to help provide inputs on whats the 
best way to approach this...


1) Use gluster --remote-host= volume info 



This is not a supported way and there is no guarantee on how long the 
--remote-host option be supported in gluster, since it has some 
security issues


2) Use gluster ::system getspec 

I tried this but it never worked for me... whats the right way of 
using this ?

For me.. it just returned back to shell w/o dumping the volfile at all!

3) Have oVirt user provide the transport type as well (while creating 
Gluster storgae domain) in addition to volfileserver:volname options


This would be easiest, since VDSM can form the gluster QEMU URI by 
directly using the transport type specified by the user, and this 
won't have a need to use the vdsm-gluster plugin, hence no need for 
VDSM host to be part of gluster peer...but this would mean addnl input 
for user to provide during Gluster domain creation and oVirt UI 
changes to take the transport type as input in addition to 
volfileserver:volname

What will happen if a user gives a wrong transport type to VDSM?



Comments/Opinions/Inputs appreciated

thanx,
deepak

(P.S. cross-posting this to VDSM and Gluster devel lists, as it 
relates to both)


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] vdsm sync call meeting minutes Monday April 8th 2013

2013-04-08 Thread Shu Ming

Dan Kenigsberg:

Speaking attendees, in no particular order: Ayal, Yaniv, Saggi, Douglas,
Adam, Dan, Federico, Toni.

Here are the meeting minutes for this week's sync call:

- two small ovirt-3.2 patches are pending
   http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:ovirt-3.2,n,z
   Federico to consider to take them into the branch.

- Federico to research more into Yuval M and Limor Gavish's nasty failure
   http://lists.ovirt.org/pipermail/users/2013-March/013635.html

- Toni: No huge leaps for NetReload during this fortnight.

- There are plenty of nice improvements by Zhou Zhen Sheng. Please help in
   reviewing them: http://gerrit.ovirt.org/#/dashboard/1000174

- Two Vdsm developers from IBM Beijing intend to come over to
   http://www.ovirt.org/OVirt_Global_Workshops#oVirt_Workshop_at_Intel_Campus
   So is /me.


I post two topic proposals to workshop...@ovirt.org.  No response?



- Saggi is busy writing a Java client for Vdsm.
   Needs review for his jsonrpc patches. Will ping Adam about it.

- It's high time to solve https://fedorahosted.org/ovirt/ticket/41
   Adam is cool about Jenkis async task, nacking an invalid commit message.

   Dan prefers to get any sinner to follow this link
   
https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=2MTZZR5UUAD5L&item_name=Secret%20Link%20Atonement&amount=2%2e00¤cy_code=USD

   But nagging ovirt infra about ticket 41 is acceptable.

- Ayal: Backup API, DRBD design work is being done.

- Subteams (mainly infra and storage) to update their list of features in
   http://www.ovirt.org/OVirt_3.3_release-management
   as the feature freaze is in less than 2 months.

See you,

Dan.
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] How to map the oVirt engine version to VDSM version by git tags?

2013-04-08 Thread Shu Ming
Hi,

I am looking for a way to map the oVirt version to VDSM version
and engine version by git tags. I can run "git tag -l" under
engine git workspace and vdsm git workspace. Here are the output
from these two "git tag -l" command.

Under oVirt engine workspace:

-bash-4.1$ git tag -l
ovirt-engine-3.0.0_0001
ovirt-engine-3.1.0
ovirt-engine-3.2.0
ovirt-engine-3.2.1


Under vdsm workspace:
-bash-4.1$ git tag -l
v4.10.0
v4.10.1
v4.10.2
v4.10.3
v4.9.0
v4.9.1
v4.9.2
v4.9.3
v4.9.3.1
v4.9.3.2
v4.9.3.3
v4.9.4
v4.9.5
v4.9.6


I can checkout the oVirt 3.2.1 snapshot in engine workspace by
"git checkout ovirt-engine-3.2.1". But how can I get the VDSM snapshot
by the tags in VDSM workspace? How can I know which change-set is for
oVirt 3.2.1 in VDSM workspace?

-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] Does we really need _addTask() in taskManager.py?

2013-03-18 Thread Shu Ming
Hi,

I am reading the code in taskManager.py. It seems that _addTask was
defined in class Taskmanager, but no else to use them. Also, class
Taskmanager get a similar function queue() which is used. Does we really
need _addTask() function or just remove it?

-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] VDSM test unit running path

2013-03-06 Thread Shu Ming
Hi,

Now, The VDSM test unit is expected to run in a un-installed enviornment
and use hackVDSM() to fake modules importing. hackVDSM() is pretty
tricky for other modules to be added because of the dependency order. I
would propose to run the VDSM unit tests in a installed VDSM path which
is the same path as other VDSM runnable binaries. By this way, we can
abandon hackVDSM() and run the unit test normally. On the other hand,
the VDSM building script installs the VDSM files to /builder by
default, it is reasonable to start the VDSM unit test after all parts of
the building script finishes. Any comments about this idea?

-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] VDSM Repository Reorganization

2013-02-23 Thread Shu Ming
there too.

To be honest I started envisioning client/common/daemon/tool as 
pure-python

directories (as they are once installed) so maybe the schema could be
placed somewhere else with other miscellaneous files (./schema? 
./config?),

but I wouldn't stress over this if it's impractical.

I would like to see also pure python directories.


A rough quick n dirty overview how I could imagine it to look like:

topdir/
build-aux/*
contrib/*
deployment/
bootstrap/
log/
reg/
doc/
vdsm
api
cli
hooks/*
m4/*
tests/*
scripts/
vdsm/*
storage/*
network/*
vdsm/
api/
json/
jsonrpc/
BindingJsonRpc.py
xml/
BindingXMLRPC.py
Bridge.py
vdsmapi-schema.json
process-schema.py
vdsmapi.py
cli/
vdsClient.py
VdsClientGluster.py
extra/
betterPopen/
shared/
vdsm/
SecureXMLRPCServer.py
config.py.in
constants.py.in
utils.py
vdscli.py.in*
utils.py
tool/
daemon/
storage/*
network/*
vdsmd/
vdsm
*




I would like to make all of the service under vdsm/daemon directory only 
reference the files under vdsm/shared.  And back-reference from 
shared/vdsm to vdsm/daemon is not allowed.  In current VDSM repository, 
the files under vdsm/storage and the files under vdsm are 
cross-referenced on each other,  that is a disaster to the effort trying 
to make stand-alone service from storage  service outside of vdsmd.  In 
addition, the files under vdsm/cli should also only depend on the files 
under vdsm/shared.


BTW: Another service under vdsm/daemon should be supervdsm, so the 
daemon should like:

...
   daemon/
 storage/*
 network/*
 vdsmd/
 supervdsmd/
...




That said, I need to assert that I am not yet fully aware of all 
connections between the files and I guess the maintainers will know 
best about their respective parts of responsibility.


I'd like to add that it'd be probably a good idea to start having 
network and storage split off the main daemon for maintenance reasons, 
even if they interface each other.
I personally even could see a scenario where network and storage are 
working as standalone daemons. However that's off topic for know I guess.





--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Meeting minutes, Jan 28

2013-01-28 Thread Shu Ming

于 2013-1-29 4:00, Dan Kenigsberg 写道:

Following are the notes that I've taking during the call.
I'm pretty sure that I'm missing something important, such as another blocker
BZ, so please reply with corrections/addtions.

- Adam: going to introduce Python binding for the new Vdsm API

- Sharad has joined our call for the first time. Welcome!
   Interested in backup/restore functionality in oVirt, to facilitate
   integration with TSM, presumably
   http://www-142.ibm.com/software/products/us/en/tivostormana/

- Federico warns that live snapshot is not enough, since we lack live
   merge. After a year of daily backups, each qcow chain would be 365
   hops deep.


That's horrible.  It looks like that Qemu 1.3 and libvirt 1.0.0 already 
supports the "live block commit" which is the base of live commit.  Do 
we have plan to support "live commit" in next oVirt release?





ovirt-3.2:
- We should probably revert http://gerrit.ovirt.org/9315 (integrate
   zombie reaper in supervdsmServer) as it tickles a bug in
   multiprocessing. See
   http://lists.ovirt.org/pipermail/users/2013-January/011857.html for
   more details in the thread starting in
   http://lists.ovirt.org/pipermail/users/2013-January/011747.html
   (thanks for raising the issue and finding the culprit, Dead Horse)

- Danken believes that all important el6-specific issues reported by dreyou are
   now handled in the master and ovirt-3.2 branches. Please report this list if
   he's wrong.

- Federico reports that udev is going to revert to its Fedora 18alpha
   behavior so that we can keep our integration with it. I'd love to see
   the udev BZ listed in our tracker
   https://bugzilla.redhat.com/showdependencytree.cgi?id=881006&hide_resolved=1
   as I do not recall it. Federico, could you add it? We should probably
   require a newer udev release.

- Federico reports that we must take a short-but-important patch from Lee
   Yarwood: http://gerrit.ovirt.org/#/c/11281/

Cross-distro:
- Adam: a guy from IBM is now listing issues that block running Vdsm on
   Ubuntu.  Yay!
- Saggi suggests that IBM contribute an Ubuntu slave for Jenkins, so
   that each and every patch is tested not to introduce Ubuntu
   regressions - just as Red Hat has recently done with EL6.
- Danken: yes, we have got el6-based unit-testing running. If you are still
   getting an unjustified X, you may need to rebase on current master.
- Federico: NetworkManager 9 has bridge support, which leads the way to
   NM-based implementation of Vdsm's configNetwork.
- Danken: Toni is working on a suggestion for refactoring configNetwork
   in such a way that multiple implementations (e.g. ifcfg-based,
   NM-based) can coexist.  Stay stuned for his Wiki page on the subject.

Other refactoring going on:
- Vinzenz has begun breaking the horrible libvirtvm+vm to edible-size
   pieces. Reviews are most welcome, particularly from Mark Wu, whose
   http://gerrit.ovirt.org/10054/ only awaits verification tick.

Functional Testing:
- Adam: a guy from IBM (Zhou, I presume) is running the functional tests
   via Jenkins on his laptop. Yay!


Zhou Zheng Sheng 


- Adam: plans to add a functional test for the getVdsCaps verb, as he
   found out that an evil Vdsm developer has added values to the caps
   without updating the schema.

- P.S. FlowID is now in for storage verbs as well (thanks, Douglas). I
   hope Haim is happier now.

Happy coding!
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Engine-devel] RFC: New Storage API

2013-01-22 Thread Shu Ming

2013-1-15 5:34, Ayal Baron:

image and volume are overused everywhere and it would be extremely confusing to 
have multiple meanings to the same terms in the same system (we have image 
today which means virtual disk and volume which means a part of a virtual disk).
Personally I don't like the distinction between image and volume done in 
ec2/openstack/etc seeing as they're treated as different types of entities 
there while the only real difference is mutability (images are read-only, 
volumes are read-write).
To move to the industry terminology we would need to first change all 
references we have today to image and volume in the system (I would say also in 
ovirt-engine side) to align with the new meaning.
Despite my personal dislike of the terms, I definitely see the value in 
converging on the same terminology as the rest of the industry but to do so 
would be an arduous task which is out of scope of this discussion imo (patches 
welcome though ;)


Another distinction between Openstack and oVirt is how the 
Nova/ovirt-engine look upon storage systems. In Openstack, a stand alone 
storage service(Cinder) exports the raw storage block device to Nova. On 
the other hand, in oVirt, storage system is highly bounded with the 
cluster scheduling system which integrates storage sub-system, VM 
dispatching sub-system, ISO image sub systems. This combination make all 
of the sub-system integrated in a whole which is easy to deploy, but it 
make the sub-system more opaque and not harder to reuse and maintain. 
This new storage API proposal give us an opportunity to distinct these 
sub-systems as new components which export better, loose-coupling APIs 
to VDSM.




___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


___
Engine-devel mailing list
engine-de...@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel




--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail:shum...@cn.ibm.com  
orshum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Live storage migration status in oVirt 3.2

2013-01-13 Thread Shu Ming

于 2013-1-11 19:07, Daniel Erez 写道:


- Original Message -

From: "Shu Ming" 
To: engine-de...@ovirt.org, "VDSM Project Development" 

Cc: "Federico Simoncelli" , de...@redhat.com
Sent: Friday, January 11, 2013 2:45:47 AM
Subject: Live storage migration status in oVirt 3.2

Hi,

I am reviewing the live storage migration with my oVirt environment
updated from the public nightly repository two weeks ago.  I found
that
the "move" button was still gray as before when the VM was up.  Only
after I deactivated the disk, did the button become into non-gray
state.
  I am wondering if the live storage migration will be supported in
  oVirt
3.2 release or not.

BTW: These patches below should enable the live storage migration
already, but I can not see  it enabled in my engine.

http://gerrit.ovirt.org/5252  Change I91e641cb: pool: live storage
migration implementation

http://gerrit.ovirt.org/8105  Change subject: core: Live Storage
Migration commands
http://gerrit.ovirt.org/8103  Change subject: core: VDS Commands for
Live Storage Migration
http://gerrit.ovirt.org/8102  Change subject: core: Adding VDSM API
for
Live Storage Migration
http://gerrit.ovirt.org/8470  Change subject: webadmin: Adding Live
Storage Migration support
http://gerrit.ovirt.org/8857 core: disable LiveStorageMigration on
3.1

--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
District, Beijing 100193, PRC


Hi Shu,

When a VM is up, the "move" button is enabled only for Data Center version 3.2.
Can you please verify that the selected disk resides on a 3.2 Data Center?


I checked my data center compatibility version is 3.1.  However, it is 
interesting that the compatibility version of the cluster in the data 
center is 3.2  Does the data center allow a higher version cluster?  I 
will try to upgrade the data center version and try the migration again.




Best Regards,
Daniel






--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] Live storage migration status in oVirt 3.2

2013-01-10 Thread Shu Ming
Hi,

I am reviewing the live storage migration with my oVirt environment
updated from the public nightly repository two weeks ago.  I found that
the "move" button was still gray as before when the VM was up.  Only
after I deactivated the disk, did the button become into non-gray state.
 I am wondering if the live storage migration will be supported in oVirt
3.2 release or not.

BTW: These patches below should enable the live storage migration
already, but I can not see  it enabled in my engine.

http://gerrit.ovirt.org/5252  Change I91e641cb: pool: live storage
migration implementation

http://gerrit.ovirt.org/8105  Change subject: core: Live Storage
Migration commands
http://gerrit.ovirt.org/8103  Change subject: core: VDS Commands for
Live Storage Migration
http://gerrit.ovirt.org/8102  Change subject: core: Adding VDSM API for
Live Storage Migration
http://gerrit.ovirt.org/8470  Change subject: webadmin: Adding Live
Storage Migration support
http://gerrit.ovirt.org/8857 core: disable LiveStorageMigration on 3.1

-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian
District, Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] ovirt nightly FC17 public repositories broken?

2013-01-07 Thread Shu Ming
Hi,

Any one was encountering this error message when using yum on ovirt
public repositories?
http://ovirt.org/releases/nightly/rpm/Fedora/17/repodata/primary.xml.gz
<http://ovirt.org/releases/nightly/rpm/Fedora/17/repodata/primary.xml.gz>:
[Errno -1] Metadata file does not match checksum

My vdsm host and engine server was installed with FC17+virt-review, so I
used FC17 nightly releases to update my VDSM and engine packages. After
a bit investigation, I found these issues.


[root@node1-sming ovirt-nightly]# ls -lh *
-rw-r--r--. 1 root root 0 Jan 7 16:47 cachecookie
-rw-r--r--. 1 root root 702K Jan 7 14:22 filelists.xml.gz
-rw-r--r--. 1 root root 132K Jan 7 14:22 other.xml.gz
-rw-r--r--. 1 root root 279K Jan 7 14:22 primary.xml.gz
-rw-r--r--. 1 root root 1.4K Jan 5 14:35 repomd.xml *<---the date was
always Jan 5 different from other files even
http://resources.ovirt.org/releases/nightly**/rpm/Fedora/17/repodata/
showed it was Jan 7, also I tried to use wget to download this file and
got the same thing.**
*
[root@node1-sming ovirt-nightly]# sha1sum other.xml.gz filelists.xml.gz
primary.xml.gz
453559e86950af07c876931f318d1e92ab58f289 other.xml.gz
5318e237c0f4d2ea3d1ec24e82b9f9bbe9d23a31 filelists.xml.gz
e667267d45b576844a5f2cb27ce7bcfb8bb9b4f0 primary.xml.gz

[root@node1-sming ovirt-nightly]# cat repomd.xml |grep open-checksum
ce6a5243ee81c124d0fe48d103174b93fabe60573e80b5851888298373d3be9e

554a301c526d335ca4a7985e7fc7b0374bca8c061ff5addbd164de81996dc5a0

be3f9e6860cd9a4ac4f3f5000ad503175e7ac79a43185a778d4ab732c0d8190c


*Note: These checksum from repmod.xml were quite different from the
result of sha1sum above.*

Any clues here?

-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Engine-devel] FW: Querying for and registering unknown disk images on a storage domain

2012-12-23 Thread Shu Ming

2012-12-20 23:18, Morrissey, Christopher:


Hi All,

I've been working on a bit of functionality for the engine that will 
allow a user to query a domain for new disk images 
(GetUnregisteredImagesQuery) for which the engine was previously 
unaware and a separate command to register those images 
(ImportImageCommand). These commands will be exposed through the REST API.


This functionality is needed as we are developing an extension/plugin 
to oVirt that will allow a NetApp storage controller to handle cloning 
the actual disks outside of oVirt and need to import them once they 
are cloned. We'll be using other existing APIs to attach the disk to 
the necessary VM once the disk is cloned. On the NetApp side, we'll 
ensure the disk is coalesced before cloning so as to avoid the issues 
of registering snapshots.


 I am just curious about how the third party tool like NetApp to make 
sure the disk of a running VM coalesced before cloning? By an agent in 
the VM to flush file-system cache out to the disk?


GetUnregisteredImagesQuery will be accessible through the disks 
resource collection on a storage domain. A "disks" resource collection 
does not yet exist and will need to be added. To access the 
unregistered images, a parameter (maybe "unregistered=true") would be 
passed. So the path to "GET" the unregistered disk images on a domain 
would be something like 
/api/storagedomains/f0dbcb33-69d3-4899-9352-8e8a02f01bbd/disks?unregistered=true. 
This will return a list of disk images that can be each used as input 
to the ImportImageCommand to get them added to oVirt.


ImportImageCommand will be accessible through "POST"ing a disk to 
/api/disks?import=true. The disk will be added to the oVirt DB based 
on the information supplied and afterward would be available to attach 
to a VM.


When querying for unregistered disk images, the 
GetUnregisteredImagesQuery command will use the getImagesList() VDSM 
command. Currently this only reports the GUIDs of all disk images in a 
domain. I had been using the getVolumesList() and getVolumeInfo() VDSM 
commands to fill in the information so that valid disk image objects 
could be registered in oVirt. It seems these two functions are set to 
be removed since they are too invasive into the internal VDSM 
workings. The VDSM team will need to either return more information 
about each disk as part of the getImagesList() function or add a new 
function getImageInfo() that will give the same information for a 
given image GUID.




Here is the project proposal for floating disk in oVirt.  I think 
unregistered images are also floating disks.

http://www.ovirt.org/Features/DetailedFloatingDisk

Note that much of this work had originally been submitted under patch 
http://gerrit.ovirt.org/#/c/9603/. After several reviews it was found 
to be lacking in its design and was using deprecated APIs that did not 
yet have replacements. I'm reworking the code now to conform to this 
design and asking for further input from the VDSM, core, and restapi 
teams to ensure we can get this done quickly and correctly as it is 
needed for the 3.2 release.


-Chris

*Chris Morrissey*

Software Engineer

NetApp Inc.

919.476.4428



___
Engine-devel mailing list
engine-de...@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel



--
---
?? Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Host bios information

2012-12-13 Thread Shu Ming
After a quick review of the wiki page, it was stated that dmidecode gave 
too much informations. Only five fields will be displayed in the 
hardware tab, "Manufactory", "Version", "Family", "UUID" and "serial 
number".  For "Family", it is mean the CPU core's family.  And it 
confuses me a bit with the "CPU name" and "CPU type" fields in general 
tab. I think we should chose the best one to characterizethe CPU type.



ybronhei:
Today in the Api we display general information about the host that 
vdsm export by getCapabilities Api.


We decided to add bios information as part of the information that is 
displayed in UI under host's general sub-tab.


To summaries the feature - We'll modify General tab to Software 
Information and add another tab for Hardware Information which will 
include all the bios data that we'll decide to gather from the host 
and display.


Following this feature page: 
http://www.ovirt.org/Features/Design/HostBiosInfo for more details.

All the parameters that can be displayed are mentioned in the wiki.

I would greatly appreciate your comments and questions.

Thanks.




--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] RFC: New Storage API

2012-12-10 Thread Shu Ming

2012-12-11 4:36, Saggi Mizrahi:


- Original Message -

From: "Adam Litke" 
To: "Saggi Mizrahi" 
Cc: "Shu Ming" , "engine-devel" , 
"VDSM Project Development"

Sent: Monday, December 10, 2012 1:39:51 PM
Subject: Re: [vdsm] RFC: New Storage API

On Thu, Dec 06, 2012 at 11:52:01AM -0500, Saggi Mizrahi wrote:


- Original Message -

From: "Shu Ming" 
To: "Saggi Mizrahi" 
Cc: "VDSM Project Development"
, "engine-devel"

Sent: Thursday, December 6, 2012 11:02:02 AM
Subject: Re: [vdsm] RFC: New Storage API

Saggi,

Thanks for sharing your thought and I get some comments below.


Saggi Mizrahi:

I've been throwing a lot of bits out about the new storage API
and
I think it's time to talk a bit.
I will purposefully try and keep implementation details away
and
concentrate about how the API looks and how you use it.

First major change is in terminology, there is no long a
storage
domain but a storage repository.
This change is done because so many things are already called
domain in the system and this will make things less confusing
for
new-commers with a libvirt background.

One other changes is that repositories no longer have a UUID.
The UUID was only used in the pool members manifest and is no
longer needed.


connectStorageRepository(repoId, repoFormat,
connectionParameters={}):
repoId - is a transient name that will be used to refer to the
connected domain, it is not persisted and doesn't have to be
the
same across the cluster.
repoFormat - Similar to what used to be type (eg. localfs-1.0,
nfs-3.4, clvm-1.2).
connectionParameters - This is format specific and will used to
tell VDSM how to connect to the repo.


Where does repoID come from? I think repoID doesn't exist before
connectStorageRepository() return.  Isn't repoID a return value
of
connectStorageRepository()?

No, repoIDs are no longer part of the domain, they are just a
transient handle.
The user can put whatever it wants there as long as it isn't
already taken by another currently connected domain.

disconnectStorageRepository(self, repoId)


In the new API there are only images, some images are mutable
and
some are not.
mutable images are also called VirtualDisks
immutable images are also called Snapshots

There are no explicit templates, you can create as many images
as
you want from any snapshot.

There are 4 major image operations:


createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
userData={}, options={}):

targetRepoId - ID of a connected repo where the disk will be
created
size - The size of the image you wish to create
baseSnapshotId - the ID of the snapshot you want the base the
new
virtual disk on
userData - optional data that will be attached to the new VD,
could
be anything that the user desires.
options - options to modify VDSMs default behavior

returns the id of the new VD

I think we will also need a function to check if a a VirtualDisk
is
based on a specific snapshot.
Like: isSnapshotOf(virtualDiskId, baseSnapshotID):

No, the design is that volume dependencies are an implementation
detail.
There is no reason for you to know that an image is physically a
snapshot of another.
Logical snapshots, template information, and any other information
can be set by the user by using the userData field available for
every image.

Statements like this make me start to worry about your userData
concept.  It's a
sign of a bad API if the user needs to invent a custom metadata
scheme for
itself.  This reminds me of the abomination that is the 'custom'
property in the
vm definition today.

In one sentence: If VDSM doesn't care about it, VDSM doesn't manage it.

userData being a "void*" is quite common and I don't understand why you would 
thing it's a sign of a bad API.
Further more, giving the user choice about how to represent it's own metadata 
and what fields it want to keep seems reasonable to me.
Especially given the fact that VDSM never reads it.

The reason we are pulling away from the current system of VDSM understanding 
the extra data is that it makes that data tied to VDSMs on disk format.
VDSM on disk format has to be very stable because of clusters with multiple 
VDSM versions.
Further more, since this is actually manager data it has to be tied to the 
manager backward compatibility lifetime as well.
Having it be opaque to VDSM ties it to only one, simpler, support lifetime 
instead of two.


Making userData being opaque gives flexibilities to the management 
applications.  To me, opaque userDaa can have two types at least. The 
first is the userData for runtime only.  The second is the userData 
expected to be persisted into the metadata disk.  For the first type, 
the management applications can store their own data structures like 
temporary task states, VDSM query caches &etc. After the VDSM host is 
fenced, th

Re: [vdsm] RFC: New Storage API

2012-12-06 Thread Shu Ming

于 2012-12-7 13:23, Deepak C Shetty:

On 12/06/2012 10:22 PM, Saggi Mizrahi wrote:


- Original Message -

From: "Shu Ming" 
To: "Saggi Mizrahi" 
Cc: "VDSM Project Development" , 
"engine-devel" 

Sent: Thursday, December 6, 2012 11:02:02 AM
Subject: Re: [vdsm] RFC: New Storage API

Saggi,

Thanks for sharing your thought and I get some comments below.


Saggi Mizrahi:

I've been throwing a lot of bits out about the new storage API and
I think it's time to talk a bit.
I will purposefully try and keep implementation details away and
concentrate about how the API looks and how you use it.

First major change is in terminology, there is no long a storage
domain but a storage repository.
This change is done because so many things are already called
domain in the system and this will make things less confusing for
new-commers with a libvirt background.

One other changes is that repositories no longer have a UUID.
The UUID was only used in the pool members manifest and is no
longer needed.


connectStorageRepository(repoId, repoFormat,
connectionParameters={}):
repoId - is a transient name that will be used to refer to the
connected domain, it is not persisted and doesn't have to be the
same across the cluster.
repoFormat - Similar to what used to be type (eg. localfs-1.0,
nfs-3.4, clvm-1.2).
connectionParameters - This is format specific and will used to
tell VDSM how to connect to the repo.


Where does repoID come from? I think repoID doesn't exist before
connectStorageRepository() return.  Isn't repoID a return value of
connectStorageRepository()?
No, repoIDs are no longer part of the domain, they are just a 
transient handle.
The user can put whatever it wants there as long as it isn't already 
taken by another currently connected domain.


So what happens when user mistakenly gives a repoID that is in use 
before.. there should be something in the return value that specifies 
the error and/or reason for error so that user can try with a new/diff 
repoID ?


I think let the user to give the repoID is meaningless and error-prune.  
Developer must maintain a a unique ID list for every storage repository 
connected.





disconnectStorageRepository(self, repoId)


In the new API there are only images, some images are mutable and
some are not.
mutable images are also called VirtualDisks
immutable images are also called Snapshots

There are no explicit templates, you can create as many images as
you want from any snapshot.

There are 4 major image operations:


createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
userData={}, options={}):

targetRepoId - ID of a connected repo where the disk will be
created
size - The size of the image you wish to create
baseSnapshotId - the ID of the snapshot you want the base the new
virtual disk on
userData - optional data that will be attached to the new VD, could
be anything that the user desires.
options - options to modify VDSMs default behavior


IIUC, i can use options to do storage offloads ? For eg. I can create 
a LUN that represents this VD on my storage array based on the 
'options' parameter ? Is this the intended way to use 'options' ?




returns the id of the new VD

I think we will also need a function to check if a a VirtualDisk is
based on a specific snapshot.
Like: isSnapshotOf(virtualDiskId, baseSnapshotID):

No, the design is that volume dependencies are an implementation detail.
There is no reason for you to know that an image is physically a 
snapshot of another.
Logical snapshots, template information, and any other information 
can be set by the user by using the userData field available for 
every image.

createSnapshot(targetRepoId, baseVirtualDiskId,
 userData={}, options={}):
targetRepoId - The ID of a connected repo where the new sanpshot
will be created and the original image exists as well.
size - The size of the image you wish to create
baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
to snapshot
userData - optional data that will be attached to the new Snapshot,
could be anything that the user desires.
options - options to modify VDSMs default behavior

returns the id of the new Snapshot

copyImage(targetRepoId, imageId, baseImageId=None, userData={},
options={})
targetRepoId - The ID of a connected repo where the new image will
be created
imageId - The image you wish to copy
baseImageId - if specified, the new image will contain only the
diff between image and Id.
If None the new image will contain all the bits of
image Id. This can be used to copy partial parts of
images for export.
userData - optional data that will be attached to the new image,
could be anything that the user desires.
options - options to modify VDSMs default behavior

Does this function mean that we can copy the image from one
repository
to another repository? Do

Re: [vdsm] RFC: New Storage API

2012-12-06 Thread Shu Ming
tatus of the image.
The status of the image can be either optimized, degraded, or broken.
"Optimized" means that the image is available and you can run VMs of it.
"Degraded" means that the image is available and will run VMs but it might be a 
better way VDSM can represent the underlying data.


What does the "represent" mean here?

"Broken" means that the image can't be used at the moment, probably because not 
all the data has been set up on the volume.

Apart from that VDSM will also return the last persisted status information 
which will conatin
hostID - the last host to try and optimize of fix the image

Any host can optimize the image? No need to be SDM?


stage - X/Y (eg. 1/10) the last persisted stage of the fix.
percent_complete - -1 or 0-100, the last persisted completion percentage of the 
aforementioned stage. -1 means that no progress is available for that operation.
last_error - This will only be filled if the operation failed because of 
something other then IO or a VDSM crash for obvious reasons.
  It will usually be set if the task was manually stopped

The user can either be satisfied with that information or as the host specified 
in host ID if it is still working on that image by checking it's running tasks.


So we need a function to know what tasks are running on the image


checkStorageRepository(self, repositoryId, options={}):
A method to go over a storage repository and scan for any existing problems. 
This includes degraded\broken images and deleted images that have no yet been 
physically deleted\merged.
It returns a list of Fix objects.
Fix objects come in 4 types:
clean - cleans data, run them to get more space.
optimize - run them to optimize a degraded image
merge - Merges two images together. Doing this sometimes
 makes more images ready optimizing or cleaning.
 The reason it is different from optimize is that
 unmerged images are considered optimized.
mend - mends a broken image

The user can read these types and prioritize fixes. Fixes also contain opaque 
FIX data and they should be sent as received to
fixStorageRepository(self, repositoryId, fix, options={}):

That will start a fix operation.


All major operations automatically start the appropriate "Fix" to bring the 
created object to an optimize\degraded state (the one that is quicker) unless one of the 
options is
AutoFix=False. This is only useful for repos that might not be able to create 
volumes on all hosts (SDM) but would like to have the actual IO distributed in 
the cluster.

Other common options is the strategy option:
It has currently 2 possible values
space and performance - In case VDSM has 2 ways of completing the same 
operation it will tell it to value one over the other. For example, whether to 
copy all the data or just create a qcow based of a snapshot.
The default is space.

You might have also noticed that it is never explicitly specified where to look 
for existing images. This is done purposefully, VDSM will always look in all 
connected repositories for existing objects.
For very large setups this might be problematic. To mitigate the problem you 
have these options:
participatingRepositories=[repoId, ...] which tell VDSM to narrow the search to 
just these repositories
and
imageHints={imgId: repoId} which will force VDSM to look for those image ID 
just in those repositories and fail if it doesn't find them there.
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] moving the collection of statistics to external process

2012-12-06 Thread Shu Ming

于 2012-12-6 4:51, Itamar Heim 写道:

On 12/05/2012 10:33 PM, Adam Litke wrote:

On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:

On 12/05/2012 10:16 PM, Adam Litke wrote:

On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:

On 12/05/2012 08:57 PM, Adam Litke wrote:

On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote:

On 12/05/2012 04:42 PM, Adam Litke wrote:
I wanted to know what do you think about it and if you have 
better
solution to avoid initiate so many threads? And if splitting 
vdsm is

a good idea here?
In first look, my opinion is that it can help and would be 
nice to
have vmStatisticService that runs and writes to separate log 
the vms

status.
Vdsm recently started requiring the MOM package. MOM also 
performs some host
and guest statistics collection as part of the policy 
framework.  I think it
would be a really good idea to consolidate all stats collection 
into MOM.  Then,
all stats become usable within the policy and by vdsm for its 
own internal
purposes.  Today, MOM has one stats collection thread per VM 
and one thread for
the host stats.  It has an API for gathering the most recently 
collected stats

which vdsm can use.



isn't this what collectd (and its libvirt plugin) or pcp are 
already doing?


Lot's of things collect statistics, but as of right now, we're 
using MOM and

we're not yet using collectd on the host, right?



I think we should have a single stats collection service and 
clients for it.

I think mom and vdsm should get their stats from that service,
rather than have either beholden to any new stats something needs to
collect.


How would this work for collecting guest statistics?  Would we 
require collectd

to be installed in all guests running under oVirt?



my understanding is collectd is installed on the host, and uses
collects libvirt plugin to collect guests statistics?


Yes, but some statistics can only be collected by making a call to 
the oVirt
guest agent (eg. guest memory statistics).  The logical next step 
would be to
write a collectd plugin for ovirt-guest-agent, but vdsm owns the 
connections to
the guest agents and probably does not want to multiplex those 
connections for

many reasons (security being the main one).



and some will come from qemu-ga which libvirt will support?
maybe a collectd vdsm plugin for the guest agent stats?



I am thinking to have the collectd as a stand alone service to collect 
the statics from both ovirt-guest and qemu-ga.  Then collected can 
export the information to host proc file system in layered 
architecture.  Then mom or other vdsm service can get the information 
from the proc file system like other OS statics exported in the host.





___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [VDSM][RFC] hsm service standalone

2012-12-06 Thread Shu Ming

2012-12-5 23:55, ybronhei:

On 12/03/2012 07:22 PM, Saggi Mizrahi wrote:
HSM is not a package it's an application. Currently it and the rest 
of VDSM share the same process but they use RPC to communicate. This 
is done so that one day we can actually have them run as different 
processes.

HSM is not something you import, it's a daemon you communicate with.

- Original Message -

From: "Dan Kenigsberg" 
To: "Saggi Mizrahi" 
Cc: "Sheldon" , a...@linux.vnet.ibm.com, 
vdsm-devel@lists.fedorahosted.org, "Zheng Sheng

ZS Zhou" 
Sent: Monday, December 3, 2012 12:01:28 PM
Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone

On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:

There are a bunch of precondition to having HSM pulled out.
On simple things is that someone needs to go through
storage/misc.py and utils.py and move all the code out to logical
packages.
There also needs to be a bit of a rearrangement of the code files
so they can import the reusable code properly.

I am also still very much against putting core VDSM in to
site-packages.


Would you elaborate on your position? Do you mind the wrong
implications
this may give about API stability?
Contrary, It may help to split internal api and external (means rpc 
between vdsm to engine and rpc between vdsm and hsm), and it might be 
easier to understand the process flow and control the request from 
engine. For example, for migration you will receive vdsm request for 
migration, and that will initiate all the internal processes that we 
do during migration by sending requests to hsm service.
The engine doesn't supposed to care about the internal rpc and those 
abilities does not supposed to be exposed via vdsm api if we don't 
allow those specific hsm operation from engine, but via HSM-python 
service that won't be available for the engine.

Might be a good division. But lots of work indeed..


Fortunately, the hsm directory in vdsm is almost self-contained. hsm 
files are depending on some files like config.py, constants.py &etc 
which are already in a vdsm site packages.  After HSM-python package is 
created, both vdsm-4.1x and HSM-python-4.1.x package will depend on 
vdsm-python-4.1x package. vdsm-python-4.1x package might be a confusing 
name that time.  It might be better named "vdsm-common-python-4.1x".







___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel







--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail:shum...@cn.ibm.com  
orshum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [VDSM][RFC] hsm service standalone

2012-12-05 Thread Shu Ming

Saggi,

It might be that the wiki page was a bit confusing.  Let me articulate 
it again and help others to understand the intention of this project.


1) Now, HSM files here refers to the files under "vdsm/storage" which 
are storage service related code.  And other VDSM files call into those 
functions in HSM files in local way(different from RPC).  So in the 
production environment, HSM files are linked in the VDSM process and 
there is no additional HSM process standing in the ovirt node.


2) This project goal is to move all the HSM files into self-contained 
workspace and a new stand alone HSM process will be started from these 
HSM files.  Also a new HSM package will be built from this new 
self-contained workspace.   Also all of the HSM APIs are public to the 
original VDSM service by RPC calls. It is possible that the HSM 
standalone workspace and the original VDSM workspace  may share some 
functions in misc.py,task.py&etc.  We can move those common files to 
vdsm site packages as the other common files.  After this goal is 
achieved, we have one more HSM process in production environment and one 
more release package for HSM files.  For backward compatibility, vdsm 
will run in two possible modes. VDSM service will try to discover if HSM 
RPC APIs are available from the new HSM process.  If the discovery is 
successful, VDSM service will make RPC calls to the HSM process.  If the 
discovery fails, it will fall back the legacy way calling the HSM 
functions by local way.


3) In order to achieve this goal,  we should remove the preconditions 
gradually and safely.  The first step here is the patch


http://gerrit.ovirt.org/#/c/9345/

That try to decouple the HSM configuration parameters from the vdsm.conf 
and store them into a new different conf file.


Saggi Mizrahi:

HSM is not a package it's an application. Currently it and the rest of VDSM 
share the same process but they use RPC to communicate. This is done so that 
one day we can actually have them run as different processes.
HSM is not something you import, it's a daemon you communicate with.

- Original Message -

From: "Dan Kenigsberg" 
To: "Saggi Mizrahi" 
Cc: "Sheldon" , a...@linux.vnet.ibm.com, 
vdsm-devel@lists.fedorahosted.org, "Zheng Sheng
ZS Zhou" 
Sent: Monday, December 3, 2012 12:01:28 PM
Subject: Re: [vdsm] [VDSM][RFC] hsm service standalone

On Mon, Dec 03, 2012 at 11:35:44AM -0500, Saggi Mizrahi wrote:

There are a bunch of precondition to having HSM pulled out.
On simple things is that someone needs to go through
storage/misc.py and utils.py and move all the code out to logical
packages.
There also needs to be a bit of a rearrangement of the code files
so they can import the reusable code properly.

I am also still very much against putting core VDSM in to
site-packages.

Would you elaborate on your position? Do you mind the wrong
implications
this may give about API stability?



___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [VDSM][RFC] hsm service standalone

2012-12-02 Thread Shu Ming

Added Saggi, Ayal, Federico into this loop.  And got two comments below.

Sheldon:

there is already a wiki page about hsm service standalone.
http://wiki.ovirt.org/HSM_service_stand_alone

And now I'm trying to split hsm as a standalone rpm according to the 
wiki proposal.



as the wiki describe:
V)The configuration parameters in vdsm.conf should be pull out to 
another hsm.conf file. And all VDSM process should not read hsm.conf 
directly. If it is necessary to get the HSM configurations, an API 
will be added into VDSM process for hsm configuration querying


I have split the the vdsm/config.py and submit a patch 
http://gerrit.ovirt.org/#/c/9345/
I moved 'irs'section from vdsm/config.py to a new 
vdsm/storage/config.py for hsm
But some of vdsm python module need to get configure items form 
'irs'section, such as vdsm/clientIF.py, vdsm/libvirtvm.py.

I moved vdsm/storage/config.py to site-packages/hsm when install hsm.
And I import all the items form 'irs'section, by "from hsm import 
config as hsmconfig"

Is it a good way to split the config.py?

Also I have split the the constants.py and submit a patch 
http://gerrit.ovirt.org/#/c/9348/
And should I create a new "hsm" user for the DISKIMAGE_USER instead of 
"vdsm" user?




as the wiki describe:
VII) All of the python modules shared by both VDSM service and HSM 
service should go into python site-packages directory which is like 
/usr/lib/python2.7/site-packages/vdsm Now, task.py doesn't sit in this 
directory, we should fix it in this proposal.


Task model is only for storage service now.  We have plan to make task 
models to all VDSM service. So it is reasonable to move task.py into 
site-packages directory.




about the site-packages directory, there are three proposals.
1. VDSM service and HSM service share the same site-packages 
directory, still used site-packages/vdsm
and that means there will be still a vdsm-python rpm, and both vdsm 
and hsm depends on them.
2. VDSM service and HSM service share the same site-packages 
directory, but it will renamed as used site-packages/hsm
Now most codes of hsm and vdsm import modules from site-packages vdsm, 
so all of they should be fixed.
3. VDSM service depend on vdsm-python rpm, and HSM service depend on 
hsm-python rpm.
I need to re-sort the public modules to hsm-python or vdsm-python rpm. 
such as fileUtils.py, misc.py, BetterPopen and qemuImg.py

which is better of the above three?


Packaging HSM site package files into a another new package other than 
vdsm-python package doesn't prevent those files contained into a same 
targeting package.  So at least we can do, two different package say, 
vdsm-core-python, vdsm-storage-python and the files in them are 
targeting the same directory /usr/lib/python2.7/site-packages/vdsm.













--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary

2012-11-26 Thread Shu Ming

Adam Litke:

On Mon, Nov 26, 2012 at 02:57:19PM +0200, Livnat Peer wrote:

On 26/11/12 03:15, Shu Ming wrote:

Livnat,

Thanks for your summary.  I got comments below.

2012-11-25 18:53, Livnat Peer:

Hi All,
We have been discussing $subject for a while and I'd like to summarized
what we agreed and disagreed on thus far.

The way I see it there are two related discussions:


1. Getting VDSM networking stack to be distribution agnostic.
- We are all in agreement that VDSM API should be generic enough to
incorporate multiple implementation. (discussed on this thread: Alon's
suggestion, Mark's patch for adding support for netcf etc.)

- We would like to maintain at least one implementation as the
working/up-to-date implementation for our users, this implementation
should be distribution agnostic (as we all acknowledge this is an
important goal for VDSM).
I also think that with the agreement of this community we can choose to
change our focus, from time to time, from one implementation to another
as we see fit (today it can be OVS+netcf and in a few months we'll use
the quantum based implementation if we agree it is better)

2. The second discussion is about persisting the network configuration
on the host vs. dynamically retrieving it from a centralized location
like the engine. Danken raised a concern that even if going with the
dynamic approach the host should persist the management network
configuration.

About dynamical retrieving from a centralized location,  when will the
retrieving start? Just in the very early stage of host booting before
network functions?  Or after the host startup and in the normal running
state of the host?  Before retrieving the configuration,  how does the
host network connecting to the engine? I think we need a basic well
known network between hosts and the engine first.  Then after the
retrieving, hosts should reconfigure the network for later management.
However, the timing to retrieve and reconfigure are challenging.


We did not discuss the dynamic approach in details on the list so far
and I think this is a good opportunity to start this discussion...

 From what was discussed previously I can say that the need for a well
known network was raised by danken, it was referred to as the management
network, this network would be used for pulling the full host network
configuration from the centralized location, at this point the engine.

About the timing for retrieving the configuration, there are several
approaches. One of them was described by Alon, and I think he'll join
this discussion and maybe put it in his own words, but the idea was to
'keep' the network synchronized at all times. When the host have
communication channel to the engine and the engine detects there is a
mismatch in the host configuration, the engine initiates 'apply network
configuration' action on the host.

Using this approach we'll have a single path of code to maintain and
that would reduce code complexity and bugs - That's quoting Alon Bar Lev
(Alon I hope I did not twisted your words/idea).

On the other hand the above approach makes local tweaks on the host
(done manually by the administrator) much harder.

I worry a lot about the above if we take the dynamic approach.  It seems we'd
need to introduce before/after 'apply network configuration' hooks where the
admin could add custom config commands that aren't yet modeled by engine.


Any other approaches ?

Static configuration has the advantage of allowing a host to bring itself back
online independent of the engine.  This is also useful for anyone who may want
to deploy a vdsm node in standalone mode.

I think it would be possible to easily support a quasi-static configuration mode
simply by extending the design of the dynamic approach slightly.  In dynamic
mode, the network configuration is passed down as a well-defined data structure.
When a particular configuration has been committed, vdsm could write a copy of
that configuration data structure to /var/run/vdsm/network-config.json.  During
a subsequent boot, if the engine cannot be contacted after activating the
management network, the cached configuration can be applied using the same code
as for dynamic mode.  We'd have to flesh out the circumstances under which this
would happen.


Multiple physical network devices are quite popular in x86 servers now. 
Can we add some smart algorithm to leverage multiple physical network 
devices? Say, one specific device for the management network and others 
for vdsm host networks. By isolating the management and host networks, 
the vdsm host can maintain a permanent management work and it is much 
more clean and not bug prone. If the host doesn't get multiple network 
devices, we should fall back to the traditional way.






I'd like to add a more general question to the discussion what are the
advantages of taking the dynamic approach?
So far I collected tw

Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary

2012-11-25 Thread Shu Ming

Livnat,

Thanks for your summary.  I got comments below.

2012-11-25 18:53, Livnat Peer:

Hi All,
We have been discussing $subject for a while and I'd like to summarized
what we agreed and disagreed on thus far.

The way I see it there are two related discussions:


1. Getting VDSM networking stack to be distribution agnostic.
- We are all in agreement that VDSM API should be generic enough to
incorporate multiple implementation. (discussed on this thread: Alon's
suggestion, Mark's patch for adding support for netcf etc.)

- We would like to maintain at least one implementation as the
working/up-to-date implementation for our users, this implementation
should be distribution agnostic (as we all acknowledge this is an
important goal for VDSM).
I also think that with the agreement of this community we can choose to
change our focus, from time to time, from one implementation to another
as we see fit (today it can be OVS+netcf and in a few months we'll use
the quantum based implementation if we agree it is better)

2. The second discussion is about persisting the network configuration
on the host vs. dynamically retrieving it from a centralized location
like the engine. Danken raised a concern that even if going with the
dynamic approach the host should persist the management network
configuration.


About dynamical retrieving from a centralized location,  when will the 
retrieving start? Just in the very early stage of host booting before 
network functions?  Or after the host startup and in the normal running 
state of the host?  Before retrieving the configuration,  how does the 
host network connecting to the engine? I think we need a basic well 
known network between hosts and the engine first.  Then after the 
retrieving, hosts should reconfigure the network for later management. 
However, the timing to retrieve and reconfigure are challenging.





Obviously the second discussion influences the API modeling. Since I
think it would be challenging to add support for generic API and change
the current implementation to match the dynamic configuration approach
simultaneously I suggest we'll focus our efforts on one change at a time.

I suggest to have a discussion on the pro's and con's of dynamic
configuration and after we get to a consensus around that we can start
modeling the generic API.

thoughts? comments?

Livnat
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Proposal to add Adam Litke as maintainer to vdsm

2012-11-20 Thread Shu Ming

+1
Itamar Heim:
Adam has been submitting numerous patches for VDSM for more than a 
year, most noticeably on improving VDSM API into, well, an API...


I'd like to propose Adam as a maintainer for vdsm.

Thanks,
   Itamar
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Adding Local Storage Domain failed

2012-11-13 Thread Shu Ming
{} resources {}
Thread-53::DEBUG::2012-11-14 
02:16:19,544::resourceManager::844::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-53::ERROR::2012-11-14 
02:16:19,544::dispatcher::69::Storage.Dispatcher.Protect::(run)
Traceback (most recent call last):
   File "/usr/share/vdsm/storage/dispatcher.py", line 61, in run
 result = ctask.prepare(self.func, *args, **kwargs)
   File "/usr/share/vdsm/storage/task.py", line 1142, in prepare
 raise self.error
Timeout
Thread-59::DEBUG::2012-11-14 
02:16:23,767::task::568::TaskManager.Task::(_updateState) 
Task=`2e64438f-64c1-4f55-82bc-1f871bba1c56`::moving from state init -> state 
preparing
Thread-59::INFO::2012-11-14 02:16:23,768::logUtils::37::dispatcher::(wrapper) 
Run and protect: repoStats(options=None)
Thread-59::INFO::2012-11-14 02:16:23,768::logUtils::39::dispatcher::(wrapper) 
Run and protect: repoStats, Return response: {}
Thread-59::DEBUG::2012-11-14 
02:16:23,768::task::1151::TaskManager.Task::(prepare) 
Task=`2e64438f-64c1-4f55-82bc-1f871bba1c56`::finished: {}
Thread-59::DEBUG::2012-11-14 
02:16:23,768::task::568::TaskManager.Task::(_updateState) 
Task=`2e64438f-64c1-4f55-82bc-1f871bba1c56`::moving from state preparing -> 
state finished
Thread-59::DEBUG::2012-11-14 
02:16:23,768::resourceManager::809::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-59::DEBUG::2012-11-14 
02:16:23,768::resourceManager::844::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-59::DEBUG::2012-11-14 
02:16:23,768::task::957::TaskManager.Task::(_decref) 
Task=`2e64438f-64c1-4f55-82bc-1f871bba1c56`::ref 0 aborting False
Thread-65::DEBUG::2012-11-14 
02:16:33,921::task::568::TaskManager.Task::(_updateState) 
Task=`2f5bd3d8-ada8-4931-a76c-fff7dad79d3d`::moving from state init -> state 
preparing
Thread-65::INFO::2012-11-14 02:16:33,921::logUtils::37::dispatcher::(wrapper) 
Run and protect: repoStats(options=None)
Thread-65::INFO::2012-11-14 02:16:33,921::logUtils::39::dispatcher::(wrapper) 
Run and protect: repoStats, Return response: {}
Thread-65::DEBUG::2012-11-14 
02:16:33,922::task::1151::TaskManager.Task::(prepare) 
Task=`2f5bd3d8-ada8-4931-a76c-fff7dad79d3d`::finished: {}
Thread-65::DEBUG::2012-11-14 
02:16:33,922::task::568::TaskManager.Task::(_updateState) 
Task=`2f5bd3d8-ada8-4931-a76c-fff7dad79d3d`::moving from state preparing -> 
state finished
Thread-65::DEBUG::2012-11-14 
02:16:33,922::resourceManager::809::ResourceManager.Owner::(releaseAll) 
Owner.releaseAll requests {} resources {}
Thread-65::DEBUG::2012-11-14 
02:16:33,922::resourceManager::844::ResourceManager.Owner::(cancelAll) 
Owner.cancelAll requests {}
Thread-65::DEBUG::2012-11-14 
02:16:33,922::task::957::TaskManager.Task::(_decref) 
Task=`2f5bd3d8-ada8-4931-a76c-fff7dad79d3d`::ref 0 aborting False


Machine with VDSM is Running on Fedora17.
Any idea?

Thanks,
Itzik
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] A question about the SPM operation permission in VDSM

2012-10-29 Thread Shu Ming
Hi,

In the VDSM code about some SPM operations like HSM.deleteImage(), It is
found that VDSM doesn't check if the operation will be launched on a SPM
host or not. It only checks if the storage pool is already acquired by
one SPM host, but not necessary the same host as the SPM operation is
delivered to. The code is like this:

HSM.deleteImage()
{
...
HSM._spmSchedule()
{
self.validateSPM(spUUID) <--- Only check if the storage pool was
acquired by one host, but not necessary this host
}
...
}


So it really depends on the node management application AKA ovirt-engine
to dispatch the SPM operations to the right VDSM host. And the VDSM host
itself doesn't check if it is the SPM host which can execute the
operation. To me, it is a bit broken. When the engine query the VDSM
host who is the SPM host, it can get the right one. However, the host
may be broken for some reason after the engine believes it is the SPM
host and the host loses the SPM privilege, another host will take the
SPM role. Then the engine continue to send the SPM operations to the
broken host. As a result, the SPM operation will be launched on a
non-SPM host. So I think there is a small window of racing to corrupt
the VDSM hosts meta data. I think VDSM host should check if it is SPM
before the SPM job is scheduled. If the host lost the SPM role already,
It should fail the RPC call from the engine to let the engine to retry
the operation after engine knows the failure of the former call.


-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Users] Is this "dd" operation harmful?

2012-10-14 Thread Shu Ming
于 2012-10-14 5:15, Dan Kenigsberg:
> On Thu, Oct 11, 2012 at 11:38:19PM +0800, Shu Ming wrote:
>> After reading the code, every mailbox should be 4096 byte size.
>> And the total mailbox size is host * 4096. Ony one host is here, so
>> the total mailbox size here is 4096. why should the 'dd' operation
>> read 1024000 byte which is 1000K byte much lager than 4096 here?
> The controlling parameter is MAX_HOST_ID=250, not the number of current
> cluster members.
I am wondering if we can do some optimization here, like to read and
write the block size linear to the current cluster members.


>


-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Users] Is this "dd" operation harmful?

2012-10-11 Thread Shu Ming

After reading the code, every mailbox should be 4096 byte size.
And the total mailbox size is host * 4096. Ony one host is here, so
the total mailbox size here is 4096. why should the 'dd' operation
read 1024000 byte which is 1000K byte much lager than 4096 here?

2012-10-11 18:54, Dan Kenigsberg:

On Thu, Oct 11, 2012 at 03:44:25PM +0800, Shu Ming wrote:

Hi,

I found some "dd" operations were launched contiguously in my vdsm.log.
Is this harmful? How was this operation caused?

That's storage.storage_mailbox.SPM_MailMonitor, polling for lvextend
requests. dd is used, since in the old days, vdsm did not have
storage.fileUtils.DirectFile.

The behavior is expected, but I cannot say that it is harmless.
The mailbox should be high on
http://wiki.ovirt.org/wiki/Vdsm_TODO#refactoring
since forking so much is a waste, as well as using strings instead of
bytearrays. Making the module as a separate, testable entity, is important,
too.


 From vdsm.log:

Dummy-51000::DEBUG::2012-10-11
15:38:57,243::__init__::1249::Storage.Misc.excCmd::(_log) 'dd
if=/rhev/data-center/6f6d4801-7447-48ea-b516-627d83e7801e/mastersd/dom_md/inbox
iflag=direct,fullblock count=1 bs=1024000' (cwd None)



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] how engine get files from node???

2012-09-24 Thread Shu Ming

于 2012-9-21 20:01, Ryan Harper:

* Sheldon  [2012-09-21 04:42]:

Hi all,
   I have submitted a patch about watchdog device
http://gerrit.ovirt.org/#/c/7535/
   If we set 'dump' action for watchdog, qemu will generate a dump file.
   And I have get some feedback, how engine get these files?

   scp? or Will vdsm support new API to list and get these files?
   IMO, The new API should  list and get not only dump files,  but
also other kinds of files.

That's a good question.  We know that vdsm captured core files today.  I
don't think there is an API mechanism to retrieve the core files
remotely yet.

If we did propose such a call, what sets of files would we want to
include?  Would this be a query for fetching debug data for a single VM,
for the host (which would/could include all cores and dumps)?


At least, the log files of VDSM(/var/log/vdsm/), the log files of 
libvirt VMs( /var/log/libvirt/qemu/), VM panic cores and Qemu crash core 
dumps can be included.  Among them, the log files of VDSM are global 
ones and the others should be queried based on a specific VM.






--
Sheldon Feng
IBM Linux Technology Center




--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Engine-devel] is gerrit.ovirt.org down?

2012-09-12 Thread Shu Ming
It seems gerrit has downed for several times recently. Is there any 
special reason?

于 2012-9-12 22:45, Alon Bar-Lev:

yes.

- Original Message -

From: "Shireesh Anjal" 
To: engine-de...@ovirt.org
Sent: Wednesday, September 12, 2012 5:43:35 PM
Subject: [Engine-devel] is gerrit.ovirt.org down? 


___
Engine-devel mailing list
engine-de...@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


___
Engine-devel mailing list
engine-de...@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel




--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [RFC] GlusterFS domain specific changes

2012-09-06 Thread Shu Ming

于 2012-9-7 13:21, M. Mohan Kumar 写道:

On Thu, 6 Sep 2012 18:59:19 -0400 (EDT), Ayal Baron  wrote:


- Original Message -

- Original Message -

From: "M. Mohan Kumar" 
To: vdsm-devel@lists.fedorahosted.org
Sent: Wednesday, July 25, 2012 1:26:15 PM
Subject: [vdsm] [RFC] GlusterFS domain specific changes


We are developing a GlusterFS server translator to export block
devices
as regular files to the client. Using block devices to serve VM
images
gives performance improvements, since it avoids some file system
bottlenecks in the host kernel. Goal is to use one block device(ie
file
at the client side) per VM image and feed this file to QEMU to get
the
performance improvements. QEMU will talk to glusterfs server
directly
using libgfapi.

Currently we support only exporting Volume groups and Logical
Volumes. Logical volumes are exported as regular files to the
client.

Are you actually using LVM behind the scenes?
If so, why bother with exposing the LVs as files and not raw block devices?


Ayal,

The idea is to provide a FS interface for managing block devices. One
can mount the Block Device Gluster Volume and create a LV and size it
just by
  $ touch lv1
  $ truncate -s5G lv1

And other file commands can be used to clone LVs, snapshot LVs
  $ ln lv1 lv2 # clones
  $ ln -s lv1 lv1.sn # creates snapshot

Do we have special reason to use "ln"?
Why not use "cp" as the comannd to do the snapshot instead of "ln"?



By enabling this feature GlusterFS can directly export storage in
SAN. We are planning to add feature to export LUNs also as regular files
in future.


IMO, The major feature of GlusterFS is to export distributed local disks 
to the clients.
If we have SAN in the backend, that means the storage block devices 
should be exported
to clients natually.  Why do we need GlusterSF to export the block 
devices in SAN?




In GlusterFS terminology a volume capable of exporting block
devices is
created by specifying the 'Volume Group' (ie VG in Logical Volume
management). Block Device translator(BD xlator) exports this volume
group as a directory and LVs under it as regular files. In the
gluster
mount point creating a file results in creating a logical volume,
removing a file results in removing logical volume etc.

When a GlusterFS volume enabled with BD xlator is used, directory
creation in that gluster mount path is not supported because
directory
maps to Volume groups in BD xlator. But it could be an issue in
VDSM
environment when a new VDSM volume is created for GlusterFS domain,
VDSM
mounts the storage domain and creates directories under that and
create
files for vm image and other uses (like meta data).
Is it possible to modify this behavior in VDSM to use flat
structure
instead of creating directories and VM images and other files
underneath
it? ie for GlusterFS domain with BD xlator VDSM will not create any
directory and only creates all required files under the mount point
directory itself.

 From your description I think that the GlusterFS for block devices is
actually more similar to what happens with the regular block domains.
You should probably need to mount the share somewhere in the system
and
then use symlinks to point to the volumes.

Create a regular block domain and look inside
/rhev/data-center/mnt/blockSD,
you'll probably get the idea of what I mean.

That said we'd need to come up with a way of extending the LVs on the
gluster server when required (for thin provisioning).

Why? if it's exposed as a file that probably means it supports sparseness.  
i.e. if this becomes a new type of block domain it should only support 
'preallocated' images.


For start using the LVs we will always do truncate for the required
size, it will resize the LV. I didn't get what you are mentioning about
thin-provisioning, but I have a dumb code using dm-thin targets showing
BD xlators can be extended to use dm-thin targets for thin-provisioning.

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [RFC]about the implement of text-based console

2012-09-03 Thread Shu Ming

于 2012-8-31 5:26, Adam Litke 写道:

On Thu, Aug 30, 2012 at 11:32:02AM +0800, Xu He Jie wrote:

Hi,

   I submited a patch for text-based console
http://gerrit.ovirt.org/#/c/7165/

the issue I want to discussing as below:
1. fix port VS dynamic port

Use fix port for all VM's console. connect console with 'ssh
vmUUID@ip -p port'.
Distinguishing VM by vmUUID.


   The current implement was vdsm will allocated port for console
dynamically and spawn sub-process when VM creating.
In sub-process the main thread responsible for accept new connection
and dispatch output of console to each connection.
When new connection is coming, main processing create new thread for
each new connection. Dynamic port will allocated
port for each VM and use range port. It isn't good for firewall rules.


   so I got a suggestion that use fix port. and connect console with
'ssh vmuuid@hostip -p fixport'. this is simple for user.
We need one process for accept new connection from fix port and when
new connection is coming, spawn sub-process for each vm.
But because the console only can open by one process, main process
need responsible for dispatching console's output of all vms and all
connection.
So the code will be a little complex then dynamic port.

   So this is dynamic port VS fix port and simple code VS complex code.

 From a usability point of view, I think the fixed port suggestion is nicer.
This means that a system administrator needs only to open one port to enable
remote console access.  If your initial implementation limits console access to
one connection per VM would that simplify the code?


Another thing we want to take care is the security.  Enabling one port 
will make all
console output accessable to the user.  We should take care about this 
to ensure that
one common user can not see the console of other vms belonging to 
another user.







--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail: shum...@cn.ibm.com or 
shum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] why password is needed in virsh after vdsm is installed?

2012-08-22 Thread Shu Ming


On 2012-8-22 19:47, xiaxia347os wrote:

Hello, guys
  after VDSM is installed, when I want to use virsh to list VMs on 
local host,
it prompt to input password and username. What could I do to make the 
connection
automatically succeed instead of input username and password 
interactively?


Do you mean to just remove the password? Or make the input process 
automatically done by virsh?


  By the way, I am also wondering which authentication method is used 
for libvirt
and vm migration in VDSM, I'd like to try these method for another 
usercase.
SASL is used for authentication between  libvirt and VDSM, also libvirt 
is configured by VDSM.
I think what your problem here is how a host can authenticate the 
incoming request from the other
host for VM migration to itself,  not the authentication between VDSM 
and libvirt.
I believe you can configure libvirt to use tls talking to the remote 
host as a certain level of
authentication between hosts without inputting username and password 
interactively.  Also, a host
must have all the client certificates for every other hosts to make sure 
it can authenticated by

the other hosts in the same cluster.


Best Regards.
Wenchao Xia


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel



--
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: 86-10-82451626  Tieline: 9051626 E-mail:shum...@cn.ibm.com  
orshum...@linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, 
Beijing 100193, PRC

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] Fatal error when migrate the VM with disk on iscsi data center

2012-07-25 Thread Shu Ming

Hi,

I am testing the VM migration in oVirt engine 3.1 bteta release and 
successfully  did the migration test on NFS date center.  However, the 
test of VM migration on iSCSI data center failed.
I dumped the error and warning segments from the vdsm logs on both the 
source and destination host.  Please ignore the time stamp out of sync 
between the two hosts.

_
_*The vdsm log segment  from the source host:*

66-4da5-8310-6dcc970e5367`::migration downtime thread exiting
Thread-115166::ERROR::2012-07-25 
23:18:55,953::vm::176::vm.Vm::(_recover) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::operation failed: Failed to 
connect to remote libvirt URI qemu+tcp://9.181.129.110/system
Dummy-115148::DEBUG::2012-07-25 
23:18:55,981::__init__::1249::Storage.Misc.excCmd::(_log) 'dd 
if=/rhev/data-center/0b9a4ea4-d487-11e1-b614-5254001498c4/mastersd/dom_md/inbox 
iflag=direct,fullblock count=1 bs=1024000' (cwd None)
Thread-115166::ERROR::2012-07-25 23:18:56,028::vm::240::vm.Vm::(run) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Failed to migrate

Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 223, in run
self._startUnderlyingMigration()
  File "/usr/share/vdsm/libvirtvm.py", line 451, in 
_startUnderlyingMigration

None, maxBandwidth)
  File "/usr/share/vdsm/libvirtvm.py", line 491, in f
ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", 
line 82, in wrapper

ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1034, in 
migrateToURI2
if ret == -1: raise libvirtError ('virDomainMigrateToURI2() 
failed', dom=self)
libvirtError: operation failed: Failed to connect to remote libvirt URI 
qemu+tcp://9.181.129.110/system



*The vdsm log segment from the destination host:*

Thread-1577::INFO::2012-07-25 
23:19:50,407::API::601::vds::(_getNetworkIp) network None: using 0
Thread-1577::INFO::2012-07-25 23:19:50,407::API::228::vds::(create) 
vmContainerLock acquired by vm 11e06222-2d66-4da5-8310-6dcc970e5367
Thread-1578::DEBUG::2012-07-25 
23:19:50,411::vm::564::vm.Vm::(_startUnderlyingVm) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Start
Thread-1577::DEBUG::2012-07-25 23:19:50,411::API::244::vds::(create) 
Total desktops after creation of 11e06222-2d66-4da5-8310-6dcc970e5367 is 1
Thread-1578::DEBUG::2012-07-25 
23:19:50,411::vm::568::vm.Vm::(_startUnderlyingVm) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::_ongoingCreations acquired
Thread-1577::DEBUG::2012-07-25 
23:19:50,412::libvirtvm::2438::vm.Vm::(waitForMigrationDestinationPrepare) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::migration destination: 
waiting 36s for path preparation
Thread-1578::INFO::2012-07-25 
23:19:50,412::libvirtvm::1285::vm.Vm::(_run) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::VM wrapper has started
Thread-1578::WARNING::2012-07-25 
23:19:50,413::vm::398::vm.Vm::(getConfDevices) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Unknown type found, device: 
'{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': 
{'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}' 
found


Thread-1578::ERROR::2012-07-25 
23:24:50,484::vm::604::vm.Vm::(_startUnderlyingVm) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::The vm start process failed

Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 584, in _startUnderlyingVm
self._waitForIncomingMigrationFinish()
  File "/usr/share/vdsm/libvirtvm.py", line 1572, in 
_waitForIncomingMigrationFinish

self._connection.lookupByUUIDString(self.id),
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", 
line 82, in wrapper

ret = f(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2608, in 
lookupByUUIDString
if ret is None:raise libvirtError('virDomainLookupByUUIDString() 
failed', conn=self)
libvirtError: Domain not found: no domain with matching uuid 
'11e06222-2d66-4da5-8310-6dcc970e5367' - Timed out (did not receive 
success event)
Thread-1578::DEBUG::2012-07-25 
23:24:50,485::vm::920::vm.Vm::(setDownStatus) 
vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Changed state to Down: 
Domain not found: no domain with matching uuid '11e0622



Any idea?

--
Shu Ming 
IBM China Systems and Technology Laboratory

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Verify the storage data integrity after some storage operations with test cases

2012-07-17 Thread Shu Ming

On 2012-7-17 23:21, Saggi Mizrahi wrote:

Actually setting up isos and installing an OS is an overkill IMHO.
Using libguestfs seems simpler as it has python bindings.

What you could do is:
1. use libguest fs to format a file system on an image
Do you mean the test case use libguest fs to format a file without 
interaction with VM?



2. Put files on said file system with libguestfs
Do you mean the test case put files to the files system with libguestfs 
directly without interaction with VM?

Or even VM is not running?


3. Snapshot

If we want to test online snapshot, we should have a running VM now.


4. run fsck with libguestfs
5. rinse
6. repeast

If you don't trust fsck to detect all issues you can use libguestfs to get an 
md5sum of the raw drive and make sure that after a snapshot it stays the same.


- Original Message -

From: "Shu Ming" 
To: "VDSM Project Development" 
Sent: Monday, July 16, 2012 10:28:25 PM
Subject: [vdsm] Verify the storage data integrity after some storage operations 
with test cases

Hi,

   To verify the storage data integrity after some storage operations
like snapshot, merging by VDSM.  Here are the test cases I am
pondering.
I would like to know your feedback about these thoughts.

1) An customized ISO image with the agent  required prepared for
bringing up a VM in VDSM
2) The test case will inform VDSM to create a VM from the customized
ISO
image
3) The test case will install an IO application to the VM
3) The test case communicate with the VDSM to inform the IO
application
in the VM to write some data intentionally.
4) The test case sends the commands to VDSM do some storage operation
like disk snapshot, volume merging, &etc.
   Say snapshot operation here for an example.
5) VDSM then tell the test case the result of the operation like the
name of the snapshot.
6) Test case can read the  snapshot made to verify the snapshot with
the
data written in 3).
   Note: currently, there is no tool to read the snapshot image
directly.  We can restart the VM with the snapshot as
   the active disk and tell the IO application in the VM to read
   the
data writen before for test case.  And test case can compare
the data read with the data it informs the application in 3).
7) If the two data matches, the storage operation succeed or it
fails.

In order to write such a test case, these VDSM features will be
required.

1) VDSM can create a VM from a specific ISO image  (Almost works)
2) Test case can install an IO application to the VM by VDSM (by
ovirt-agent?)
3) Test case must have some protocols with the IO application in VM
for
passing the command to the VM and returning the result from the VM
   to the test case(by ovirt-agent?).
4) The IO application can be seen as an test agent.  We may extend
the
existing agent like ovirt-agent as the IO application.


--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel




--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] Verify the storage data integrity after some storage operations with test cases

2012-07-16 Thread Shu Ming

Hi,

 To verify the storage data integrity after some storage operations 
like snapshot, merging by VDSM.  Here are the test cases I am pondering.

I would like to know your feedback about these thoughts.

1) An customized ISO image with the agent  required prepared for 
bringing up a VM in VDSM
2) The test case will inform VDSM to create a VM from the customized ISO 
image

3) The test case will install an IO application to the VM
3) The test case communicate with the VDSM to inform the IO application 
in the VM to write some data intentionally.
4) The test case sends the commands to VDSM do some storage operation 
like disk snapshot, volume merging, &etc.

 Say snapshot operation here for an example.
5) VDSM then tell the test case the result of the operation like the 
name of the snapshot.
6) Test case can read the  snapshot made to verify the snapshot with the 
data written in 3).
 Note: currently, there is no tool to read the snapshot image 
directly.  We can restart the VM with the snapshot as
 the active disk and tell the IO application in the VM to read the 
data writen before for test case.  And test case can compare

  the data read with the data it informs the application in 3).
7) If the two data matches, the storage operation succeed or it fails.

In order to write such a test case, these VDSM features will be required.

1) VDSM can create a VM from a specific ISO image  (Almost works)
2) Test case can install an IO application to the VM by VDSM (by 
ovirt-agent?)
3) Test case must have some protocols with the IO application in VM for 
passing the command to the VM and returning the result from the VM

 to the test case(by ovirt-agent?).
4) The IO application can be seen as an test agent.  We may extend the 
existing agent like ovirt-agent as the IO application.



--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm

2012-07-16 Thread Shu Ming
rt a supportable API.  XMLRPC is a perfectly reasonable
remote/wire protocol
and I think we should continue using it as a base for the
next
generation API.
Using a schema will ensure that the new API is
well-structured.

There major problems with XML-RPC (and to some extent with
REST
as
well) are high call overhead and no two way communication
(push
events). Basing on XML-RPC means that we will never be able to
solve
these issues.

I am not sure I am ready to conceed that XML-RPC is too slow
for
our
needs.  Can
you provide some more detail around this point and possibly
suggest an
alternative that has even lower overhead without sacrificing
the
ubiquity and
usability of XML-RPC?  As far as the two-way communication
point,
what
are the
options besides AMQP/ZeroMQ?  Aren't these even worse from an
overhead
perspective than XML-RPC?  Regarding two-way communication: you
can
write AMQP
brokers based on the C API and run one on each vdsm host.
Assuming
the C API
supports events, what else would you need?

I personally think that using something like AMQP for inter-node
communication and engine - node would be optimal.  With a rest
interface
that just send messages though something like AMQP.

I would also not dismiss AMQP so soon
we want a bug with more than a single listener at engine side
(engine, history db, maybe event correlation service).
collectd as a means for statistics already supports it as well.
I'm for having REST as well, but not sure as main one for a
consumer
like ovirt engine.

I agree that a message bus could be a very useful model of
communication between
ovirt-engine components and multiple vdsm instances.  But the
complexities and
dependencies of AMQP do not make it suitable for use as a
low-level
API.  AMQP
will repel new adopters.  Why not establish a libvdsm that is more
minimalist
and can be easily used by everyone?  Then AMQP brokers can be
built
on top of
the stable API with ease.  All AMQP should require of the
low-level
API are
standard function calls and an events mechanism.


Thanks
Robert

The current XML-RPC API contains a lot of decencies and
inefficiencies and we
would like to retire it as soon as we possibly can. Engine
would
like us to
move to a message based API and 3rd parties want something
simple
like REST so
it looks like no one actually wants to use XML-RPC. Not even
us.

I am proposing that AMQP brokers and REST APIs could be
written
against the
public API.  In fact, they need not even live in the vdsm
tree
anymore if that
is what we choose.  Core vdsm would only be responsible for
providing
libvdsm
and whatever language bindings we want to support.

If we take the libvdsm route, the only reason to even have a
REST
bridge is only to support OSes other then Linux which is
something
I'm not sure we care about at the moment.

That might be true regarding the current in-tree
implementation.
However, I can
almost guarantee that someone wanting to write a web GUI on top
of
standalone
vdsm would want a REST API to talk to.  But libvdsm makes this
use
case of no
concern to the core vdsm developers.


I do think that having C supportability in our API is a good
idea,
but the
current API should not be used as the base.

Let's _start_ with a schema document that describes today's
API
and
then clean
it up.  I think that will work better than starting from
scratch.
   Once my
schema is written I will post it and we can 'patch' it as a
community
until we
arrive at a 1.0 version we are all happy with.

+1

Ok.  Redoubling my efforts to get this done.  Describing the
output of
list(True) takes awhile :)



___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel

--
Adam Litke 
IBM Linux Technology Center

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel





___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm

2012-07-12 Thread Shu Ming

On 2012-7-12 20:41, Adam Litke wrote:

On Thu, Jul 12, 2012 at 08:11:17AM +0800, Shu Ming wrote:

Basically,  my understanding is that we can generate two versions of
libvdsm from the schema file for both the node and the management
application.  First, the transportation protocols(XMLRPC, REST-API)
will depend on libvdsm(node version) to export the APIs to remote
management application.  Secondly, the management application can
use libvdsm(application version ) to emit the remote call to the
node.   Also, transportation protocols like REST API and XML RPC API
can also be generated automatically by the schema file with C, Java,
Python bindings.

I think this might be a bit too complex of a model.  Here's how I see it...

The schema generates C/gObject code which can be compiled into libvdsm.  We can
use the gObject introspection library to automatically generate language
bindings for Java, Python, Perl, etc.

Do you mean Java, Python, Perl, etc bindings of libvdsm?



The libvdsm library talks to vdsmd using a wire protocol that works locally and
remotely.  This wire protocol is completely hidden from library users.  It's an
implementation detail that can be changed later if necessary.  Today I would
recommend that we use xmlrpc.  This means that ovirt-engine or another remote
program could use libvdsm in the exact same manner as a local program.  The
library user just needs to call libvdsm.connect(uri).


Except libvdsm.connet() or libvdsm.disconnet(), do you expect the engine 
will not change any of its xml-rpc interfaces now after libvdsm is 
introduced?  I mean engine may not want to change much of its current 
code, so it expect the least change.




Finally, REST and AMQP bridges would be written solely against libvdsm.  These


So we should write REST, AMQP bridges manually with Java, Python, Perl 
language on various bindings of libvdsm?



bridges are probably not suitable for code generation (but we can revisit that
as a separate issue because it's up to the bridge writer to determine the best
approach).


On 2012-7-12 2:29, Saggi Mizrahi wrote:

I'm sorry, but I don't really understand the drawing

- Original Message -

From: "Shu Ming" 
To: "Adam Litke" 
Cc: vdsm-devel@lists.fedorahosted.org
Sent: Wednesday, July 11, 2012 10:24:49 AM
Subject: Re: [vdsm] [RFC] An alternative way to provide a supported interface 
-- libvdsm

Adam,

Maybe,  I don't fully understand your proposal.  Here is my
understanding of libvdsm in the picture. Please check the following
link
for the picture.

http://www.ovirt.org/wiki/File:Libvdsm.JPG


http://www.ovirt.org/wiki/File:Libvdsm.JPG

On 2012-7-9 21:56, Adam Litke wrote:

On Fri, Jul 06, 2012 at 03:53:08PM +0300, Itamar Heim wrote:

On 07/06/2012 01:15 AM, Robert Middleswarth wrote:

On 07/05/2012 04:45 PM, Adam Litke wrote:

On Thu, Jul 05, 2012 at 03:47:42PM -0400, Saggi Mizrahi wrote:

- Original Message -

From: "Adam Litke" 
To: "Saggi Mizrahi" 
Cc: "Anthony Liguori" , "VDSM Project
Development" 
Sent: Thursday, July 5, 2012 2:34:50 PM
Subject: Re: [RFC] An alternative way to provide a supported
interface -- libvdsm

On Wed, Jun 27, 2012 at 02:50:02PM -0400, Saggi Mizrahi wrote:

The idea of having a supported C API was something I was
thinking
about doing
(But I'd rather use gobject introspection and not schema
generation) But the
problem is not having a C API is using the current XML RPC
API as
it's base

I want to disect this a bit to find out exactly where there
might be
agreement
and disagreement.

C API is a good thing to implement - Agreed.

I also want to use gobject introspection but I don't agree
that using
glib
precludes the use of a formalized schema.  My proposal is that
we
write a schema
definition and generate the glib C code from that schema.

I agree that the _current_ xmlrpc API makes a pretty bad base
from
which to
start a supportable API.  XMLRPC is a perfectly reasonable
remote/wire protocol
and I think we should continue using it as a base for the next
generation API.
Using a schema will ensure that the new API is
well-structured.

There major problems with XML-RPC (and to some extent with REST
as
well) are high call overhead and no two way communication (push
events). Basing on XML-RPC means that we will never be able to
solve
these issues.

I am not sure I am ready to conceed that XML-RPC is too slow for
our
needs.  Can
you provide some more detail around this point and possibly
suggest an
alternative that has even lower overhead without sacrificing the
ubiquity and
usability of XML-RPC?  As far as the two-way communication
point, what
are the
options besides AMQP/ZeroMQ?  Aren't these even worse from an
overhead
perspective than XML-RPC?  Regarding two-way communication: you
can
write AMQP
brokers based on the C API and run one on each vdsm host.
  Assuming
the C API
supports events, what else would you ne

Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm

2012-07-11 Thread Shu Ming
Basically,  my understanding is that we can generate two versions of 
libvdsm from the schema file for both the node and the management 
application.  First, the transportation protocols(XMLRPC, REST-API) will 
depend on libvdsm(node version) to export the APIs to remote management 
application.  Secondly, the management application can use 
libvdsm(application version ) to emit the remote call to the node.   
Also, transportation protocols like REST API and XML RPC API can also be 
generated automatically by the schema file with C, Java, Python bindings.


On 2012-7-12 2:29, Saggi Mizrahi wrote:

I'm sorry, but I don't really understand the drawing

- Original Message -

From: "Shu Ming" 
To: "Adam Litke" 
Cc: vdsm-devel@lists.fedorahosted.org
Sent: Wednesday, July 11, 2012 10:24:49 AM
Subject: Re: [vdsm] [RFC] An alternative way to provide a supported interface 
-- libvdsm

Adam,

Maybe,  I don't fully understand your proposal.  Here is my
understanding of libvdsm in the picture. Please check the following
link
for the picture.

http://www.ovirt.org/wiki/File:Libvdsm.JPG


http://www.ovirt.org/wiki/File:Libvdsm.JPG

On 2012-7-9 21:56, Adam Litke wrote:

On Fri, Jul 06, 2012 at 03:53:08PM +0300, Itamar Heim wrote:

On 07/06/2012 01:15 AM, Robert Middleswarth wrote:

On 07/05/2012 04:45 PM, Adam Litke wrote:

On Thu, Jul 05, 2012 at 03:47:42PM -0400, Saggi Mizrahi wrote:

- Original Message -

From: "Adam Litke" 
To: "Saggi Mizrahi" 
Cc: "Anthony Liguori" , "VDSM Project
Development" 
Sent: Thursday, July 5, 2012 2:34:50 PM
Subject: Re: [RFC] An alternative way to provide a supported
interface -- libvdsm

On Wed, Jun 27, 2012 at 02:50:02PM -0400, Saggi Mizrahi wrote:

The idea of having a supported C API was something I was
thinking
about doing
(But I'd rather use gobject introspection and not schema
generation) But the
problem is not having a C API is using the current XML RPC
API as
it's base

I want to disect this a bit to find out exactly where there
might be
agreement
and disagreement.

C API is a good thing to implement - Agreed.

I also want to use gobject introspection but I don't agree
that using
glib
precludes the use of a formalized schema.  My proposal is that
we
write a schema
definition and generate the glib C code from that schema.

I agree that the _current_ xmlrpc API makes a pretty bad base
from
which to
start a supportable API.  XMLRPC is a perfectly reasonable
remote/wire protocol
and I think we should continue using it as a base for the next
generation API.
Using a schema will ensure that the new API is
well-structured.

There major problems with XML-RPC (and to some extent with REST
as
well) are high call overhead and no two way communication (push
events). Basing on XML-RPC means that we will never be able to
solve
these issues.

I am not sure I am ready to conceed that XML-RPC is too slow for
our
needs.  Can
you provide some more detail around this point and possibly
suggest an
alternative that has even lower overhead without sacrificing the
ubiquity and
usability of XML-RPC?  As far as the two-way communication
point, what
are the
options besides AMQP/ZeroMQ?  Aren't these even worse from an
overhead
perspective than XML-RPC?  Regarding two-way communication: you
can
write AMQP
brokers based on the C API and run one on each vdsm host.
  Assuming
the C API
supports events, what else would you need?

I personally think that using something like AMQP for inter-node
communication and engine - node would be optimal.  With a rest
interface
that just send messages though something like AMQP.

I would also not dismiss AMQP so soon
we want a bug with more than a single listener at engine side
(engine, history db, maybe event correlation service).
collectd as a means for statistics already supports it as well.
I'm for having REST as well, but not sure as main one for a
consumer
like ovirt engine.

I agree that a message bus could be a very useful model of
communication between
ovirt-engine components and multiple vdsm instances.  But the
complexities and
dependencies of AMQP do not make it suitable for use as a low-level
API.  AMQP
will repel new adopters.  Why not establish a libvdsm that is more
minimalist
and can be easily used by everyone?  Then AMQP brokers can be built
on top of
the stable API with ease.  All AMQP should require of the low-level
API are
standard function calls and an events mechanism.


Thanks
Robert

The current XML-RPC API contains a lot of decencies and
inefficiencies and we
would like to retire it as soon as we possibly can. Engine
would
like us to
move to a message based API and 3rd parties want something
simple
like REST so
it looks like no one actually wants to use XML-RPC. Not even
us.

I am proposing that AMQP brokers and REST APIs could be
written
against the
public API.  In fact, they need not even live in the vdsm tree
anymore if that

Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm

2012-07-11 Thread Shu Ming
7;patch' it as a community
until we
arrive at a 1.0 version we are all happy with.

+1

Ok.  Redoubling my efforts to get this done.  Describing the output of
list(True) takes awhile :)



___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] Creating a new VM with a template

2012-07-11 Thread Shu Ming

Hi,

I created a VM with from existing template and found that the image of 
the new VM actually copied the image from the template instead of 
sharing a base from the template.  I am wondering why the new VM doesn't 
use a new cow image as its image pointing to the base image in 
template.  By that way, all the new VMs will share a common parent image 
in template and save the space in the storage domain. Is there any other 
concern not to do this?


--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Readonly leases error in vdsm log

2012-07-05 Thread Shu Ming

On 2012-7-5 20:12, Deepak C Shetty wrote:


It looks like its trying to use the cdrom disk device as a disk lea


It seems this bug was hit:

*Bug 828633* <https://bugzilla.redhat.com/show_bug.cgi?id=828633> 
-Sanlock locking failed for readonly devices

https://bugzilla.redhat.com/show_bug.cgi?id=828633

--
Shu Ming 
IBM China Systems and Technology Laboratory

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager

2012-06-27 Thread Shu Ming

On 2012-6-27 0:18, Adam Litke wrote:

On Tue, Jun 26, 2012 at 04:47:35PM +0100, Daniel P. Berrange wrote:

On Tue, Jun 26, 2012 at 05:37:26PM +0300, Dan Kenigsberg wrote:

On Mon, Jun 25, 2012 at 04:19:28PM -0500, Adam Litke wrote:

1) Completely define the current XMLRPC API including all functions, parameters,
and return values.  Complex data structures can be broken down into their basic
types.  These are: int, str, bool, list, dict, typed-dict, enum.  I have already
started this process and am using Qemu's QAPI schema language.  You can see that
here [1].  For an example of what that looks like describing the vdsm API see
this snippet [2].

2) Import the parser/generator code from qemu for the above schema.  Vdsm will
require a few extensions such as typed-dictionaries, tuples, and type aliases.
Adapt the generator so that it can produce a libvdsm which provides API language
bindings for python, c, and java.

3) Implement a vdsm shell in terms of libvdsm.  In fact, this can be largely
auto-generated from the schema and accompanying documentation.  This can serve
to model how new transports can be written.  For example, an AMQP implementation
can be implemented entirely outside of the vdsm project if we wished.  It only
needs to talk to vdsm via libvdsm.

Easy as 1,2,3 :)

[1] 
http://git.qemu.org/?p=qemu.git;a=blob;f=qapi-schema.json;h=3b6e3468b440b4b681f321c9525a3d83bea2137a;hb=HEAD
[2] http://fpaste.org/rt96/

Probably more than you bargained for when asking for more info... :)

Indeed!

I am still at a loss why the languages take such a prominent place in
your choice for an API. A good API is easily consumable by any language.

I think you are both right here.  A good API is easily consumed from any
language, but this doesn't mean there is zero cost to starting to consume
it from a client. You either way to be able to auto-generate code for the
client side APIs in all your languages of choice, or even better, you want
the client side APIs to be just do runtime dynamic dispatch based on
published schema.

Thanks for commenting!  On one hand, dynamic dispatch seems attractive but I
think dramatically increases complexity on both the client and server sides.
Does anyone know of a prominent open source project that has been successful
with dynamic dispatch?  I am inclined to go with the C library approach because
it is tried and tested and it fits the model of other virtualization libraries
that I am familiar with.


If you go down the route of writing a C based libvdsm for VDSM, then my
recommendation would be to use the GObject APIs. You can then take full
advantage of the GObject Introspection capabilities to have full dynamic
dispatch in languages like python, perl, javascript, or full auto-generation
of code in Vala, C#, etc

Ahh, thanks for reminding me of this.  GObject definitely seems like the way to
go.  I assume there are no real implications for the schema definition and that
the heavy-lifting for GObject support would be limited to the C code generator.
Time to take a closer look at the GObject stuff.


Even if we can generate the various API bindings from the schema, we 
still need to change the transportation layers to connect the new APIs 
from the client to the VDSM services, whether or not the layer is 
XMLRPC, REST or any other way.  I really hate to change the 
transportation layer from time to time to know new APIs.  It is better 
to let the transportation layer to figure out the new API automatically 
and keep zero change with new APIs.





I certainly wouldn't waste time writing your own code-generator for all
the various languages, since that's just reinventing the wheel that
GObject Introspection already provides for the most part.

Agreed.  I would love to avoid this!




--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager

2012-06-26 Thread Shu Ming
heir basic
types.  These are: int, str, bool, list, dict, typed-dict, enum.  I have already
started this process and am using Qemu's QAPI schema language.  You can see that
here [1].  For an example of what that looks like describing the vdsm API see
this snippet [2].

2) Import the parser/generator code from qemu for the above schema.  Vdsm will
require a few extensions such as typed-dictionaries, tuples, and type aliases.
Adapt the generator so that it can produce a libvdsm which provides API language
bindings for python, c, and java.

3) Implement a vdsm shell in terms of libvdsm.  In fact, this can be largely
auto-generated from the schema and accompanying documentation.  This can serve
to model how new transports can be written.  For example, an AMQP implementation
can be implemented entirely outside of the vdsm project if we wished.  It only
needs to talk to vdsm via libvdsm.

Maybe API layer can load API interface definition from this schema.
Then we only define the api interface at one place. :)
I am not sure now. still thinking.

I think the code for vdsm-shell can be auto-generated from the schema at build
time.  I prefer this over dynamic discovery because it is much simpler.`


Easy as 1,2,3 :)

[1] 
http://git.qemu.org/?p=qemu.git;a=blob;f=qapi-schema.json;h=3b6e3468b440b4b681f321c9525a3d83bea2137a;hb=HEAD
[2] http://fpaste.org/rt96/

Probably more than you bargained for when asking for more info... :)






--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration

2012-06-25 Thread Shu Ming

On 2012-6-25 22:14, Deepak C Shetty wrote:

On 06/25/2012 07:47 AM, Shu Ming wrote:

On 2012-6-25 10:10, Andrew Cathrow wrote:


- Original Message -

From: "Andy Grover" 
To: "Shu Ming" 
Cc: libstoragemgmt-de...@lists.sourceforge.net, 
engine-de...@ovirt.org, "VDSM Project Development"


Sent: Sunday, June 24, 2012 10:05:45 PM
Subject: Re: [vdsm] [Engine-devel] RFC: Writeup on 
VDSM-libstoragemgmtintegration


On 06/24/2012 07:28 AM, Shu Ming wrote:

On 2012-6-23 20:40, Itamar Heim wrote:

On 06/23/2012 03:09 AM, Andy Grover wrote:

On 06/22/2012 04:46 PM, Itamar Heim wrote:

On 06/23/2012 02:31 AM, Andy Grover wrote:

On 06/18/2012 01:15 PM, Saggi Mizrahi wrote:

Also, there is no mention on credentials in any part of the
process.
How does VDSM or the host get access to actually modify the
storage
array? Who holds the creds for that and how? How does the user
set
this up?

It seems to me more natural to have the oVirt-engine use
libstoragemgmt
directly to allocate and export a volume on the storage array,
and
then
pass this info to the vdsm on the node creating the vm. This
answers
Saggi's question about creds -- vdsm never needs array
modification
creds, it only gets handed the params needed to connect and use
the
new
block device (ip, iqn, chap, lun).

Is this usage model made difficult or impossible by the current
software
architecture?

what about live snapshots?

I'm not a virt guy, so extreme handwaving:

vm X uses luns 1&  2

engine ->  vdsm "pause vm X"

that's pausing the VM. live snapshot isn't supposed to do so.

Tough we don't expect to do a pausing operation to the VM when live
snaphot is undergoing, the VM should be blocked on the access to
specific luns for a while.  The blocking time should be very short
to
avoid the storage IO time out in the VM.

OK my mistake, we don't pause the VM during live snapshot, we block
on
access to the luns while snapshotting. Does this keep live snapshots
working and mean ovirt-engine can use libsm to config the storage
array
instead of vdsm?

Because that was really my main question, should we be talking about
engine-libstoragemgmt integration rather than vdsm-libstoragemgmt
integration.
for snapshotting wouldn't we want VDSM to handle the coordination of 
the various atomic functions?


I think VDSM-libstoragemgmt will let the storage array itself to make 
the snapshot and handle the coordination of the various atomic 
functions. VDSM should be blocked on the following access to the 
specific luns which are under snapshotting.


I kind of agree. If snapshot is being done at the array level, then 
the array takes care of quiesing the I/O, taking the snapshot and 
allowing the I/O, why does VDSM have to worry about anything here, it 
should all happen transparently for VDSM, isnt it ?
The only issue is the quiesing may time out the VDSM io functions if it 
takes a non-trivial time.  Not sure if VDSM can handle all the time out 
gracefully.




--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Can somebody look why I don't have privilege to access bug #833425

2012-06-25 Thread Shu Ming

On 2012-6-20 18:37, Itamar Heim wrote:

On 06/20/2012 06:44 AM, Shu Ming wrote:

Hi,

My account was registered by IBM email account .
This bug should be VDSM specific and I think we should have enough
privilege to access VDSM bugs.



fixed now.

Thanks.  Itamar.   Also bug #784931 can not be accessed.  I was 
wondering if we can have a general rule for the bug category triage?


--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] About code segment in vdsm/storage/image.py

2012-06-25 Thread Shu Ming

On 2012-6-25 15:21, Shu Ming wrote:

Hi,

I am reading the code of merge() function in image.py and get one 
question about the code.  I wonder why we should get the volume 
parameters from the parent of the destination volume if the parent is 
existing instead from the destination volume itself?


1101if dstParentUUID != sd.BLANK_UUID:
volParams = vols[dstParentUUID].getVolumeParams()
else:
 1105   volParams = dstVol.getVolumeParams()

In my understanding,  the volParams is different in internal volume 
merge and base volume merge(merging to a top most base without parent).  
For internal volume merge, the volParams comes from the parent of the 
destination volume.  While for base volume merge, the volParams comes 
from the destination volume.  Is it correct?


--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] About code segment in vdsm/storage/image.py

2012-06-25 Thread Shu Ming

Hi,

I am reading the code of merge() function in image.py and get one 
question about the code.  I wonder why we should get the volume 
parameters from the parent of the destination volume if the parent is 
existing instead from the destination volume itself?


1101if dstParentUUID != sd.BLANK_UUID:
volParams = vols[dstParentUUID].getVolumeParams()
else:
 1105   volParams = dstVol.getVolumeParams()

--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration

2012-06-24 Thread Shu Ming

On 2012-6-25 10:10, Andrew Cathrow wrote:


- Original Message -

From: "Andy Grover" 
To: "Shu Ming" 
Cc: libstoragemgmt-de...@lists.sourceforge.net, engine-de...@ovirt.org, "VDSM 
Project Development"

Sent: Sunday, June 24, 2012 10:05:45 PM
Subject: Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt  
integration

On 06/24/2012 07:28 AM, Shu Ming wrote:

On 2012-6-23 20:40, Itamar Heim wrote:

On 06/23/2012 03:09 AM, Andy Grover wrote:

On 06/22/2012 04:46 PM, Itamar Heim wrote:

On 06/23/2012 02:31 AM, Andy Grover wrote:

On 06/18/2012 01:15 PM, Saggi Mizrahi wrote:

Also, there is no mention on credentials in any part of the
process.
How does VDSM or the host get access to actually modify the
storage
array? Who holds the creds for that and how? How does the user
set
this up?

It seems to me more natural to have the oVirt-engine use
libstoragemgmt
directly to allocate and export a volume on the storage array,
and
then
pass this info to the vdsm on the node creating the vm. This
answers
Saggi's question about creds -- vdsm never needs array
modification
creds, it only gets handed the params needed to connect and use
the
new
block device (ip, iqn, chap, lun).

Is this usage model made difficult or impossible by the current
software
architecture?

what about live snapshots?

I'm not a virt guy, so extreme handwaving:

vm X uses luns 1&  2

engine ->  vdsm "pause vm X"

that's pausing the VM. live snapshot isn't supposed to do so.

Tough we don't expect to do a pausing operation to the VM when live
snaphot is undergoing, the VM should be blocked on the access to
specific luns for a while.  The blocking time should be very short
to
avoid the storage IO time out in the VM.

OK my mistake, we don't pause the VM during live snapshot, we block
on
access to the luns while snapshotting. Does this keep live snapshots
working and mean ovirt-engine can use libsm to config the storage
array
instead of vdsm?

Because that was really my main question, should we be talking about
engine-libstoragemgmt integration rather than vdsm-libstoragemgmt
integration.

for snapshotting wouldn't we want VDSM to handle the coordination of the 
various atomic functions?


I think VDSM-libstoragemgmt will let the storage array itself to make 
the snapshot and handle the coordination of the various atomic 
functions. VDSM should be blocked on the following access to the 
specific luns which are under snapshotting.

Thanks -- Regards -- Andy
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel




--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [Engine-devel] RFC: Writeup on VDSM-libstoragemgmt integration

2012-06-24 Thread Shu Ming

On 2012-6-23 20:40, Itamar Heim wrote:

On 06/23/2012 03:09 AM, Andy Grover wrote:

On 06/22/2012 04:46 PM, Itamar Heim wrote:

On 06/23/2012 02:31 AM, Andy Grover wrote:

On 06/18/2012 01:15 PM, Saggi Mizrahi wrote:

Also, there is no mention on credentials in any part of the process.
How does VDSM or the host get access to actually modify the storage
array? Who holds the creds for that and how? How does the user set
this up?


It seems to me more natural to have the oVirt-engine use 
libstoragemgmt
directly to allocate and export a volume on the storage array, and 
then

pass this info to the vdsm on the node creating the vm. This answers
Saggi's question about creds -- vdsm never needs array modification
creds, it only gets handed the params needed to connect and use the 
new

block device (ip, iqn, chap, lun).

Is this usage model made difficult or impossible by the current 
software

architecture?


what about live snapshots?


I'm not a virt guy, so extreme handwaving:

vm X uses luns 1&  2

engine ->  vdsm "pause vm X"


that's pausing the VM. live snapshot isn't supposed to do so.


Tough we don't expect to do a pausing operation to the VM when live 
snaphot is undergoing, the VM should be blocked on the access to 
specific luns for a while.  The blocking time should be very short to 
avoid the storage IO time out in the VM.





engine ->  libstoragemgmt "snapshot luns 1, 2 to luns 3, 4"
engine ->  vdsm "snapshot running state of X to Y"
engine ->  vdsm "unpause vm X"


if engine had any failure before this step, the VM will remain paused. 
i.e., we compromised the VM to take a live snapshot.



engine ->  vdsm "change Y to use luns 3, 4"

?

-- Andy


___
Engine-devel mailing list
engine-de...@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel




--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] refactor clientif move the api implement to sub-module

2012-06-20 Thread Shu Ming

On 2012-6-21 10:35, Xu He Jie wrote:

于 2012年06月21日 03:10, Adam Litke 写道:

On Wed, Jun 20, 2012 at 06:08:57PM +0800, Xu He Jie wrote:

Hi, folks

I am trying move api implement to sub-module, then we don't need
singleton and passing clientif to anywhere. So I sent this mail to
descript the idea. I think it's good for review.

So I add api registeration mechanism. Other modules can register
it's api implement to api layer.
I try to move VM api and vm stuff(like clientif.vmContainer, etc) to
a new module called vmm. the vmm.VMM is similar with hsm. It's
responsiable for managing vm and register VM implement to api layer.
Same with hsm, I move all storage related api to hsm, and hsm will
register them to api layer.

   After all api move to submodule, we can rename clientif and
needn't passing client to any where. :)

   I have already submit a rough version to gerrit: 
http://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:wip_refactor_clientif,n,z
I took a cursory look at the code you have submitted.  I think you 
need to more
thoroughly describe your design.  I was particularly confused by your 
use of
Abstract Base Classes for this scenario.  Can you explain in more 
depth why you
have done this?  Is there a simpler way to accomplish what you need 
to do?  I
looked at your VMM patch and I am unsure why you need to define 
VMBase and
VMImpl separately.  It means declaring the set of functions in two 
separate

files.  Thanks for shining some more light on your methodology.


Adam, Thanks for reply!

I want to force people declare their API interface before they 
implement it. the vdsm API should be
standard, stable, it won't be modified frequently. And anyone should 
implement the API as the definition.
So I added Abstract Base Classes. The API interface must be declared 
at API.py, then people must

implement as the API.py defined.

I think it more clear for people want to implement the vdsm API. 
People can find the API definition from

API.py and implement it as the interface definition.

As example, VMBase is the vdsm VM API interface definition in API.py. 
VMBase is an Abstract Base Class.
so VMImpl is an implement of VM API. When I register VMImpl to API 
layer by the function 'API.registAPI()'.
'API.registAPI()' will check VMImpl to ensure VMImpl implement VM API 
as definition of VMBase in API.py.


If we didn't define API interface, so people will confuse what is vdsm 
API looks like and which implement of

API is standard.


Who will give the definition of the API interfaces? In my mind,  the one 
giving the definitions may be different form the implementors. What is 
the format of the definitions will be like?




Thanks!

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager

2012-06-20 Thread Shu Ming
API supportability guidelines
2. Decide on an API transport (e.g. REST, ZMQ, AMQP)
3. Make the API easily consumable (e.g. proper docs, example
code, extending
   the API, etc)
4. Implement the API itself

I agree with the list, but I'd like to work on the redesign
discussion so
that we're not doing all of 1-4 around the existing API that's
engine-focused.

I'm over due for posting a feature page on vdsm standalone mode,
and
I
have some other thoughts on various uses.

Some other paths of thought for use-cases I've been mulling over:

 - Simplifying using QEMU/KVM
 - consuming qemu via command line
 - can we manage/support developers launching qemu
 directly
 - consuming qemu via libvirt
 - can we integrate with systems that are already
 using
 libvirt

 - Addressing issues with libvirt
 - are there kvm specific features we can exploit that
 libvirt
 doesn't?
 
 - Scale-up/fail-over

 - can we support a single vdsm node, but allow for
 building
 up
 clusters/groups without bringing in something like
 ovirt-engine
 - can we look at decentralized fail-over for reliability
 without
 a central mgmt server?

 - pluggability
 - can we support an API that allows for third-party
 plugins
 to
 support new features or changes in implementation?

 - kvm tool integration into the API
 - there are lots of different kvm virt tools for various
 tasks
 and they are all stand-alone tools.  Can we integrate
 their
 use into the node level API.  Think libguestfs,
 virt-install,
 p2v/v2v tooling.  All of these are available, but there
 isn't
 an
 easy way to use this tools through an API.

 - host management operations
 - vdsm already does some host level configuration (see
   networking e.g.) it would be good to think about
   extending
 the API to cover other areas of configuration and updates
 - hardware enumeration
 - driver level information
 - storage configuration
 (we've got a bit of a discussion going around
  libstoragemgmt here)

 - performance monitoring/debugging
 - is the host collecting enough information to do
 debug/perf
 analysis
 - can we support specific configurations of a host that
 optimize
 for specific workloads
 - and can we do this in the API such that
 third-parties
 can
 supply and maintain specific workload configurations


All of these are dependent on one another and the permutations
are
endless.
This is why I think we should try and work on each one
separately.
All
discussions will be done openly on the mailing list and until
the
final version
comes out nothing is set in stone.

If you think you have anything to contribute to this process,
please do so
either by commenting on the discussions or by sending
code/docs/whatever
patches. Once the API solidifies it will be quite difficult to
change
fundamental things, so speak now or forever hold your peace.
Note
that this is
just an introductory email. There will be a quick follow up
email
to kick start
the discussions.




___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ry...@us.ibm.com



--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ry...@us.ibm.com



___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Can somebody look why I don't have privilege to access bug #833425

2012-06-20 Thread Shu Ming

Thanks.  It worked.

On 2012-6-20 18:37, Itamar Heim wrote:

On 06/20/2012 06:44 AM, Shu Ming wrote:

Hi,

My account was registered by IBM email account .
This bug should be VDSM specific and I think we should have enough
privilege to access VDSM bugs.



fixed now.




--
Shu Ming 
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] Can somebody look why I don't have privilege to access bug #833425

2012-06-19 Thread Shu Ming

Hi,

My account was registered by IBM email account .  
This bug should be VDSM specific and I think we should have enough 
privilege to access VDSM bugs.


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [virt-node] VDSM as a general purpose virt host manager

2012-06-18 Thread Shu Ming
fs, virt-install,
 p2v/v2v tooling.  All of these are available, but there isn't
 an
 easy way to use this tools through an API.

 - host management operations
 - vdsm already does some host level configuration (see
   networking e.g.) it would be good to think about
   extending
 the API to cover other areas of configuration and updates
 - hardware enumeration
 - driver level information
 - storage configuration
 (we've got a bit of a discussion going around
  libstoragemgmt here)

 - performance monitoring/debugging
 - is the host collecting enough information to do debug/perf
 analysis
 - can we support specific configurations of a host that
 optimize
 for specific workloads
 - and can we do this in the API such that third-parties
 can
 supply and maintain specific workload configurations


All of these are dependent on one another and the permutations are
endless.
This is why I think we should try and work on each one separately.
All
discussions will be done openly on the mailing list and until the
final version
comes out nothing is set in stone.

If you think you have anything to contribute to this process,
please do so
either by commenting on the discussions or by sending
code/docs/whatever
patches. Once the API solidifies it will be quite difficult to
change
fundamental things, so speak now or forever hold your peace. Note
that this is
just an introductory email. There will be a quick follow up email
to kick start
the discussions.




___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ry...@us.ibm.com



___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] RFC: Writeup on VDSM-libstoragemgmt integration

2012-06-18 Thread Shu Ming
er/pool free/used 
space, raid type etc.


Need to make sure above info is listed in a coherent way across arrays 
(number of LUNs, raid type used? free/total per container/pool, per 
LUN?. Also need I/O statistics wherever possible.



_______
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Change in vdsm[master]: Make vdsm/caps.py PEP8 clean

2012-06-13 Thread Shu Ming

Dan,

Thanks for your work to merge these files.  I notice that there are 
three files conflicting with the master branch not yet merged.  It is 
because the change these files are big and not synced and get a higher 
possibility to conflict with other changes.  In order to save both the 
maintainers and the submitter's time, I think we should have a protocol 
to merge these big files.  If the maintainer think the files are ready, 
the maintainer should inform the submitter to rebase the patches with 
the latest master head in one day(considering the time zone difference, 
one day is a must).  Also, it is the submitter's responsibility to 
rebase the patches time to time, but not necessary every day.


Pending files:
a3cda3e Make vdsm/caps.py PEP8 cleanZhengsheng
b2cb196 Make vdsm/clientIF.py PEP8 cleanZhengsheng
970c465 Make vdsm/guestIF.py PEP8 cleanZhengsheng



On 2012-6-14 2:35, dan...@redhat.com wrote:

Dan Kenigsberg has posted comments on this change.

Change subject: Make vdsm/caps.py PEP8 clean
..


Patch Set 3: I would prefer that you didn't submit this

currently requires non-trivial rebase.

and btw, sorry of the excessive noise of my scripted push of former patches.

--
To view, visit http://gerrit.ovirt.org/4801
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I000b2e63f31d1e2ca64e05d367b6b9e1b79d2cdf
Gerrit-PatchSet: 3
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Shu Ming
Gerrit-Reviewer: Dan Kenigsberg
Gerrit-Reviewer: Shu Ming
Gerrit-Reviewer: Zhou Zheng Sheng




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] __init__ module in VDSM?

2012-06-08 Thread Shu Ming

Hi,

When I check my vdsm.log,   I found those logs.   Based on the log 
format, the log should be printed by a module with name "__init__".  
However,  after checking the whole files of VDSM workspace,  I couldn't 
find a "__init__.py" file with line 1249.  My question is where is the 
"__init__.py"  file under VDSM?


Thread-807::DEBUG::2012-06-08 
00:01:10,730::__init__::1249::Storage.Misc.excCmd::(_log) '/bin/rpm -q 
--qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None)
Thread-807::DEBUG::2012-06-08 
00:01:10,752::__init__::1249::Storage.Misc.excCmd::(_log) SUCCESS:  
= '';  = 0
Thread-807::DEBUG::2012-06-08 
00:01:10,753::__init__::1249::Storage.Misc.excCmd::(_log) '/bin/rpm -q 
--qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' 
(cwd None)
Thread-807::DEBUG::2012-06-08 
00:01:10,773::__init__::1249::Storage.Misc.excCmd::(_log) SUCCESS:  
= '';  = 0


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] About the pep8cleaning topic branch

2012-06-07 Thread Shu Ming

Hi,

I am working on pep8cleaning topic branch and include twelve different 
files in the first batch of topic branch merging into the upstream.  And 
I found the most challenging job was to keep the topic branch synced 
with the latest master head, especially PEP8 clean  can change a file 
across all the lines of the file.  So I am really hoping to narrow the 
time window of merging the topic branch into  master.  So I really 
appreciate that more people can help to review these patches and approve 
the patches.


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] A Tool for PEP 8 Patches to Find Code Logic Changes

2012-06-07 Thread Shu Ming

On 2012-6-7 21:26, Adam Litke wrote:

On Thu, Jun 07, 2012 at 12:03:30PM +0800, Zhou Zheng Sheng wrote:

Hi,

Since there is no coverage report on tests in vdsm, if a PEP 8 patch
passes the tests, we still not sure if there is no mistake in it.
Viewing the diff reports on all the changes consumes a lot of time, and
some small but fatal mistakes(like misspelling variable name) can easily
be ignored by human eyes.

So I have a try on the compiler module of Python. I write a tool named
'pydiff'. pydiff parses two Python scripts into Abstract Syntax Trees.
These data structures can reflect the logic of the code, and pydiff
performs a recursive compare on the trees. Then pydiff reports
differences and the corresponding line numbers. In this way, pydiff
ignores code style changes, and reports only logical changes of the code.

I think this tool can save us a lot of time. After a PEP 8 patch passes
vdsm tests and pydiff, I will get some confidence on the patch and it
probably does not break anything in vdsm.

This is a very nice tool.  Thanks for sharing it.  I would like to see all
authors of PEP8 patches use this to check their patches for semantic
correctness.  This should greatly improve our ability to complete the PEP8
cleanup quickly.


Yes, I agree with you.  Also, we should merge this tool into vdsm as a 
helper for PEP8 clean work.






Here is a usage example:

 test_o.py 
def foo(a, b):
pass

if __name__ == '__main__':
A = [1, 2, 3]
print (4, 5, 6), \
"over"
foo(1, 2)
print 'Hello World'


 test_n.py 
def foo(a, b):
pass

if __name__ == '__main__':
A = [1,
2, 3]
print (4, 5, 6), "over"
fooo(
1, 2)
print ('Hello '
'World')


Some differences of the files are just a matter of style. The only
significant difference is the function call "foo()" is misspelled in
"test_n.py".

Run pydiff.py, it will report:

$ python pydiff.py test_*.py
1 difference(s)
first file: test_n.py
second file: test_o.py

((8, 'fooo'), (8, 'foo'))

This report tells us that 'fooo' in line 8 of "test_n.py" is different
from 'foo' in line 8 of "test_o.py".


It can also find insertions or deletions. Here is another simple example:

 old.py 
print 'Hello 1'
print 'Hello 2'
print 'Hello 3'
print 'Hello 4'
print 'Hello 5'

 new.py 
print 'Hello 1'
print 'Hello 3'
print 'Hello 4'
print 'Hello 5'
print 'Hello 5'

Run pydiff:

$ pydiff old.py new.py
2 difference(s)
first file: old.py
second file: new.py

((2, Printnl([Const('Hello 2')], None)), (2, None))

((5, None), (5, Printnl([Const('Hello 5')], None)))

Here "((2, Printnl([Const('Hello 2')], None)), (2, None))" means there
is a print statement in line 2 of old.py, but no corresponding statement
in new.py, so we can know the statement is deleted in new.py.
"((5, None), (5, Printnl([Const('Hello 5')], None)))" means there is a
print statement in line 5 of new.py, but no corresponding statement in
old.py, so we can know the statement is inserted in new.py.


Sometimes the change in code logic is acceptable, for example, change
"aDict.has_key(Key)" into "Key in aDict". pydiff can report a difference
in this case, but it is up to the user to judge whether it's acceptable.
pydiff is just a tool to help you finding these changes.

I hope it can be helpful for PEP 8 patch reviewers. If you find any
bugs, please let me know. The script is in the attachment.

--
Thanks and best regards!

Zhou Zheng Sheng / 周征晟
E-mail: zhshz...@linux.vnet.ibm.com
Telephone: 86-10-82454397




___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel





--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] About the work of "New repository system"

2012-05-30 Thread Shu Ming

On 2012-5-28 19:36, Ayal Baron wrote:


- Original Message -

Hi,
I am noticing that Saggi post a set of patches to gerrit named
"[WIP]
New repository system".  The latest update was post in Jan 9 and the
latest update of depending patch was post in Feb 19.  It seems that
the
patches are pending on "Refactor upgrade process to facilitate
repository format conversions".  I am wondering if there is any

Repository format conversions is now in.
We're now starting to look at these again.

Do you mean "Orthogonal storage repository conversion"?
http://gerrit.ovirt.org/#change,3045

The latest status shows it is still under working.




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] About the problems to create a topic branch in gerrit

2012-05-23 Thread Shu Ming

On 2012-5-24 9:56, Anthony Liguori wrote:

On 05/23/2012 08:44 PM, Shu Ming wrote:

No feedback or don't understand the problem I described?
On 2012-5-23 23:45, Shu Ming wrote:

Hi,

Recently, I am working on combining smaller pep8 clean works into 
one big
batch. Because the smaller patches come from different folks with 
different
change logs, when I tried to push these changes into gerrit to 
create a pep8
topic branch, I hope the change logs can be kept in the topic 
branch. However,
I found I can not push the patches with others' name. Errors are 
like the

blow. So I have two proposals to fix this.


Can you share the output of:

git log --format="%ae, %ce, %s" origin/master..HEAD

in the branch you're trying to push?

Yes.  That is the branch I am trying to push.

shuming@kvm-rhel-01 vdsm-pep8]$ git log --format="%ae, %ec, %s" 
origin/master..HEAD

zhshz...@linux.vnet.ibm.com, c, clean PEP 8 problems in vdsm/guestIF.py
zhshz...@linux.vnet.ibm.com, c, clean PEP 8 problems in vdsm/clientIF.py
zhshz...@linux.vnet.ibm.com, c, clean PEP 8 problems in vdsm/caps.py
m...@linux.vnet.ibm.com, c, Fixing pep8 in vdsm/define.py
shao...@linux.vnet.ibm.com, c, change the code style of 
before_vm_start.py for P
shao...@linux.vnet.ibm.com, c, change the code style of BindingXMLRPC.py 
for PEP
shao...@linux.vnet.ibm.com, c, change the code style of 
before_vm_start.py for P
shao...@linux.vnet.ibm.com, c, change the code style of 
SecureXMLRPCServer.py fo
shao...@linux.vnet.ibm.com, c, change the code style of devicemapper.py 
for PEP8

m...@linux.vnet.ibm.com, c, Fixing pep8 in vdsm/hooks.py
shao...@linux.vnet.ibm.com, c, change the code style of volume.py for 
PEP8 compl

xiaw...@linux.vnet.ibm.com, c, pep8 clean vdsm/storage/hba.py




The second column should be "shum...@linux.vnet.ibm.com".  If it's 
not, I assume it's because you let someone else push to your 
repository or pulled from there repository.  Gerrit won't let you do that.


I cherry-picked others' patch to my workspace and want to preserve the 
change log of others'.  But gerrit don't let me to push those change 
logs back with the patches.





One way to work around this is to do the following:

$ mkdir patches
$ git format-patch -o patches/ origin/master
$ git reset --hard origin/master
$ git am -s patches/
$ rm -rf patches

This will effectively change committer to be you for all of the 
commits.  The '-s' flag for git am also adds your Signed-off-by.  If 
it's already there, you can omit that option.


Regards,

Anthony Liguori



I) Allow the submitter to push the patches signed off by others to 
gerrit.
II) Let the maintainer to pull the patches from my workspace to 
merge the

patches.
III) Collapse the change logs from different folks into one log 
signed off by
me and then push that big patch into gerrit. Others's signature will 
go way, bad.

IV) other ideas?

---
remote: Resolving deltas: 66% (26/39)
remote:
remote: ERROR: In commit f81413cdc87208f6a72a639798816064421c8b3a
remote: ERROR: committer email address zhshz...@linux.vnet.ibm.com
remote: ERROR: does not match your user account.
remote: ERROR:
remote: ERROR: The following addresses are currently registered:
remote: ERROR: shum...@linux.vnet.ibm.com
remote: ERROR: smin...@gmail.com
remote: ERROR:
remote: ERROR: To register an email address, please visit:
remote: ERROR: http://gerrit.ovirt.org/#settings,contact
remote:
remote:
To ssh://sm...@gerrit.ovirt.org:29418/vdsm.git
! [remote rejected] HEAD -> refs/for/master/pep8cleaning (invalid 
committer)
error: failed to push some refs to 
'ssh://sm...@gerrit.ovirt.org:29418/vdsm.git'










--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] About the problems to create a topic branch in gerrit

2012-05-23 Thread Shu Ming

No feedback or don't understand the problem I described?
On 2012-5-23 23:45, Shu Ming wrote:

Hi,

Recently,  I am working on combining smaller pep8 clean works into one 
big batch.  Because the smaller patches come from different folks with 
different change logs,  when I tried to push these changes into gerrit 
to create a pep8 topic branch,  I hope the change logs can be kept in 
the topic branch.  However,  I found I can not push the patches with 
others' name.  Errors are like the blow.   So I have two proposals to 
fix this.


I)  Allow the submitter to push the patches signed off by others to 
gerrit.
II)  Let the maintainer to pull the patches from my workspace to merge 
the patches.
III) Collapse the change logs from different folks into one log signed 
off by me and then push that big patch into gerrit.  Others's 
signature will go way, bad.

IV) other ideas?

---
remote: Resolving deltas:  66% (26/39)
remote:
remote: ERROR:  In commit f81413cdc87208f6a72a639798816064421c8b3a
remote: ERROR:  committer email address zhshz...@linux.vnet.ibm.com
remote: ERROR:  does not match your user account.
remote: ERROR:
remote: ERROR:  The following addresses are currently registered:
remote: ERROR:shum...@linux.vnet.ibm.com
remote: ERROR:smin...@gmail.com
remote: ERROR:
remote: ERROR:  To register an email address, please visit:
remote: ERROR:  http://gerrit.ovirt.org/#settings,contact
remote:
remote:
To ssh://sm...@gerrit.ovirt.org:29418/vdsm.git
 ! [remote rejected] HEAD -> refs/for/master/pep8cleaning (invalid 
committer)
error: failed to push some refs to 
'ssh://sm...@gerrit.ovirt.org:29418/vdsm.git'





--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] About the problems to create a topic branch in gerrit

2012-05-23 Thread Shu Ming

Hi,

Recently,  I am working on combining smaller pep8 clean works into one 
big batch.  Because the smaller patches come from different folks with 
different change logs,  when I tried to push these changes into gerrit 
to create a pep8 topic branch,  I hope the change logs can be kept in 
the topic branch.  However,  I found I can not push the patches with 
others' name.  Errors are like the blow.   So I have two proposals to 
fix this.


I)  Allow the submitter to push the patches signed off by others to gerrit.
II)  Let the maintainer to pull the patches from my workspace to merge 
the patches.
III) Collapse the change logs from different folks into one log signed 
off by me and then push that big patch into gerrit.  Others's signature 
will go way, bad.

IV) other ideas?

---
remote: Resolving deltas:  66% (26/39)
remote:
remote: ERROR:  In commit f81413cdc87208f6a72a639798816064421c8b3a
remote: ERROR:  committer email address zhshz...@linux.vnet.ibm.com
remote: ERROR:  does not match your user account.
remote: ERROR:
remote: ERROR:  The following addresses are currently registered:
remote: ERROR:shum...@linux.vnet.ibm.com
remote: ERROR:smin...@gmail.com
remote: ERROR:
remote: ERROR:  To register an email address, please visit:
remote: ERROR:  http://gerrit.ovirt.org/#settings,contact
remote:
remote:
To ssh://sm...@gerrit.ovirt.org:29418/vdsm.git
 ! [remote rejected] HEAD -> refs/for/master/pep8cleaning (invalid 
committer)
error: failed to push some refs to 
'ssh://sm...@gerrit.ovirt.org:29418/vdsm.git'


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Agenda for tomorrow's call

2012-05-20 Thread Shu Ming

How about the new repository system latest status?
http://gerrit.ovirt.org/#change,192

On 2012-5-21 3:55, Ayal Baron wrote:

Hi all,

I would like to discuss the following on our upcoming call:

- reviewers are missing!
- reviewers/verifiers are missing for pep8 patches. I would like to
   ask a volunteer to aggregate them all in one branch, and get some
   folks from Red Hat QE to run some sanity test on them.
- functional tests: Wenchao Xia's http://gerrit.ovirt.org/#change,4454
   and Adam Litke's http://gerrit.ovirt.org/#change,4451
- Saggi's unicode fixes to betterPopen
- Stories about negative flows hurt by commit 1676396f18cf5c300d87e181
   "Change safelease APIs to match SANLock flow"
- Upcoming oVirt-3.1 release: when to break from master branch?

Anyone else has more interesting stuff? We can skip my bullets for a few
of yours if we do not have time.

Regards,
Dan.
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] [libvirt] F17's libvirt takes comments into LIBVIRTD_ARGS

2012-05-16 Thread Shu Ming

On 2012-5-16 23:35, Daniel P. Berrange wrote:

On Wed, May 16, 2012 at 06:08:51PM +0300, Dan Kenigsberg wrote:

On Wed, May 16, 2012 at 11:05:16PM +0800, Shu Ming wrote:

On 2012-5-16 18:46, Dan Kenigsberg wrote:

On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote:

On 2012-5-14 7:30, Dan Kenigsberg wrote:

On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:

Hi,
   Recently, I found that my host in engine was always in a
"unassigned state" after the host node was installed.  After looking
into the vdsm.log,  it seemed that vdsm failed to call libvirt as an
error,  "libvirtError: Cannot write data: Broken pipe".   When I
started virsh in the host node at that time, a warning was given
"WARNING: no socket to connect to" and core dumped with "virsh
net-list".   It looks like that no right socket was created for
virsh to connect to libvirtd.  Any comments about this problem?  The
followings are my steps in the node:

[root@ovirt-node1 ~]# rpm -qa |grep vdsm
vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64
vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep libvirt

libvirt-daemon-0.9.11-1.fc17.x86_64
libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64
libvirt-client-0.9.11-1.fc17.x86_64
libvirt-daemon-config-network-0.9.11-1.fc17.x86_64
libvirt-python-0.9.11-1.fc17.x86_64

[root@ovirt-node1 ~]# virsh net-list
WARNING: no socket to connect to
Segmentation fault

I think that merits a libvirt bug. please attach strace output to
bugzilla.


[root@ovirt-node1 ~]#


[root@ovirt-node1 ~]# ps -ef |grep vdsm
root  1299 1  0 23:10 ?00:00:00 /usr/sbin/libvirtd
--listen # by vdsm

The command line of libvirt process is very odd - the comment that vdsm
puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad,
and may be related to Fedora 17's systemd services. Try to remove the
comment and restart libvirtd to see if this is the case.

The comment come from

[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd:

I know that (see my text above). However, in F16 and before, comments have been 
stripped
before being passed to commandline. Have you tested if all is well when
the commment is removed?

I removed the "#  by vdsm " line from the config file.  And
restarted the libvirtd and vdsmd service.
But no luck to make "virsh net-list" successful, still got
"Segmentation fault", while "virsh -c qemu:///system -r" worked.

Thanks. Please strace that segfault, there's a libvirt bug lying there.
And thanks again for finding the vdsm/libvirt configuration problem in
F17.

Specifically do the following

  # debuginfo-install libvirt
  # gdb --args virsh net-list
  (gdb) run
   ...wait for crash..
  (gdb) thread apply all backtrace

Regards,
Daniel

[root@ovirt-node1 ~]# debuginfo-install libvirt
Loaded plugins: langpacks, presto, refresh-packagekit
enabling fedora-debuginfo
enabling updates-debuginfo
No debuginfo packages available to install
[root@ovirt-node1 ~]# debuginfo-install libvirt 
--releasever=17 Loaded plugins: langpacks, presto, 
refresh-packagekit

enabling fedora-debuginfo
enabling updates-debuginfo
No debuginfo packages available to install
[root@ovirt-node1 ~]#

--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] F17's libvirt takes comments into LIBVIRTD_ARGS

2012-05-16 Thread Shu Ming

On 2012-5-16 18:46, Dan Kenigsberg wrote:

On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote:

On 2012-5-14 7:30, Dan Kenigsberg wrote:

On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:

Hi,
   Recently, I found that my host in engine was always in a
"unassigned state" after the host node was installed.  After looking
into the vdsm.log,  it seemed that vdsm failed to call libvirt as an
error,  "libvirtError: Cannot write data: Broken pipe".   When I
started virsh in the host node at that time, a warning was given
"WARNING: no socket to connect to" and core dumped with "virsh
net-list".   It looks like that no right socket was created for
virsh to connect to libvirtd.  Any comments about this problem?  The
followings are my steps in the node:

[root@ovirt-node1 ~]# rpm -qa |grep vdsm
vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64
vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep libvirt

libvirt-daemon-0.9.11-1.fc17.x86_64
libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64
libvirt-client-0.9.11-1.fc17.x86_64
libvirt-daemon-config-network-0.9.11-1.fc17.x86_64
libvirt-python-0.9.11-1.fc17.x86_64

[root@ovirt-node1 ~]# virsh net-list
WARNING: no socket to connect to
Segmentation fault

I think that merits a libvirt bug. please attach strace output to
bugzilla.


[root@ovirt-node1 ~]#


[root@ovirt-node1 ~]# ps -ef |grep vdsm
root  1299 1  0 23:10 ?00:00:00 /usr/sbin/libvirtd
--listen # by vdsm

The command line of libvirt process is very odd - the comment that vdsm
puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad,
and may be related to Fedora 17's systemd services. Try to remove the
comment and restart libvirtd to see if this is the case.

The comment come from

[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd:

I know that (see my text above). However, in F16 and before, comments have been 
stripped
before being passed to commandline. Have you tested if all is well when
the commment is removed?


I removed the "#  by vdsm " line from the config file.  And restarted 
the libvirtd and vdsmd service.
But no luck to make "virsh net-list" successful, still got "Segmentation 
fault", while "virsh -c qemu:///system -r" worked.


virsh # net-list
Segmentation fault
[root@ovirt-node1 ~]# virsh net-list
Segmentation fault
[root@ovirt-node1 ~]# virsh -c qemu:///system
Please enter your authentication name: ^C
[root@ovirt-node1 ~]# virsh -c qemu:///system -r
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
   'quit' to quit

virsh > net-list
Name State  Autostart
-
vdsm-ovirtmgmt   active yes

virsh >




Let's see what our friends in libvir-list think.

Dan.




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] libvirtError: Cannot write data: Broken pipe when vdsm try to call libvirt

2012-05-15 Thread Shu Ming

On 2012-5-14 7:30, Dan Kenigsberg wrote:

On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:

Hi,
   Recently, I found that my host in engine was always in a
"unassigned state" after the host node was installed.  After looking
into the vdsm.log,  it seemed that vdsm failed to call libvirt as an
error,  "libvirtError: Cannot write data: Broken pipe".   When I
started virsh in the host node at that time, a warning was given
"WARNING: no socket to connect to" and core dumped with "virsh
net-list".   It looks like that no right socket was created for
virsh to connect to libvirtd.  Any comments about this problem?  The
followings are my steps in the node:

[root@ovirt-node1 ~]# rpm -qa |grep vdsm
vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64
vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep libvirt

libvirt-daemon-0.9.11-1.fc17.x86_64
libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64
libvirt-client-0.9.11-1.fc17.x86_64
libvirt-daemon-config-network-0.9.11-1.fc17.x86_64
libvirt-python-0.9.11-1.fc17.x86_64

[root@ovirt-node1 ~]# virsh net-list
WARNING: no socket to connect to
Segmentation fault

I think that merits a libvirt bug. please attach strace output to
bugzilla.


[root@ovirt-node1 ~]#


[root@ovirt-node1 ~]# ps -ef |grep vdsm
root  1299 1  0 23:10 ?00:00:00 /usr/sbin/libvirtd
--listen # by vdsm

The command line of libvirt process is very odd - the comment that vdsm
puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad,
and may be related to Fedora 17's systemd services. Try to remove the
comment and restart libvirtd to see if this is the case.

The comment come from

[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd:


# Override the default config file
#LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf

# Listen for TCP/IP connections
# NB. must setup TLS/SSL keys prior to using this
#LIBVIRTD_ARGS="--listen"

# Override Kerberos service keytab for SASL/GSSAPI
#KRB5_KTNAME=/etc/libvirt/krb5.tab

# Override the QEMU/SDL default audio driver probing when
# starting virtual machines using SDL graphics
#
# NB these have no effect for VMs using VNC, unless vnc_allow_host_audio
# is enabled in /etc/libvirt/qemu.conf
#QEMU_AUDIO_DRV=sdl
#
#SDL_AUDIODRIVER=pulse
LIBVIRTD_ARGS=--listen # by vdsm
DAEMON_COREFILE_LIMIT=unlimited # by vdsm







/usr/share/vdsm/respawn --minlifetime 10 --daemon --masterpid
/var/run/vdsm/respawn.pid /usr/share vdsm/vdsm
vdsm  1919  1917  0 23:10 ?00:00:06 /usr/bin/python
/usr/share/vdsm vdsm
root  1940  1919  0 23:10 ?00:00:00 /usr/bin/sudo -n
/usr/bin/python /usr/share/vdsm/supervdsmServer.py
709dfdde-a668-4227-a206-3d8686b4cfa1 1919
root  1941  1940  0 23:10 ?00:00:00 /usr/bin/python
/usr/share/vdsm/supervdsmServer.py
709dfdde-a668-4227-a206-3d8686b4cfa1 1919
root  3711  3055  0 23:22 pts/000:00:00 vim /var/log/vdsm/vdsm.log
root  4358  4103  0 23:30 pts/100:00:00 grep --color=auto vdsm
[

[root@ovirt-node1 ~]# ps -ef |grep libvirtd
root  4421 1  2 23:31 ?00:00:00 /usr/sbin/libvirtd
--listen # by vdsm
root  4750  4103  0 23:31 pts/100:00:00 grep --color=auto libvirtd

Thanks for testing on F17!

Dan.




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] libvirtError: Cannot write data: Broken pipe when vdsm try to call libvirt

2012-05-13 Thread Shu Ming

Hi,
  Recently, I found that my host in engine was always in a "unassigned 
state" after the host node was installed.  After looking into the 
vdsm.log,  it seemed that vdsm failed to call libvirt as an error,  
"libvirtError: Cannot write data: Broken pipe".   When I started virsh 
in the host node at that time, a warning was given "WARNING: no socket 
to connect to" and core dumped with "virsh net-list".   It looks like 
that no right socket was created for virsh to connect to libvirtd.  Any 
comments about this problem?  The followings are my steps in the node:


[root@ovirt-node1 ~]# rpm -qa |grep vdsm
vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64
vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep libvirt

libvirt-daemon-0.9.11-1.fc17.x86_64
libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64
libvirt-client-0.9.11-1.fc17.x86_64
libvirt-daemon-config-network-0.9.11-1.fc17.x86_64
libvirt-python-0.9.11-1.fc17.x86_64

[root@ovirt-node1 ~]# virsh net-list
WARNING: no socket to connect to
Segmentation fault
[root@ovirt-node1 ~]#


[root@ovirt-node1 ~]# ps -ef |grep vdsm
root  1299 1  0 23:10 ?00:00:00 /usr/sbin/libvirtd 
--listen # by vdsm
vdsm  1917 1  0 23:10 ?00:00:00 /bin/bash -e 
/usr/share/vdsm/respawn --minlifetime 10 --daemon --masterpid 
/var/run/vdsm/respawn.pid /usr/share vdsm/vdsm
vdsm  1919  1917  0 23:10 ?00:00:06 /usr/bin/python 
/usr/share/vdsm vdsm
root  1940  1919  0 23:10 ?00:00:00 /usr/bin/sudo -n 
/usr/bin/python /usr/share/vdsm/supervdsmServer.py 
709dfdde-a668-4227-a206-3d8686b4cfa1 1919
root  1941  1940  0 23:10 ?00:00:00 /usr/bin/python 
/usr/share/vdsm/supervdsmServer.py 709dfdde-a668-4227-a206-3d8686b4cfa1 1919

root  3711  3055  0 23:22 pts/000:00:00 vim /var/log/vdsm/vdsm.log
root  4358  4103  0 23:30 pts/100:00:00 grep --color=auto vdsm
[

[root@ovirt-node1 ~]# ps -ef |grep libvirtd
root  4421 1  2 23:31 ?00:00:00 /usr/sbin/libvirtd 
--listen # by vdsm

root  4750  4103  0 23:31 pts/100:00:00 grep --color=auto libvirtd


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] About the work of "New repository system"

2012-05-11 Thread Shu Ming

Hi,
  I am noticing that Saggi post a set of patches to gerrit named "[WIP] 
New repository system".  The latest update was post in Jan 9 and the 
latest update of depending patch was post in Feb 19.  It seems that the 
patches are pending on "Refactor upgrade process to facilitate 
repository format conversions".  I am wondering if there is any update 
about these patches and if there is any new patch based on the latest 
VDSM code.  Recently, I am trying to test these code on the latest VDSM 
upstream.


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] gerrit problem when reviewing VDSM code

2012-05-07 Thread Shu Ming

On 2012-5-8 13:24, Itamar Heim wrote:

On 05/08/2012 04:50 AM, Shu Ming wrote:

Hi,
In my reviewing patches in gerrit for VDSM code, I found it was a bit
awkward to understand the diffs without much context above or below the
diffs. I hope I can read the unchanged lines around the diffs to get
more information. The unchanged lines should be in folding state when I
don't care about them and can be unfolded when I need more information.
What do you think about this?



just changed in gerrit the context to show you 'whole file' rather 
than '10 lines'?


Yes.  10 lines don't have much meaning to the people not familiar with 
the file, but still have an interesting to review the code.  More 
context will help them to review the code without patching the diffs 
into the original file.


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] gerrit problem when reviewing VDSM code

2012-05-07 Thread Shu Ming

Hi,
  In my reviewing patches in gerrit for VDSM code,  I found it was a 
bit awkward to understand the diffs without  much context above or below 
the diffs.  I hope I can read the unchanged lines around the diffs to 
get more information.  The unchanged lines should be in folding state  
when I don't care about them and can be unfolded when I need more 
information.  What do you think about this?


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] vds_bootstrap.py 's residency

2012-05-03 Thread Shu Ming

Hi,
  I am checking the VDSM and ovirt-engine workspace for 
"vds_bootstrap.py" file.  It was found that vds_bootstrap.py file was in 
VDSM workspace and was packaged into vdsm-bootstrap rpm package.   Also, 
it was found that in the host installation process, host node will try 
to get the "vds_bootstrap.py" from the engine server.  But 
"vds_bootstrap.py" is not included in any engine packages.  Does that 
mean we should install vdsm-bootstrap package into engine server?  Why 
not package this file into engine packages instead of vdsm packages?


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Introducing a validation test package to vdsm

2012-04-26 Thread Shu Ming
One more comment about the test package version.  Most likely, the 
package version will be the same version as the VDSM package version.  
The rule we need to consider is: Should we keep the back compatibility 
with the VDSM files? Say allow newer version test package running on 
older VDSM files? Or just allow run the same version of test package and 
VDSM package.One more comment about the test package version.  Most 
likely, the package version will be the same version as the VDSM package 
version.  The rule we need to consider is: Should we keep the back 
compatibility with the VDSM files? Say allow newer version test package 
running on older VDSM files? Or just allow run the same version of test 
package and VDSM package.


On 2012-4-26 21:57, Adam Litke wrote:

On Thu, Apr 26, 2012 at 05:24:29PM +0800, wenchao xia wrote:

Hello,
   vdsm now have UT suits for developer, but sometimes building and
installation machine is not the same one, and additional check is need
which is ignored at building time, so I think some test cases should be
also run on target machine to check potential errors, Then I want to
introduce a sub package as VT suits.
Purpose:
   UT: for developers, more likely a white box, running on building
environment.
   VT: for user and deployment, more likely a black box, running on
  product or testing environment, all known issue should be covered.

Supposed approach:
   1 modify building system to generate package: vdsm-VT.rpm.

I would prefer the package name 'vdsm-test.rpm' and this package should include
unit tests and verification tests.


   2 install as an option, after install, user type "vdsm-VT" would make
the test begin.

The test runner should be able to run the full suite of unit tests and
verification tests (with an option to run only unit tests or only verification
tests).  This can be the same program that we use in the build environment
except that it will set the PYTHONPATH differently to target the installed
files.


Planned details:
   1 Going to place cases in vdsm project in ./tests/VT.
   2 On installation will move some useful UT cases into VT.

If you follow my approach above, you would simply package the whole tests/
directory and no copying would be necessary.


   3 use same framework UT used.
   4 two sub dir in test/VT: user_case_test;general_test.

What is the difference between these two types of tests?


   It is just a scratch from my mind, so I'd like hear your opinions.

Thanks for the idea!  Do you have a sample test for the verification test suite?
Will it be your pipe deadlock test?




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Installation dependency for VDSM-4.9.6-0.109 package on FC16

2012-04-23 Thread Shu Ming
Yes,  I tried these.  It worked well except some gpg check errors which 
are harmless I think.

On 2012-4-23 20:04, Federico Simoncelli wrote:

Hi Shu,
  you can get sanlock and libvirt from fedora 17:

# yum --releasever=17 update libvirt
# yum --releasever=17 install sanlock

And lvm2 from my repository:

http://fsimonce.fedorapeople.org/lvm2/fedora-16/x86_64/




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] Installation dependency for VDSM-4.9.6-0.109 package on FC16

2012-04-22 Thread Shu Ming
   Requires: libvirt >= 0.9.10
   Installed: libvirt-0.9.6-4.fc16.x86_64 (@updates)
   libvirt = 0.9.6-4.fc16
   Available: libvirt-0.9.6-2.fc16.x86_64 (fedora)
   libvirt = 0.9.6-2.fc16
   Available: libvirt-0.9.6-5.fc16.x86_64 (updates)
   libvirt = 0.9.6-5.fc16
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@ovirt-node2 ~]#


--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Installing VDSM on an empty Fedora 16 machine?

2012-03-07 Thread Shu Ming

It is seems the authentication get some problems.

You can use this link as a install reference to disable the 
authentication .

http://www.ovirt.org/wiki/Installing_VDSM_from_rpm

On 2012-3-8 0:33, Saša Tomić wrote:


On 03/07/2012 04:24 PM, Dan Kenigsberg wrote:

Time to debug further in... Clues in /var/log/libvirtd.log?
Do you see anything when you run libvirtd --listen from the command 
line?
Dan. 

We are getting closer now, it appears that some paths are broken...

[stomic@localhost ~]$ sudo tail /var/log/libvirtd.log
2012-03-07 15:42:58.129+: 1590: debug : virRegisterDriver:783 : 
driver=0x739960 name=QEMU
2012-03-07 15:42:58.129+: 1590: debug : virRegisterDriver:807 : 
registering QEMU as driver 10
2012-03-07 15:42:58.129+: 1590: debug : virRegisterDriver:783 : 
driver=0x739f60 name=LXC
2012-03-07 15:42:58.129+: 1590: debug : virRegisterDriver:807 : 
registering LXC as driver 11
2012-03-07 15:42:58.129+: 1590: debug : virRegisterDriver:783 : 
driver=0x73aa40 name=UML
2012-03-07 15:42:58.129+: 1590: debug : virRegisterDriver:807 : 
registering UML as driver 12
2012-03-07 15:42:58.131+: 1590: debug : virHookCheck:115 : No hook 
script /etc/libvirt/hooks/daemon
2012-03-07 15:42:58.131+: 1590: debug : virHookCheck:115 : No hook 
script /etc/libvirt/hooks/qemu
2012-03-07 15:42:58.131+: 1590: debug : virHookCheck:115 : No hook 
script /etc/libvirt/hooks/lxc
2012-03-07 15:42:58.134+: 1590: error : 
virNetTLSContextCheckCertBasicConstraints:166 : The certificate 
/etc/pki/vdsm/certs/vdsmcert.pem basic constraints show a CA, but we 
need one for a server



[stomic@localhost ~]$ libvirtd --listen
2012-03-07 16:26:00.931+: 4893: info : libvirt version: 0.9.6, 
package: 4.fc16 (Fedora Project, 2011-12-19-20:27:37, 
x86-01.phx2.fedoraproject.org)
2012-03-07 16:26:00.931+: 4893: error : 
virNetTLSContextCheckCertFile:92 : Cannot read CA certificate 
'/etc/pki/CA/cacert.pem': No such file or directory


[stomic@localhost ~]$ locate cacert.pem
/etc/pki/vdsm/certs/cacert.pem

[stomic@localhost pki]$ ls -l /etc/pki/vdsm/certs/cacert.pem
lrwxrwxrwx. 1 root root   32 Mar  6 18:15 
/etc/pki/vdsm/certs/cacert.pem -> /etc/pki/vdsm/certs/vdsmcert.pem


[stomic@localhost pki]$ ls -l /etc/pki/vdsm/certs/vdsmcert.pem
-rw---. 1 vdsm kvm 1188 Mar  6 18:15 /etc/pki/vdsm/certs/vdsmcert.pem




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Installing VDSM on an empty Fedora 16 machine?

2012-03-07 Thread Shu Ming

On 2012-3-7 21:38, Saša Tomić wrote:

Hi to all,

Does anybody have step-by-step instructions how to install vdsm on an 
empty Fedora 16 machine?
I need a git version of vdsm (to compile from source), so the binary 
rpms will not do the work.


What I have so far is:
# for resolving the sanlock bug
sudo yum update --enablerepo=updates-testing
# installing additional packages for devel
sudo yum install @development-tools
sudo yum install fedora-packager
sudo yum install git autoconf automake gcc gcc-c++ pyflakes rpm-build
git clone http://gerrit.ovirt.org/p/vdsm.git
cd vdsm
./autogen.sh --system && make rpm
sudo yum -y install /home/stomic/rpmbuild/RPMS/x86_64/vdsm-*.rpm 
/home/stomic/rpmbuild/RPMS/noarch/vdsm-cli*.rpm

vdsClient -s 0 getVdsCaps


Does "vdsClient  0 getVdsCaps" without '-s" work? "-s" means secure 
connection.


  #This should show you Vdsm's response about its capabilities, but 
returns:

  # Connection to 0:54321 refused
sudo vdsClient -s 0 getVdsStats
  # If I try as root, nothing changes:
  # Connection to root@localhost.localdomain:54321 refused


Thank you in advance!
Sasha


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Failed dependencies when installing VDSM from git

2012-03-04 Thread Shu Ming

On 2012-3-4 19:29, Itzik Brown wrote:

Hi,

After running the following :
# git clone http://gerrit.ovirt.org/p/vdsm.git
# cd vdsm
# git fetch http://gerrit.ovirt.org/p/vdsm refs/changes/95/1695/10&&  git 
checkout FETCH_HEAD
# cd ~/rpmbuild/RPMS/x86_64
# rpm -Uvh vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64.rpm

I get the following error:

error: Failed dependencies:
 libvirt>= 0.9.8 is needed by 
vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64
 libvirt-python>= 0.9.8 is needed by 
vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64
 logrotate>= 3.8.0 is needed by 
vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64
 qemu-img>= 2:0.12.1.2-2.227 is needed by 
vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64
 qemu-kvm>= 2:0.12.1.2-2.227 is needed by 
vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64
 sanlock>= 1.8 is needed by 
vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64
 sanlock-python is needed by 
vdsm-4.9.4-0.95.gita6a2e31.el6.root1330852654.x86_64

The latest libvirt version from RHEVM Beta is:
libvirt-0.9.4-21.el6.x86_64

I have the following channels registered:

rhel-x86_64-rhev-mgmt-agent-6-beta
rhel-x86_64-server-6
rhel-x86_64-server-6-beta

Should I work with Fedora when  working with upstream ?

FC16 is more popular in developing VDSM.



Thanks,
Itzik
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Error using vdsClient

2012-02-28 Thread Shu Ming

On 2012-2-29 1:40, Deepak C Shetty wrote:

On 02/29/2012 01:01 AM, Douglas Landgraf wrote:

On 02/28/2012 12:21 PM, Deepak C Shetty wrote:

Hello,
Trying vdsClient for the first time.
 I am getting the below error for any cmd I am trying to use...

[root@llm65 ~]# vdsClient llm65.in.ibm.com getVGList
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 1972, in 
code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 524, in getVGList
vgs = self.s.getVGList()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1575, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in 
single_request

response = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1027, in getresponse
response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 371, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''


[root@llm65 ~]# vdsClient llm65.in.ibm.com getConnectedStoragePoolsList
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 1972, in 
code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 1056, in 
getConnectedStoragePoolsList

pools = self.s.getConnectedStoragePoolsList()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1575, in __request
verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in 
single_request

response = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1027, in getresponse
response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 371, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''

Note : I am running the above on the host (llm65) & already have 
configured FC, storage via OE
and via OE, I am able to create and run VMs also, but vdsClient 
fails to give the VG and SP lists.


If your vdsm is running with SSL, you must pass the argument -s 
(SSL), like:

vdsClient -s myHost command

Otherwise, -s should be ignored.

'Guess i was running with vdsm deafults so ssl was on. Its hard to 
relate the 'BadStatusLine" to ssl issue:)
`vdsClient -s llm65.in.ibm.com getConnectedStoragePoolsList` worked 
for me


instead of -s , -s 0 also worked, as I learnt from the 
vdsClient manpage.

Does 0 stand for localhost here ?


Yes. 0 means localhost here.  If you run the vdsClient in the same host 
as vdsmd, you can use "vdsClient 0 getVGList".





Please let me know if this is your case or not.

Thanks!



___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] VDSM host network configuration

2012-02-15 Thread Shu Ming

On 2012-2-16 0:05, Lei Li wrote:

Hi,

We are working on VDSM network REST APIs, to support the functions we 
need to get

the list of configured networks. I found that VDSM network has a function
'listNetworks' in configNetwork.py. It can get and display the current 
configured

network like this:

# python configNetwork.py list
Networks: ['bridge_one', 'bridge_three', 'bridge_two']
Vlans: []
Nics: ['eth0']
Bondings: []

But there are some problems with it. It can not display the defined 
networks after
host restart, but the created config file are still 
there(/etc/sysconfig/network-scripts/..).

Did I miss anything? Or Is there some way to avoid this?


Lei,
  How about the real network configurations after the host reboot?
You can run "brctl show" and "ifconfig all" to get those.  Also, service 
log can be checked to know if there

is any failure in the boot process





Your suggestion and thoughts would be appreciated.



Lei

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel



--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


[vdsm] About the wiki page installing_VDSM_from_rpm

2012-02-12 Thread Shu Ming

Hi,
  I was installing the VDSM packaging following this link:
http://www.ovirt.org/wiki/Installing_VDSM_from_rpm

It seems that the network configuration section is not correct.   In 
this wiki page, the static IP is assigned to the host network interface 
like eth0, em1 and the outing interface of the bridge is  attached  to 
the same host interface.  However, it didn't work in my server.  After 
moving the static IP into the bridge script *"ifcfg-ovirtmgmt*", the 
network configuration is OK now.  I was wondering if it was someone's 
typo to write this wiki.  See the change in below:


Add the following content into a new file named: 
*/etc/sysconfig/network-scripts/ifcfg-ovirtmgmt*:


DEVICE=ovirtmgmt
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=dhcp

*--->  Should be changed to:*

DEVICE=ovirtmgmt
TYPE=Bridge
ONBOOT=yes
DELAY=0
BOOTPROTO=static
IPADDR=XXX.XXX.XXX.XXX
NETMASK=255.255.255.0
GATEWAY=XXX.XXX.XXX.XXX

 

Add the following line into the configuration file of your out going 
interface (usually em1/eth0) the file is located at: 
*/etc/sysconfig/network-scripts/ifcfg-em1* (assuming the device is em1)


BRIDGE=ovirtmgmt

Full Example

DEVICE=em1
BOOTPROTO=static
IPADDR=192.168.1.212
NETMASK=255.255.255.0
ONBOOT=yes
BRIDGE=ovirtmgmt

---> should be changed to

DEVICE=em1
BOOTPROTO=static
ONBOOT=yes
BRIDGE=ovirtmgmt




--
Shu Ming
IBM China Systems and Technology Laboratory

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Live Merge and Storage Live Migration

2012-02-07 Thread Shu Ming
Another question about the Storage Live Migration wiki page, in the 
Engine-VDSM flow:


createVolumeCrossSD():
 domain1: [base(raw)]<-[snap1(qcow2)]<-+-(VM)
 domain2:  +-[snap2(qcow2)]


blockMigrate():
 domain1: [base(raw)]<-[snap1(qcow2)]<-+
 domain2:  +-[snap2(qcow2)]<-(VM)


I think there must have a storage quiesce time between the snap2 creation and 
setting the VM on top of snap2.
When the snap2 is being created, VM cannot access snap1. Only after VM is set 
on top of snap2, the quiesce period is end.


On 2012-2-7 4:34, Federico Simoncelli wrote:

During my endless journey back from FOSDEM (24 hours stuck in airports,
taxi and trains because of one of the largest blizzards in Italy ever),
I've been briefly working on the design for both live merge and storage
live migration:

http://www.ovirt.org/wiki/Features/Design/LiveMerge
http://www.ovirt.org/wiki/Features/Design/StorageLiveMigration

They're far for being complete but I'm not able to do anything better
now.

Please review if you're interested, the general idea should be correct
even though there are several unfinished parts.




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Live Merge and Storage Live Migration

2012-02-07 Thread Shu Ming

On 2012-2-7 20:23, Federico Simoncelli wrote:

- Original Message -

From: "Shu Ming"
To: "Federico Simoncelli"
Cc: "VDSM Project Development"
Sent: Tuesday, February 7, 2012 4:10:38 AM
Subject: Re: [vdsm] Live Merge and Storage Live Migration

I am just curious how HSM interact with SPM after HSM get merge
request no matter if they are in the same host.

At the moment the engine is responsible to poll the merge status on
the HSM (and restart it if failed). When it's successfully completed
the deleteVolume command must be issued to the SPM.

Sorry for confusing.  Please let me explain my question again.
In the Engine-VDSM Flow of "LiveMerge" wiki page, the engine will call 
the merge
interface from engine to HSM to request the merge.  I was wondering what 
HSM will do after it get the merge
request.  Does it call further interfaces in SPM by whatever way to make 
SPM to
commit the real merge operation or just complete the operation by it 
self? If it does,
how SPM will tell HSM the status of the merge 
operation?synchronizedly or asnchronizedly?




In the future we might consider the mailbox as mean of communication
between the HSM and the SPM, even though it's now enabled only for
block storage domains (and if the irs.vol_extend_policy is ON).

Still the entry point would be the deleteVolume call on the SPM.
Does it mean that deleteVolume call will be from HSM to SPM directly 
instead of from engine?





--
Shu Ming
IBM China Systems and Technology Laboratory

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] Live Merge and Storage Live Migration

2012-02-06 Thread Shu Ming
I am just curious how HSM interact with SPM after HSM get merge request 
no matter

if they are in the same host.
On 2012-2-7 4:34, Federico Simoncelli wrote:

During my endless journey back from FOSDEM (24 hours stuck in airports,
taxi and trains because of one of the largest blizzards in Italy ever),
I've been briefly working on the design for both live merge and storage
live migration:

http://www.ovirt.org/wiki/Features/Design/LiveMerge
http://www.ovirt.org/wiki/Features/Design/StorageLiveMigration

They're far for being complete but I'm not able to do anything better
now.

Please review if you're interested, the general idea should be correct
even though there are several unfinished parts.




--
Shu Ming
IBM China Systems and Technology Laboratory


___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel


Re: [vdsm] oVirt Live Snapshots

2012-02-02 Thread Shu Ming

  Can someone explain what is "DB" in this wiki page?
See,

Live snapshots operation extend regular snapshots as follow:

 * Create a locked snapshot in DB




On 2012-1-30 19:00, Federico Simoncelli wrote:

Hi,
   oVirt, and more specifically VDSM, is currently implementing the live
snapshot feature using the API/commands provided by libvirt and qemu.
It would be great if you could review the design and the current open
issues at:

   http://ovirt.org/wiki/Live_Snapshots

Thank you,



--
Shu Ming
IBM China Systems and Technology Laboratory

___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel