Re: [openstack-dev] Debian OpenStack packages switching to Py3 for Queens

[Jianghua Wang]

Thomas, H. and Bob,

  Please note only the scripts under "os_xenapi/dom0/etc/xapi.d/plugins/" will 
run in dom0 only. During deployment an OpenStack environment, we usually copy 
the plugins into dom0 from the installed package (installed in DomU). In this 
way, it helps us to ensure the plugins are from the same release as the 
remaining part (e.g. the wrapper APIs invoked by Nova/Neutron/Ceilometer). 
Otherwise if we split plugins out, it will be difficult to ensure the 
compatibility. So I'd suggest we keep these plugins in the package.

  I had a chat with Bob on how to resolve the python 2 issue by include these 
plugins. We think a solution is to rename those plugins without the .py suffix 
so that they won't be treated as python files. Thomas, please help to confirm 
if it works for packaging. I can take responsibility to handle the needed 
change in os-xenapi.

Thanks,
Jianghua


From: Bob Ball [mailto:bob.b...@citrix.com]
Sent: Friday, February 16, 2018 3:02 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Debian OpenStack packages switching to Py3 for 
Queens

Hi,

> If this code is meant to run on Dom0, fine, then we won't package it,
> but we also have to decouple that dependency from Nova, Neutron,
> Ceilometer etc... to either
communicate directly through an API
> endpoint or a light wrapper around it.

There is already a light wrapper here - other parts of os-xenapi provide the 
API to Nova/Neutron/etc which make calls through to the plugins in Dom0.

These projects should now know nothing about the actual plugins or how they are 
called.

Bob

From: Haïkel >
Sent: Thursday, 15 February 2018 6:39 p.m.
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: Re: [openstack-dev] Debian OpenStack packages switching to Py3 for 
Queens

2018-02-15 11:25 GMT+01:00 Bob Ball 
>:
> Hi Thomas,
>
> As noted on the patch, XenServer only has python 2 (and some versions of 
> XenServer even has Python 2.4) in domain0.  This is code that will not run in 
> Debian (only in XenServer's dom0) and therefore can be ignored or removed 
> from the Debian package.
> It's not practical to convert these to support python 3.
>
> Bob
>H.

We're not there yet but we also plan to work on migrating RDO to Python 3.
And I have to disagree, this code is called by other projects and their tests,
so it will likely be an impediment in migrating OpenStack to Python 3, not just
a "packaging" issue.

If this code is meant to run on Dom0, fine, then we won't package it,
but we also
have to decouple that dependency from Nova, Neutron, Ceilometer etc... to either
communicate directly through an API endpoint or a light wrapper around it.

Regards,
H.

> -Original Message-
> From: Thomas Goirand [mailto:z...@debian.org]
> Sent: 15 February 2018 08:31
> To: 
> openstack-dev@lists.openstack.org
> Subject: [openstack-dev] Debian OpenStack packages switching to Py3 for Queens
>
> Hi,
>
> Since I'm getting some pressure from other DDs to actively remove Py2 support 
> from my packages, I'm very much considering switching all of the Debian 
> packages for Queens to using exclusively Py3. I would have like to read some 
> opinions about this. Is it a good time for such move? I hope it is, because 
> I'd like to maintain as few Python package with Py2 support at the time of 
> Debian Buster freeze.
>
> Also, doing Queens, I've noticed that os-xenapi is still full of py2 only 
> stuff in os_xenapi/dom0. Can we get those fixes? Here's my patch:
>
> https://review.openstack.org/544809
>
> Cheers,
>
> Thomas Goirand (zigo)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] vGPUs support for Nova

[Jianghua Wang]

Sahid,

   Just share some background. XenServer doesn't expose vGPUs as mdev or pci 
devices. I proposed a spec about one year ago to make fake pci devices so that 
we can use the existing PCI mechanism to cover vGPUs. But that's not a good 
design and got strongly objection. After that, we switched to use the resource 
providers by following the advice from the core team.

Regards,
Jianghua

-Original Message-
From: Sahid Orentino Ferdjaoui [mailto:sferd...@redhat.com] 
Sent: Monday, September 25, 2017 11:01 PM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] vGPUs support for Nova

On Mon, Sep 25, 2017 at 09:29:25AM -0500, Matt Riedemann wrote:
> On 9/25/2017 5:40 AM, Jay Pipes wrote:
> > On 09/25/2017 05:39 AM, Sahid Orentino Ferdjaoui wrote:
> > > There is a desire to expose the vGPUs resources on top of Resource 
> > > Provider which is probably the path we should be going in the long 
> > > term. I was not there for the last PTG and you probably already 
> > > made a decision about moving in that direction anyway. My personal 
> > > feeling is that it is premature.
> > > 
> > > The nested Resource Provider work is not yet feature-complete and 
> > > requires more reviewer attention. If we continue in the direction 
> > > of Resource Provider, it will need at least 2 more releases to 
> > > expose the vGPUs feature and that without the support of NUMA, and 
> > > with the feeling of pushing something which is not 
> > > stable/production-ready.
> > > 
> > > It's seems safer to first have the Resource Provider work well 
> > > finalized/stabilized to be production-ready. Then on top of 
> > > something stable we could start to migrate our current virt 
> > > specific features like NUMA, CPU Pinning, Huge Pages and finally PCI 
> > > devices.
> > > 
> > > I'm talking about PCI devices in general because I think we should 
> > > implement the vGPU on top of our /pci framework which is 
> > > production ready and provides the support of NUMA.
> > > 
> > > The hardware vendors building their drivers using mdev and the 
> > > /pci framework currently understand only SRIOV but on a quick 
> > > glance it does not seem complicated to make it support mdev.
> > > 
> > > In the /pci framework we will have to:
> > > 
> > > * Update the PciDevice object fields to accept NULL value for
> > >    'address' and add new field 'uuid'
> > > * Update PciRequest to handle a new tag like 'vgpu_types'
> > > * Update PciDeviceStats to also maintain pool of vGPUs
> > > 
> > > The operators will have to create alias(-es) and configure 
> > > flavors. Basically most of the logic is already implemented and 
> > > the method 'consume_request' is going to select the right vGPUs 
> > > according the request.
> > > 
> > > In /virt we will have to:
> > > 
> > > * Update the field 'pci_passthrough_devices' to also include GPUs
> > >    devices.
> > > * Update attach/detach PCI device to handle vGPUs
> > > 
> > > We have a few people interested in working on it, so we could 
> > > certainly make this feature available for Queen.
> > > 
> > > I can take the lead updating/implementing the PCI and libvirt 
> > > driver part, I'm sure Jianghua Wang will be happy to take the lead 
> > > for the virt XenServer part.
> > > 
> > > And I trust Jay, Stephen and Sylvain to follow the developments.
> > 
> > I understand the desire to get something in to Nova to support 
> > vGPUs, and I understand that the existing /pci modules represent the 
> > fastest/cheapest way to get there.
> > 
> > I won't block you from making any of the above changes, Sahid. I'll 
> > even do my best to review them. However, I will be primarily 
> > focusing this cycle on getting the nested resource providers work 
> > feature-complete for (at least) SR-IOV PF/VF devices.
> > 
> > The decision of whether to allow an approach that adds more to the 
> > existing /pci module is ultimately Matt's.
> > 
> > Best,
> > -jay
> > 
> > 
> > __ OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: 
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> Nested resource providers is not merged or

Re: [openstack-dev] [third-party][ci] openstack CI VM template

[Jianghua Wang]

Ruijing,
   I'm not sure about other cases, but XenServer CI is using nested 
virtualization.
Regards,
Jianghua

From: Guo, Ruijing [mailto:ruijing@intel.com]
Sent: Tuesday, February 28, 2017 4:52 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [third-party][ci] openstack CI VM template

Hi, CI Team,

I'd like to know if openstack CI VM support nested virtualization.

Thanks,
-Ruijing
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] proposal to resolve a rootwrap problem for XenServer

[Jianghua Wang]

Thanks Ihar, Thierry and Bob. I think we've agreed to go with the 1st option - 
"Get Neutron to call XenAPI directly rather than trying to use a daemon".
I will refine the POC patch to make it ready for the formal review.

R.g.t the test, I did some basic test in a real lab manually and it worked as 
expected. For sure more tests will be done forwarding. We will evaluate if to 
change the XenServer CI to use the daemon mode once this fix got merged.

Regards,
Jianghua

-Original Message-
From: Ihar Hrachyshka [mailto:ihrac...@redhat.com] 
Sent: Thursday, November 03, 2016 10:40 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] proposal to resolve a rootwrap problem 
for XenServer

Bob Ball  wrote:

> Hi Ihar,
>
>> I am puzzled. Is Neutron the only component that need to call to dom0?
>
> No it's not.  Nova has similar code to call plugins in dom0[1], and 
> Ceilometer will also need to make the calls for some metrics not 
> exposed through the formal API.
>
> We don't want code duplication, and are working on a common os-xenapi 
> library which will include session management.
> It would, of course, make sense for Neutron to use this common library 
> when it is available to replace the session management already 
> existing[2], but I'd argue that as there is existing XenAPI session 
> management code, the refactor to avoid using a per-command rootwrap 
> should be independent of using the session code from os-xenapi.

Seems like you are in a position that requires you to hook into neutron 
processes somehow, and it’s either neutron itself (your solution), or a library 
used by neutron (oslo.rootwrap or similar). I understand why you picked the 
first option.

>
>> I would think that Neutron is not in business of handling hypervisor 
>> privilege isolation mechanics, and that some other components will 
>> handle that for Neutron (and other services that may need it), that’s 
>> why I suggested to consider oslo.* realm for the proposed code.
>
> This is less about hypervisor privilege isolation and more about the 
> location of the logical component being updated.  Neutron is assuming 
> that the OVS being updated is running in the same location as Neutron 
> itself.  For XenAPI that is not true; the OVS is running in the 
> hypervisor, whereas Neutron is running in a VM (or potentially 
> elsewhere entirely).
>
> If oslo.* is going to decide whether to run a command using a specific 
> abstraction or locally, then it would need some way of making that 
> decision - perhaps either command-based (very ugly and fragile) or 
> with the caller telling oslo.* what logical component was being 
> affected by the call.  The latter sounds to me much more as a 
> Neutron-specific decision.

I believe os-xenapi is a nice path forward. I understand it will take some time 
to shape.

As for the dom0/domU routing decision, yes, it’s indeed on Neutron to make it. 
But it does not mean that we could not rely on existing filtering mechanisms 
(oslo.rootwrap ‘.filters’ files) to define that decision. The fact that current 
netwrap script for Xen duplicates filters from rootwrap is unfortunate. It 
should be a single source of truth.

It would probably require some extensive work in the library, and considering 
that oslo folks moved the library into maintenance mode, it probably won’t 
happen. As for oslo.privsep, that would be a better place for such a feature, 
but we won’t get there in Ocata. Bummer…

I guess I will unblock the patch, and we’ll see what others think. I left some 
initial review comments.

>
>> Side note: if we are going to make drastic changes to existing Xen-wrap  
>> script, we should first have Xen
>> third-party CI testing running against it, not to introduce regressions.  
>> AFAIK it’s not happening right now.
>
> It already is running, and has been for several months - see "Citrix  
> XenServer CI"s "dsvm-tempest-neutron-network" job on  
> https://review.openstack.org/#/c/391308/ as an example.  The CI is  
> non-voting but if it were added to the neutron-ci group we would be very  
> happy to make it voting.

Oh right. It does not validate the new change though. Would be nice to see  
the new ‘daemon’-ic mode behaves in real world.

Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap problem for XenServer

[Jianghua Wang]

Thanks Doug. Please see my response inline starts with .

Jianghua

-Original Message-
From: Doug Hellmann [mailto:d...@doughellmann.com] 
Sent: Wednesday, November 2, 2016 9:31 PM
To: openstack-dev <openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap 
problem for XenServer

Excerpts from Jianghua Wang's message of 2016-11-02 04:14:48 +:
> Ihar and Tony,
>  Thanks for the input.
>  In order to run command in dom0, it uses XenAPI to create a session which 
> can be used to remotely call a plugin - netwrap which is located in dom0. The 
> netwrap plugin is executed as root. It will validate the command basing on 
> the allowed command list and execute it.
> The source code for netwrap is in neutron project: 
> https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/d
> rivers/openvswitch/agent/xenapi/etc/xapi.d/plugins/netwrap
> 
> So at least we can see there are two dependences: 
> 1. it depends on XenAPI which is XenServer specific.
> 2. it depends on Neutron's plugin netwrap.
> Is it acceptable to add such dependences in this common library of 
> oslo.rootwrap? 

Why would they need to be dependencies of oslo.rootwrap? They are dependencies 
of the driver, not the base library, right?

 With a second thought, I think we can pass the plugin name netwrap 
as a parameter to the rootwrap; so maybe not a dependence. But if we host the 
XenAPI session creation in oslo.rootwrap, I think we should import XenAPI in 
oslo.rootwrap. Then it is a dependence in the base library, isn't it?

> And most of the code in oslo.rootwrap is to:
> 1. spawn a daemon process and maintain the connection between the 
> client and daemon; 2. filter commands in the daemon process.
> But both can't be re-used for this XenAPI/XenServer case as the daemon 
> process is already running in dom0; the command filtering is done in dom0's 
> netwrap plugin. In order to hold this in oslo.rootwrap, it requires some 
> refactoring work to make it looks reasonable. Is it worthy to do that? 
> Specially by considering it has determined to replace oslo.wrap with 
> oslo.provsep?
> 
> Maybe it's a good option to cover this dom0 case in oslo.provsep at the 
> beginning. But it becomes more complicated. Maybe we can run a daemon process 
> in dom0 with the privileges set properly and listening on a dedicated tcp 
> port . But that's much different from the initial provsep design [1]. And 
> also it makes the mechanism very different from the current XenServer 
> OpenStack which is using XenAPI plugin. Anyway, I think we have long to go 
> with a good solution to cover it in provsep.

What sort of refactoring do you have in mind for privsep? I could see something 
analogous to the preexec_fn argument to subprocess.Popen() to let the XenServer 
driver ensure that its privileged process is running in dom0.

Sorry, I don't have a clear idea on refactorying in privsep now. It 
seems privsep attempts to create a daemon process and set caps for this daemon. 
But for XenServer/XenAPI, the daemon and caps in daemon seems useless. As it 
sends all commands to the a common XAPI daemon running in dom0. No additional 
daemon is needed. TBH I don't know how to apply caps at here. But to make it to 
resolve the current issue, what I'm thinking is to create a XenAPI session and 
keep it in the rootwrap's client; then each command will be passed to dom0 via 
this same session.

Doug

> 
> But this issue is urgent for XenAPI/XenServer OpenStack. Please the details 
> described in the bug[2]. So I still think the PoC is a better option, unless 
> both oslo and Neutron guys agree it's acceptable to refactor oslo.rootwrap 
> and allow the above dependences introduced to this library.
> 
> [1]https://specs.openstack.org/openstack/oslo-specs/specs/liberty/priv
> sep.html [2] https://bugs.launchpad.net/neutron/+bug/1585510
> 
> Regards,
> Jianghua
> 
> On Tue, Nov 01, 2016 at 12:45:43PM +0100, Ihar Hrachyshka wrote:
> 
> > I suggested in the bug and the PoC review that neutron is not the 
> > right project to solve the issue. Seems like oslo.rootwrap is a 
> > better place to maintain privilege management code for OpenStack. 
> > Ideally, a solution would be found in scope of the library that 
> > would not require any changes per-project.
> 
> With the change of direction from oslo.roowrap to oslo.provsep I doubt that 
> there is scope to land this in oslo.rootwarp.
> 
> Yours Tony.
> 
> -Original Message-
> From: Ihar Hrachyshka [mailto:ihrac...@redhat.com]
> Sent: Tuesday, November 01, 2016 7:46 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] proposal to resolve a rootwrap 
> problem for XenServer
> 
>

Re: [openstack-dev] [neutron] proposal to resolve a rootwrap problem for XenServer

[Jianghua Wang]

Thanks Thierry.
Is Neutron ready to switch oslo.rootwrap to oslo.privsep?
Oslo.privsep seem try to launch a daemon process and set caps for this daemon; 
but for XenAPI, there is no need to spawn the daemon. All of the commands to be 
executed are sent to the common dom0 XAPI daemon (which will invoke a dedicated 
plugin to execute the commands). So I'm confused how to apply the 
privileged.entrypoint function. Could you help to share more details? Thanks 
very much.

Jianghua

-Original Message-
From: Thierry Carrez [mailto:thie...@openstack.org] 
Sent: Wednesday, November 2, 2016 10:06 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] proposal to resolve a rootwrap problem 
for XenServer

Ihar Hrachyshka wrote:
> Tony Breeds  wrote:
> 
>> On Tue, Nov 01, 2016 at 12:45:43PM +0100, Ihar Hrachyshka wrote:
>>
>>> I suggested in the bug and the PoC review that neutron is not the 
>>> right project to solve the issue. Seems like oslo.rootwrap is a 
>>> better place to maintain privilege management code for OpenStack. 
>>> Ideally, a solution would be found in scope of the library that 
>>> would not require any changes per-project.
>>
>> With the change of direction from oslo.roowrap to oslo.provsep I 
>> doubt that there is scope to land this in oslo.rootwarp.
> 
> It may take a while for projects to switch to caps for privilege 
> separation.

oslo.privsep doesn't require projects to switch to caps (just that you rewrite 
the commands you call in Python) and can be done incrementally (while keeping 
rootwrap around for not-yet-migrated stuff)...

> It may be easier to unblock xen folks with a small enhancement in 
> oslo.rootwrap scope and handle transition to oslo.privsep on a 
> separate schedule. I would like to hear from oslo folks on where 
> alternative hypervisors fit in their rootwrap/privsep plans.

Like Tony said at this point new features are added to oslo.privsep rather than 
oslo.rootwrap. In this specific case the most forward-looking solution (and 
also best performance and security) would be to write a Neutron 
@privileged.entrypoint function to call into XenAPI and cache the connection.

https://review.openstack.org/#/c/155631 failed to land in Newton, would be 
great if someone could pick it up (maybe a smaller version to introduce privsep 
first, then migrate commands one by one).

--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap problem for XenServer

[Jianghua Wang]

Ihar and Tony,
 Thanks for the input.
 In order to run command in dom0, it uses XenAPI to create a session which can 
be used to remotely call a plugin - netwrap which is located in dom0. The 
netwrap plugin is executed as root. It will validate the command basing on the 
allowed command list and execute it.
The source code for netwrap is in neutron project: 
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/xenapi/etc/xapi.d/plugins/netwrap

So at least we can see there are two dependences: 
1. it depends on XenAPI which is XenServer specific.
2. it depends on Neutron's plugin netwrap.
Is it acceptable to add such dependences in this common library of 
oslo.rootwrap? 

And most of the code in oslo.rootwrap is to:
1. spawn a daemon process and maintain the connection between the client and 
daemon; 
2. filter commands in the daemon process.
But both can't be re-used for this XenAPI/XenServer case as the daemon process 
is already running in dom0; the command filtering is done in dom0's netwrap 
plugin. In order to hold this in oslo.rootwrap, it requires some refactoring 
work to make it looks reasonable. Is it worthy to do that? Specially by 
considering it has determined to replace oslo.wrap with oslo.provsep?

Maybe it's a good option to cover this dom0 case in oslo.provsep at the 
beginning. But it becomes more complicated. Maybe we can run a daemon process 
in dom0 with the privileges set properly and listening on a dedicated tcp port 
. But that's much different from the initial provsep design [1]. And also it 
makes the mechanism very different from the current XenServer OpenStack which 
is using XenAPI plugin. Anyway, I think we have long to go with a good solution 
to cover it in provsep.

But this issue is urgent for XenAPI/XenServer OpenStack. Please the details 
described in the bug[2]. So I still think the PoC is a better option, unless 
both oslo and Neutron guys agree it's acceptable to refactor oslo.rootwrap and 
allow the above dependences introduced to this library.

[1]https://specs.openstack.org/openstack/oslo-specs/specs/liberty/privsep.html
[2] https://bugs.launchpad.net/neutron/+bug/1585510

Regards,
Jianghua

On Tue, Nov 01, 2016 at 12:45:43PM +0100, Ihar Hrachyshka wrote:

> I suggested in the bug and the PoC review that neutron is not the 
> right project to solve the issue. Seems like oslo.rootwrap is a better 
> place to maintain privilege management code for OpenStack. Ideally, a 
> solution would be found in scope of the library that would not require 
> any changes per-project.

With the change of direction from oslo.roowrap to oslo.provsep I doubt that 
there is scope to land this in oslo.rootwarp.

Yours Tony.

-Original Message-
From: Ihar Hrachyshka [mailto:ihrac...@redhat.com] 
Sent: Tuesday, November 01, 2016 7:46 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] proposal to resolve a rootwrap problem 
for XenServer

Jianghua Wang <jianghua.w...@citrix.com> wrote:

> Hi Neutron guys,
>
> I’m trying to explain a problem with the XenServer rootwrap and give a 
> proposal to resolve it. I need some input on how to proceed with this
> proposal: e.g. if requires a spec? Any concerns need further 
> discussion or clarification?
>
> Problem description:
> As we’ve known, some neutron services need run commands with root 
> privileges and it’s achieved by running commands via the rootwrap. And 
> in order to resolve performance issue, it has been improved to support 
> daemon mode for the rootwrap [1]. Either way has the commands running 
> on the same node/VM which has relative neutron services running on.
>
> But as a type-1 hypervisor, XenServer OpenStack has different behavior.  
> Neutron’s compute agent neutron-openvswitch-agent need run commands in 
> dom0, as the tenants’ interfaces are plugged in an integration OVS 
> which locates in Dom0. Currently the script of 
> https://github.com/openstack/neutron/blob/master/bin/neutron-rootwrap-
> xen-dom0is used as XenServer OpenStack’s rootwrap. This script will 
> create a XenAPI session with dom0 and passes the commands to dom0 for 
> the real execution.
> Each command execution will run this script once. So it has the 
> similar performance issue as the non-daemon mode of rootwrap on other
> hypervisors:  For each command, it has to parse the
> neutron-rootwrap-xen-dom0 script and the rootwrap configure file.  
> Furthermore, this rootwrap script will create a XenAPI for each 
> command execution and XenServer by default will log the XenAPI session 
> creation events. It will cause frequent log file rotation and so other 
> real useful log is lost.
>
> Proposal:
> The os.rootwrap support daemon mode for other hypervisors; but  
> XenServer’s compute agent can’t use that as again it need run com

[openstack-dev] [neutron] proposal to resolve a rootwrap problem for XenServer

[Jianghua Wang]

Hi Neutron guys,

I'm trying to explain a problem with the XenServer rootwrap and give a proposal 
to resolve it. I need some input on how to proceed with this proposal: e.g. if 
requires a spec? Any concerns need further discussion or clarification?

Problem description:
As we've known, some neutron services need run commands with root privileges 
and it's achieved by running commands via the rootwrap. And in order to resolve 
performance issue, it has been improved to support daemon mode for the rootwrap 
[1]. Either way has the commands running on the same node/VM which has relative 
neutron services running on.

But as a type-1 hypervisor, XenServer OpenStack has different behavior. 
Neutron's compute agent neutron-openvswitch-agent need run commands in dom0, as 
the tenants' interfaces are plugged in an integration OVS which locates in 
Dom0. Currently the script of 
https://github.com/openstack/neutron/blob/master/bin/neutron-rootwrap-xen-dom0 
is used as XenServer OpenStack's rootwrap. This script will create a XenAPI 
session with dom0 and passes the commands to dom0 for the real execution. Each 
command execution will run this script once. So it has the similar performance 
issue as the non-daemon mode of rootwrap on other hypervisors:  For each 
command, it has to parse the neutron-rootwrap-xen-dom0 script and the rootwrap 
configure file. Furthermore, this rootwrap script will create a XenAPI for each 
command execution and XenServer by default will log the XenAPI session creation 
events. It will cause frequent log file rotation and so other real useful log 
is lost.

Proposal:
The os.rootwrap support daemon mode for other hypervisors; but XenServer's 
compute agent can't use that as again it need run commands in Dom0. But we can 
refer to that design and implement the daemon mode for XenServer. After 
creating a XenAPI session, Dom0's XAPI will accept the command running requests 
from the session and reply with the running result. So logically we've had a 
daemon in dom0. So we can support daemon mode rootwrap with the following 
design:
1. Develop a daemon client module for XenServer: The agent service will use 
this client module to create a XenAPI session, and keep this session during the 
service's whole life.
2. once need run command on dom0, use the above client to runs commands in dom0.
It should be able to result the issues mentioned above, as the client module 
need import only once for each agent service and only use a single session for 
all commands. The prototype code[3] works well.

Any concern or comments for the above proposal? And how I can proceed with 
solution? We've filed a RFE bug[2] which is in wishlist status. Per 
the neutron policy[4], it seems need neutron-drivers team to evaluate the RFE 
and determine if a spec is required. Could anyone help to evaluate this 
proposal and tell me how I should proceed? And I'm also open and happy for any 
comments. Thanks very much.

[1] 
https://specs.openstack.org/openstack/oslo-specs/specs/juno/rootwrap-daemon-mode.html
[2] https://bugs.launchpad.net/neutron/+bug/1585510
[3]prototype code: https://review.openstack.org/#/c/390931/
[4] http://docs.openstack.org/developer/neutron/policies/blueprints.html

Regards,
Jianghua

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Discussion on to enable "Citrix XenServer CI" to vote openstack/tempest

[Jianghua Wang]

Thanks Masayuki for the comments and thanks Bob for the explanation.

All, 
  Please let's know if there are more comments/ concerns or if we're good to 
proceed for voting. Thanks very much.
Regards,
Jianghua

-Original Message-
From: Bob Ball 
Sent: Tuesday, June 14, 2016 6:23 PM
To: OpenStack Development Mailing List (not for usage questions); Jianghua Wang
Subject: RE: [openstack-dev] [tempest] Discussion on to enable "Citrix 
XenServer CI" to vote openstack/tempest

Hi Masayuki,

We have been running against Tempest and commenting for many months (years?) 
and run in the Rackspace public cloud so have no capacity issues.

Indeed the execution time is longer than other jobs, because we actually have 
to use double nesting (Devstack running on Ubuntu in a VM under XenServer 
running on a VM under XenServer) to run these jobs in the Rackspace cloud.
We are not asking to block Jenkins reporting on these jobs, so taking a little 
longer than Jenkins to report shouldn't be a problem.

We're currently re-processing a backlog of tests from over the weekend, due to 
the broken Tempest change, so for the next 24 hours I expect the reporting 
times to be delayed while we process the backlog.  Looking at previous changes, 
such as https://review.openstack.org/#/c/325243/ or 
https://review.openstack.org/#/c/218355/ you can see that the Citrix XenServer 
CI consistently reports before Jenkins.

Thanks,

Bob

-Original Message-
From: Masayuki Igawa [mailto:masayuki.ig...@gmail.com]
Sent: 14 June 2016 11:07
To: Jianghua Wang <jianghua.w...@citrix.com>
Cc: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [tempest] Discussion on to enable "Citrix 
XenServer CI" to vote openstack/tempest

Hi Jianghua and QA team,

Thank you for bringing this up.

IMO, it's good to make it voting in openstack/tempest. Because it's already a 
voting job in Nova and seems having enough stability.
And we should keep test stability for projects.

My concern is that the resource of "Citrix XenServer CI". Do you have enough 
resource for it?
And it seems like "Citrix XenServer CI" job execution time is the longest one 
in Tempest jobs.

Best regards,
-- Masayuki

On Tue, Jun 14, 2016 at 2:36 PM, Jianghua Wang <jianghua.w...@citrix.com> wrote:
> Added project prefix in the subject and loop in Masayuki and Ghanshyam 
> who know the background as well. Thanks.
>
>
>
> Jianghua
>
>
>
> From: Jianghua Wang
> Sent: Tuesday, June 14, 2016 12:46 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: Jianghua Wang
> Subject: Discussion on to enable "Citrix XenServer CI" to vote 
> openstack/tempest
>
>
>
> Hi all,
>
>Recently the "Citrix XenServer CI" was broken due to a bad 
> commit[1] to openstack/tempest. As the commit was merged on Friday 
> which is vacation at here, it had been in failure for more than three 
> days before we noticed and fixed[2] this problem. As this CI votes for 
> openstack/nova, it had been keeping to vote -1 until disabled voting.
>
>So I suggest we also enable this XenServer CI voting on tempest 
> change to avoid similar cases in the future. We see in this case, the 
> tempest commit didn’t consider the different case for type-1 
> hypervisors, so it broke XenServer test. Actually “Citrix XenServer 
> CI” verified that patch set with failure result but which got ignored 
> due to no voting. So let’s enable the voting to make life easierJ
>
> Currently we have this CI voting for openstack/nova. Per the history 
> experience, it has been a stable CI(more stable than the Jenkins
> check) normally if there is no bad commit breaking it.
>
> Thanks for any comments.
>
>
>
> [1] https://review.openstack.org/#/c/316672
>
> [2] https://review.openstack.org/#/c/328836/
>
>
>
> Regards,
>
> Jianghua
>
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] Discussion on to enable "Citrix XenServer CI" to vote openstack/tempest

[Jianghua Wang]

Added project prefix in the subject and loop in Masayuki and Ghanshyam who know 
the background as well. Thanks.

Jianghua

From: Jianghua Wang
Sent: Tuesday, June 14, 2016 12:46 PM
To: OpenStack Development Mailing List (not for usage questions)
Cc: Jianghua Wang
Subject: Discussion on to enable "Citrix XenServer CI" to vote openstack/tempest

Hi all,
   Recently the "Citrix XenServer CI" was broken due to a bad commit[1] to 
openstack/tempest. As the commit was merged on Friday which is vacation at 
here, it had been in failure for more than three days before we noticed and 
fixed[2] this problem. As this CI votes for openstack/nova, it had been keeping 
to vote -1 until disabled voting.
   So I suggest we also enable this XenServer CI voting on tempest change to 
avoid similar cases in the future. We see in this case, the tempest commit 
didn't consider the different case for type-1 hypervisors, so it broke 
XenServer test. Actually "Citrix XenServer CI" verified that patch set with 
failure result but which got ignored due to no voting. So let's enable the 
voting to make life easier:)
Currently we have this CI voting for openstack/nova. Per the history 
experience, it has been a stable CI(more stable than the Jenkins check) 
normally if there is no bad commit breaking it.
Thanks for any comments.

[1] https://review.openstack.org/#/c/316672
[2] https://review.openstack.org/#/c/328836/

Regards,
Jianghua

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Discussion on to enable "Citrix XenServer CI" to vote openstack/tempest

[Jianghua Wang]

Hi all,
   Recently the "Citrix XenServer CI" was broken due to a bad commit[1] to 
openstack/tempest. As the commit was merged on Friday which is vacation at 
here, it had been in failure for more than three days before we noticed and 
fixed[2] this problem. As this CI votes for openstack/nova, it had been keeping 
to vote -1 until disabled voting.
   So I suggest we also enable this XenServer CI voting on tempest change to 
avoid similar cases in the future. We see in this case, the tempest commit 
didn't consider the different case for type-1 hypervisors, so it broke 
XenServer test. Actually "Citrix XenServer CI" verified that patch set with 
failure result but which got ignored due to no voting. So let's enable the 
voting to make life easier:)
Currently we have this CI voting for openstack/nova. Per the history 
experience, it has been a stable CI(more stable than the Jenkins check) 
normally if there is no bad commit breaking it.
Thanks for any comments.

[1] https://review.openstack.org/#/c/316672
[2] https://review.openstack.org/#/c/328836/

Regards,
Jianghua

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] [xen] [ovs]: how to handle the ports when there are multiple ports existing for a single VM vif

[Jianghua Wang]

Hi,
  The problem to configure both tapx.0 and vifx.0 is that the iface-id was 
thought to be unique for each port and all the ports are indexed with iface-id. 
But in our case, both tapx.0 and vifx.0 share the same iface-id. I'm thinking 
to use the port's name as the identification as the name seems unique on the 
ovs bridge. Could any experts help to confirm if there is any potential issue?
  And another idea I'm thinking of is: the ifac-id is unique for each "active" 
port; so one potential resolution is continue to use iface-id as active ports 
but treat the inactive ports as the subsidiary part to the active port. And add 
function to sync the configuration to inactive ports once any update on the 
active port.
  Any comments are welcome and appreciated.

Thanks,
Jianghua

-Original Message-
Date: Mon, 12 Oct 2015 16:12:23 +0000
From: Jianghua Wang <jianghua.w...@citrix.com>
To: "openstack-dev@lists.openstack.org"
<openstack-dev@lists.openstack.org>
Subject: [openstack-dev] [Neutron] [xen] [ovs]: how to handle the
ports when there are multiple ports existing for a single VM vif
Message-ID:
<382648c81696da498287d6ce037fbf6a0c3...@sinpex01cl02.citrite.net>
Content-Type: text/plain; charset="us-ascii"

Hi guys,
   I'm working on a bug #1268955 which is due to neutron ovs agent/plugin can't 
process the ports correctly when multiple ports existing for a single VM vif. I 
originally identified two potential solutions but one of them requires not 
minor change; and the other one may result in a race condition. So I'm posting 
it at here to seek help. Please let me know if you have any comments or advice. 
Thanks in advance.

Bug description:

When the guest VM is running under HVM mode, neutron doesn't set the vlan tag 
to the proper port. So guest VM lost network communication.

Problem analysis:
When VM is under HVM mode, ovs will create two ports and two interfaces for a 
single vif inside the VM: If the domID is x, one port/interface is named as 
tapx.0
which is qemu-emulated NIC, used when no PV drivers installed; The other one is 
named as vifx.0 which is the xen network frontend NIC, used when VM has PV 
drivers installed. Depending on the PV driver's existing, either port/interface 
may be used. But current ovs agent/plugin use the VM's vif id(iface-id) to 
identify the port. So depending on the ports sequence retrieved from ovs; only 
one port will be processed by neutron. Then the network problem occurs if the 
finally used port is not the same one processed by neutron (e.g. set vlan tag).



Two of my potential solutions:

1.  configure both ports regardless which port will be used finally; so both 
have the same configuration. It should be able to resolve the problem. But the 
existing code uses the iface-id as the key for each port. Both tapx.0 and 
vifx.0 have the same iface-id. With this solution, I have to change the data 
structure to hold both ports and change relative functions; such required 
change spreads at many places. So it will take much more effort by comparing to 
the 2nd choice. And I have a concern if there will be potential issues to 
configure the inactive port although I can't point it out currently.



2.  if there are multiple choices, ovs set the field of "iface-status" as 
active for the one taking effective; and others will be inactive. So the other 
solution is to return the active one only. If there is any switchover happens 
in later phase, treat this port as updated and then it will configure the new 
chosen port accordingly. In this way it will ensure the active port to be 
configured properly. The needed change is very limited. Please see the draft 
patch set for this solution: https://review.openstack.org/#/c/233498/



But the problem is it will introduce a race condition. E.g. if it sets tag on 
tapx.0 firstly; the guest VM get connection via tapx.0; then the PM driver 
loaded, so the active port switch to vifx.0; but depending on the neutron agent 
polling interval, the vifx.0 may not be tagged for a while; then during this 
period the connection is lost.


Could you share your insights? Thanks a lot.

B.R.
Jianghua Wang

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] [xen] [ovs]: how to handle the ports when there are multiple ports existing for a single VM vif

[Jianghua Wang]

Hi guys,
   I'm working on a bug #1268955 which is due to neutron ovs agent/plugin can't 
process the ports correctly when multiple ports existing for a single VM vif. I 
originally identified two potential solutions but one of them requires not 
minor change; and the other one may result in a race condition. So I'm posting 
it at here to seek help. Please let me know if you have any comments or advice. 
Thanks in advance.

Bug description:

When the guest VM is running under HVM mode, neutron doesn't set the vlan tag 
to the proper port. So guest VM lost network communication.

Problem analysis:
When VM is under HVM mode, ovs will create two ports and two interfaces for a 
single vif inside the VM: If the domID is x, one port/interface is named as 
tapx.0
which is qemu-emulated NIC, used when no PV drivers installed; The other one is 
named as vifx.0 which is the xen network frontend NIC, used when VM has PV 
drivers installed. Depending on the PV driver's existing, either port/interface 
may be used. But current ovs agent/plugin use the VM's vif id(iface-id) to 
identify the port. So depending on the ports sequence retrieved from ovs; only 
one port will be processed by neutron. Then the network problem occurs if the 
finally used port is not the same one processed by neutron (e.g. set vlan tag).



Two of my potential solutions:

1.  configure both ports regardless which port will be used finally; so both 
have the same configuration. It should be able to resolve the problem. But the 
existing code uses the iface-id as the key for each port. Both tapx.0 and 
vifx.0 have the same iface-id. With this solution, I have to change the data 
structure to hold both ports and change relative functions; such required 
change spreads at many places. So it will take much more effort by comparing to 
the 2nd choice. And I have a concern if there will be potential issues to 
configure the inactive port although I can't point it out currently.



2.  if there are multiple choices, ovs set the field of "iface-status" as 
active for the one taking effective; and others will be inactive. So the other 
solution is to return the active one only. If there is any switchover happens 
in later phase, treat this port as updated and then it will configure the new 
chosen port accordingly. In this way it will ensure the active port to be 
configured properly. The needed change is very limited. Please see the draft 
patch set for this solution: https://review.openstack.org/#/c/233498/



But the problem is it will introduce a race condition. E.g. if it sets tag on 
tapx.0 firstly; the guest VM get connection via tapx.0; then the PM driver 
loaded, so the active port switch to vifx.0; but depending on the neutron agent 
polling interval, the vifx.0 may not be tagged for a while; then during this 
period the connection is lost.


Could you share your insights? Thanks a lot.

B.R.
Jianghua Wang
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev