Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap problem for XenServer

[Doug Hellmann]

Excerpts from Jianghua Wang's message of 2016-11-02 15:52:22 +:
> Thanks Doug. Please see my response inline starts with .
> 
> Jianghua
> 
> -Original Message-
> From: Doug Hellmann [mailto:d...@doughellmann.com] 
> Sent: Wednesday, November 2, 2016 9:31 PM
> To: openstack-dev 
> Subject: Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap 
> problem for XenServer
> 
> Excerpts from Jianghua Wang's message of 2016-11-02 04:14:48 +:
> > Ihar and Tony,
> >  Thanks for the input.
> >  In order to run command in dom0, it uses XenAPI to create a session which 
> > can be used to remotely call a plugin - netwrap which is located in dom0. 
> > The netwrap plugin is executed as root. It will validate the command basing 
> > on the allowed command list and execute it.
> > The source code for netwrap is in neutron project: 
> > https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/d
> > rivers/openvswitch/agent/xenapi/etc/xapi.d/plugins/netwrap
> > 
> > So at least we can see there are two dependences: 
> > 1. it depends on XenAPI which is XenServer specific.
> > 2. it depends on Neutron's plugin netwrap.
> > Is it acceptable to add such dependences in this common library of 
> > oslo.rootwrap? 
> 
> Why would they need to be dependencies of oslo.rootwrap? They are 
> dependencies of the driver, not the base library, right?
> 
>  With a second thought, I think we can pass the plugin name 
> netwrap as a parameter to the rootwrap; so maybe not a dependence. But if we 
> host the XenAPI session creation in oslo.rootwrap, I think we should import 
> XenAPI in oslo.rootwrap. Then it is a dependence in the base library, isn't 
> it?

I don't think we want to build Xen-specific features or dependencies
into any of the Oslo libraries unless we absolutely can't avoid it.

> 
> > And most of the code in oslo.rootwrap is to:
> > 1. spawn a daemon process and maintain the connection between the 
> > client and daemon; 2. filter commands in the daemon process.
> > But both can't be re-used for this XenAPI/XenServer case as the daemon 
> > process is already running in dom0; the command filtering is done in dom0's 
> > netwrap plugin. In order to hold this in oslo.rootwrap, it requires some 
> > refactoring work to make it looks reasonable. Is it worthy to do that? 
> > Specially by considering it has determined to replace oslo.wrap with 
> > oslo.provsep?
> > 
> > Maybe it's a good option to cover this dom0 case in oslo.provsep at the 
> > beginning. But it becomes more complicated. Maybe we can run a daemon 
> > process in dom0 with the privileges set properly and listening on a 
> > dedicated tcp port . But that's much different from the initial provsep 
> > design [1]. And also it makes the mechanism very different from the current 
> > XenServer OpenStack which is using XenAPI plugin. Anyway, I think we have 
> > long to go with a good solution to cover it in provsep.
> 
> What sort of refactoring do you have in mind for privsep? I could see 
> something analogous to the preexec_fn argument to subprocess.Popen() to let 
> the XenServer driver ensure that its privileged process is running in dom0.
> 
> Sorry, I don't have a clear idea on refactorying in privsep now. 
> It seems privsep attempts to create a daemon process and set caps for this 
> daemon. But for XenServer/XenAPI, the daemon and caps in daemon seems 
> useless. As it sends all commands to the a common XAPI daemon running in 
> dom0. No additional daemon is needed. TBH I don't know how to apply caps at 
> here. But to make it to resolve the current issue, what I'm thinking is to 
> create a XenAPI session and keep it in the rootwrap's client; then each 
> command will be passed to dom0 via this same session.

OK. I think Thierry's question in the other thread (about why the
XenAPI calls have to be made from a privileged process at all) is
useful for thinking about any API changes. Let's keep the discussion
over there to avoid drift or confusion.

Doug

> 
> Doug
> 
> > 
> > But this issue is urgent for XenAPI/XenServer OpenStack. Please the details 
> > described in the bug[2]. So I still think the PoC is a better option, 
> > unless both oslo and Neutron guys agree it's acceptable to refactor 
> > oslo.rootwrap and allow the above dependences introduced to this library.
> > 
> > [1]https://specs.openstack.org/openstack/oslo-specs/specs/liberty/priv
> > sep.html [2] https://bugs.launchpad.net/neutron/+bug/1585510
> > 
> > Regards,
> > Jianghua
> > 
> > On Tue, Nov 01, 2016 at 12:45:43PM +0100, Ihar Hrachyshka wrote:
> > 
> > > I suggested in the bug and the PoC review that neutron is not the 
> > > right project to solve the issue. Seems like oslo.rootwrap is a 
> > > better place to maintain privilege management code for OpenStack. 
> > > Ideally, a solution would be found in scope of the library that 
> > > would not require any changes per-project.
> > 
> > With the change of 

Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap problem for XenServer

[Jianghua Wang]

Thanks Doug. Please see my response inline starts with .

Jianghua

-Original Message-
From: Doug Hellmann [mailto:d...@doughellmann.com] 
Sent: Wednesday, November 2, 2016 9:31 PM
To: openstack-dev 
Subject: Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap 
problem for XenServer

Excerpts from Jianghua Wang's message of 2016-11-02 04:14:48 +:
> Ihar and Tony,
>  Thanks for the input.
>  In order to run command in dom0, it uses XenAPI to create a session which 
> can be used to remotely call a plugin - netwrap which is located in dom0. The 
> netwrap plugin is executed as root. It will validate the command basing on 
> the allowed command list and execute it.
> The source code for netwrap is in neutron project: 
> https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/d
> rivers/openvswitch/agent/xenapi/etc/xapi.d/plugins/netwrap
> 
> So at least we can see there are two dependences: 
> 1. it depends on XenAPI which is XenServer specific.
> 2. it depends on Neutron's plugin netwrap.
> Is it acceptable to add such dependences in this common library of 
> oslo.rootwrap? 

Why would they need to be dependencies of oslo.rootwrap? They are dependencies 
of the driver, not the base library, right?

 With a second thought, I think we can pass the plugin name netwrap 
as a parameter to the rootwrap; so maybe not a dependence. But if we host the 
XenAPI session creation in oslo.rootwrap, I think we should import XenAPI in 
oslo.rootwrap. Then it is a dependence in the base library, isn't it?

> And most of the code in oslo.rootwrap is to:
> 1. spawn a daemon process and maintain the connection between the 
> client and daemon; 2. filter commands in the daemon process.
> But both can't be re-used for this XenAPI/XenServer case as the daemon 
> process is already running in dom0; the command filtering is done in dom0's 
> netwrap plugin. In order to hold this in oslo.rootwrap, it requires some 
> refactoring work to make it looks reasonable. Is it worthy to do that? 
> Specially by considering it has determined to replace oslo.wrap with 
> oslo.provsep?
> 
> Maybe it's a good option to cover this dom0 case in oslo.provsep at the 
> beginning. But it becomes more complicated. Maybe we can run a daemon process 
> in dom0 with the privileges set properly and listening on a dedicated tcp 
> port . But that's much different from the initial provsep design [1]. And 
> also it makes the mechanism very different from the current XenServer 
> OpenStack which is using XenAPI plugin. Anyway, I think we have long to go 
> with a good solution to cover it in provsep.

What sort of refactoring do you have in mind for privsep? I could see something 
analogous to the preexec_fn argument to subprocess.Popen() to let the XenServer 
driver ensure that its privileged process is running in dom0.

Sorry, I don't have a clear idea on refactorying in privsep now. It 
seems privsep attempts to create a daemon process and set caps for this daemon. 
But for XenServer/XenAPI, the daemon and caps in daemon seems useless. As it 
sends all commands to the a common XAPI daemon running in dom0. No additional 
daemon is needed. TBH I don't know how to apply caps at here. But to make it to 
resolve the current issue, what I'm thinking is to create a XenAPI session and 
keep it in the rootwrap's client; then each command will be passed to dom0 via 
this same session.

Doug

> 
> But this issue is urgent for XenAPI/XenServer OpenStack. Please the details 
> described in the bug[2]. So I still think the PoC is a better option, unless 
> both oslo and Neutron guys agree it's acceptable to refactor oslo.rootwrap 
> and allow the above dependences introduced to this library.
> 
> [1]https://specs.openstack.org/openstack/oslo-specs/specs/liberty/priv
> sep.html [2] https://bugs.launchpad.net/neutron/+bug/1585510
> 
> Regards,
> Jianghua
> 
> On Tue, Nov 01, 2016 at 12:45:43PM +0100, Ihar Hrachyshka wrote:
> 
> > I suggested in the bug and the PoC review that neutron is not the 
> > right project to solve the issue. Seems like oslo.rootwrap is a 
> > better place to maintain privilege management code for OpenStack. 
> > Ideally, a solution would be found in scope of the library that 
> > would not require any changes per-project.
> 
> With the change of direction from oslo.roowrap to oslo.provsep I doubt that 
> there is scope to land this in oslo.rootwarp.
> 
> Yours Tony.
> 
> -Original Message-
> From: Ihar Hrachyshka [mailto:ihrac...@redhat.com]
> Sent: Tuesday, November 01, 2016 7:46 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] proposal to resolve a rootwrap 
> problem for XenServer
> 
> Jianghua Wang  wrote:
> 
> > Hi Neutron guys,
> >
> > I’m trying to explain a problem with the XenServer rootwrap and give 
> > a proposal to resolve it. I need some input on how to proceed 

Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap problem for XenServer

[Doug Hellmann]

Excerpts from Jianghua Wang's message of 2016-11-02 04:14:48 +:
> Ihar and Tony,
>  Thanks for the input.
>  In order to run command in dom0, it uses XenAPI to create a session which 
> can be used to remotely call a plugin - netwrap which is located in dom0. The 
> netwrap plugin is executed as root. It will validate the command basing on 
> the allowed command list and execute it.
> The source code for netwrap is in neutron project: 
> https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/xenapi/etc/xapi.d/plugins/netwrap
> 
> So at least we can see there are two dependences: 
> 1. it depends on XenAPI which is XenServer specific.
> 2. it depends on Neutron's plugin netwrap.
> Is it acceptable to add such dependences in this common library of 
> oslo.rootwrap? 

Why would they need to be dependencies of oslo.rootwrap? They are
dependencies of the driver, not the base library, right?

> And most of the code in oslo.rootwrap is to:
> 1. spawn a daemon process and maintain the connection between the client and 
> daemon; 
> 2. filter commands in the daemon process.
> But both can't be re-used for this XenAPI/XenServer case as the daemon 
> process is already running in dom0; the command filtering is done in dom0's 
> netwrap plugin. In order to hold this in oslo.rootwrap, it requires some 
> refactoring work to make it looks reasonable. Is it worthy to do that? 
> Specially by considering it has determined to replace oslo.wrap with 
> oslo.provsep?
> 
> Maybe it's a good option to cover this dom0 case in oslo.provsep at the 
> beginning. But it becomes more complicated. Maybe we can run a daemon process 
> in dom0 with the privileges set properly and listening on a dedicated tcp 
> port . But that's much different from the initial provsep design [1]. And 
> also it makes the mechanism very different from the current XenServer 
> OpenStack which is using XenAPI plugin. Anyway, I think we have long to go 
> with a good solution to cover it in provsep.

What sort of refactoring do you have in mind for privsep? I could see
something analogous to the preexec_fn argument to subprocess.Popen() to
let the XenServer driver ensure that its privileged process is running
in dom0.

Doug

> 
> But this issue is urgent for XenAPI/XenServer OpenStack. Please the details 
> described in the bug[2]. So I still think the PoC is a better option, unless 
> both oslo and Neutron guys agree it's acceptable to refactor oslo.rootwrap 
> and allow the above dependences introduced to this library.
> 
> [1]https://specs.openstack.org/openstack/oslo-specs/specs/liberty/privsep.html
> [2] https://bugs.launchpad.net/neutron/+bug/1585510
> 
> Regards,
> Jianghua
> 
> On Tue, Nov 01, 2016 at 12:45:43PM +0100, Ihar Hrachyshka wrote:
> 
> > I suggested in the bug and the PoC review that neutron is not the 
> > right project to solve the issue. Seems like oslo.rootwrap is a better 
> > place to maintain privilege management code for OpenStack. Ideally, a 
> > solution would be found in scope of the library that would not require 
> > any changes per-project.
> 
> With the change of direction from oslo.roowrap to oslo.provsep I doubt that 
> there is scope to land this in oslo.rootwarp.
> 
> Yours Tony.
> 
> -Original Message-
> From: Ihar Hrachyshka [mailto:ihrac...@redhat.com] 
> Sent: Tuesday, November 01, 2016 7:46 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] proposal to resolve a rootwrap problem 
> for XenServer
> 
> Jianghua Wang  wrote:
> 
> > Hi Neutron guys,
> >
> > I’m trying to explain a problem with the XenServer rootwrap and give a 
> > proposal to resolve it. I need some input on how to proceed with this
> > proposal: e.g. if requires a spec? Any concerns need further 
> > discussion or clarification?
> >
> > Problem description:
> > As we’ve known, some neutron services need run commands with root 
> > privileges and it’s achieved by running commands via the rootwrap. And 
> > in order to resolve performance issue, it has been improved to support 
> > daemon mode for the rootwrap [1]. Either way has the commands running 
> > on the same node/VM which has relative neutron services running on.
> >
> > But as a type-1 hypervisor, XenServer OpenStack has different behavior.  
> > Neutron’s compute agent neutron-openvswitch-agent need run commands in 
> > dom0, as the tenants’ interfaces are plugged in an integration OVS 
> > which locates in Dom0. Currently the script of 
> > https://github.com/openstack/neutron/blob/master/bin/neutron-rootwrap-
> > xen-dom0is used as XenServer OpenStack’s rootwrap. This script will 
> > create a XenAPI session with dom0 and passes the commands to dom0 for 
> > the real execution.
> > Each command execution will run this script once. So it has the 
> > similar performance issue as the non-daemon mode of rootwrap on other
> > hypervisors:  For each 

Re: [openstack-dev] [neutron][oslo] proposal to resolve a rootwrap problem for XenServer

[Jianghua Wang]

Ihar and Tony,
 Thanks for the input.
 In order to run command in dom0, it uses XenAPI to create a session which can 
be used to remotely call a plugin - netwrap which is located in dom0. The 
netwrap plugin is executed as root. It will validate the command basing on the 
allowed command list and execute it.
The source code for netwrap is in neutron project: 
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/xenapi/etc/xapi.d/plugins/netwrap

So at least we can see there are two dependences: 
1. it depends on XenAPI which is XenServer specific.
2. it depends on Neutron's plugin netwrap.
Is it acceptable to add such dependences in this common library of 
oslo.rootwrap? 

And most of the code in oslo.rootwrap is to:
1. spawn a daemon process and maintain the connection between the client and 
daemon; 
2. filter commands in the daemon process.
But both can't be re-used for this XenAPI/XenServer case as the daemon process 
is already running in dom0; the command filtering is done in dom0's netwrap 
plugin. In order to hold this in oslo.rootwrap, it requires some refactoring 
work to make it looks reasonable. Is it worthy to do that? Specially by 
considering it has determined to replace oslo.wrap with oslo.provsep?

Maybe it's a good option to cover this dom0 case in oslo.provsep at the 
beginning. But it becomes more complicated. Maybe we can run a daemon process 
in dom0 with the privileges set properly and listening on a dedicated tcp port 
. But that's much different from the initial provsep design [1]. And also it 
makes the mechanism very different from the current XenServer OpenStack which 
is using XenAPI plugin. Anyway, I think we have long to go with a good solution 
to cover it in provsep.

But this issue is urgent for XenAPI/XenServer OpenStack. Please the details 
described in the bug[2]. So I still think the PoC is a better option, unless 
both oslo and Neutron guys agree it's acceptable to refactor oslo.rootwrap and 
allow the above dependences introduced to this library.

[1]https://specs.openstack.org/openstack/oslo-specs/specs/liberty/privsep.html
[2] https://bugs.launchpad.net/neutron/+bug/1585510

Regards,
Jianghua

On Tue, Nov 01, 2016 at 12:45:43PM +0100, Ihar Hrachyshka wrote:

> I suggested in the bug and the PoC review that neutron is not the 
> right project to solve the issue. Seems like oslo.rootwrap is a better 
> place to maintain privilege management code for OpenStack. Ideally, a 
> solution would be found in scope of the library that would not require 
> any changes per-project.

With the change of direction from oslo.roowrap to oslo.provsep I doubt that 
there is scope to land this in oslo.rootwarp.

Yours Tony.

-Original Message-
From: Ihar Hrachyshka [mailto:ihrac...@redhat.com] 
Sent: Tuesday, November 01, 2016 7:46 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] proposal to resolve a rootwrap problem 
for XenServer

Jianghua Wang  wrote:

> Hi Neutron guys,
>
> I’m trying to explain a problem with the XenServer rootwrap and give a 
> proposal to resolve it. I need some input on how to proceed with this
> proposal: e.g. if requires a spec? Any concerns need further 
> discussion or clarification?
>
> Problem description:
> As we’ve known, some neutron services need run commands with root 
> privileges and it’s achieved by running commands via the rootwrap. And 
> in order to resolve performance issue, it has been improved to support 
> daemon mode for the rootwrap [1]. Either way has the commands running 
> on the same node/VM which has relative neutron services running on.
>
> But as a type-1 hypervisor, XenServer OpenStack has different behavior.  
> Neutron’s compute agent neutron-openvswitch-agent need run commands in 
> dom0, as the tenants’ interfaces are plugged in an integration OVS 
> which locates in Dom0. Currently the script of 
> https://github.com/openstack/neutron/blob/master/bin/neutron-rootwrap-
> xen-dom0is used as XenServer OpenStack’s rootwrap. This script will 
> create a XenAPI session with dom0 and passes the commands to dom0 for 
> the real execution.
> Each command execution will run this script once. So it has the 
> similar performance issue as the non-daemon mode of rootwrap on other
> hypervisors:  For each command, it has to parse the
> neutron-rootwrap-xen-dom0 script and the rootwrap configure file.  
> Furthermore, this rootwrap script will create a XenAPI for each 
> command execution and XenServer by default will log the XenAPI session 
> creation events. It will cause frequent log file rotation and so other 
> real useful log is lost.
>
> Proposal:
> The os.rootwrap support daemon mode for other hypervisors; but  
> XenServer’s compute agent can’t use that as again it need run commands in  
> Dom0. But we can refer to that design and implement the daemon mode for  
> XenServer. After creating a XenAPI