Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

On Tue, 1 Dec 2015 at 00:20 Thierry Carrez  wrote:

> Sean Dague wrote:
> > Right, so to me this seems that privsep just needs a NULL mode, and
> > we're done. If oslo.rootrwap was never supported on windows, I don't
> > think privsep really needs to be in a real way.
>
> +1
>

Agreed.  I need to make it safe to import, and then execute any wrapped
functions in-process instead of across a privsep IPC channel.

This is easy enough, except for verifying whether I've caught all the
import issues.  At this point I think my easiest path is to wait until I
have a nova -> os_brick -> privsep change stack, and then go look at the
nova hyper-v test results.

If someone (hyper-v folks?) with a suitable windows development environment
available locally wants to jump in before then, I would certainly
appreciate the assistance.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Davanum Srinivas]

Angus,

I took a stab at it:
https://review.openstack.org/#/c/253070/

Also, can you please request a oslo.privsep release in the
openstack/releases repo for the library with a specific SHA when you
think we are ready?

Thanks,
Dims

On Thu, Dec 3, 2015 at 6:01 PM, Angus Lees  wrote:
> On Tue, 1 Dec 2015 at 00:20 Thierry Carrez  wrote:
>>
>> Sean Dague wrote:
>> > Right, so to me this seems that privsep just needs a NULL mode, and
>> > we're done. If oslo.rootrwap was never supported on windows, I don't
>> > think privsep really needs to be in a real way.
>>
>> +1
>
>
> Agreed.  I need to make it safe to import, and then execute any wrapped
> functions in-process instead of across a privsep IPC channel.
>
> This is easy enough, except for verifying whether I've caught all the import
> issues.  At this point I think my easiest path is to wait until I have a
> nova -> os_brick -> privsep change stack, and then go look at the nova
> hyper-v test results.
>
> If someone (hyper-v folks?) with a suitable windows development environment
> available locally wants to jump in before then, I would certainly appreciate
> the assistance.
>
>  - Gus
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Sean Dague]

On 11/24/2015 06:01 AM, Claudiu Belu wrote:
> Hello,
> 
> Thanks Dims for raising the concern and Angus for reaching out. :)
> 
> Most of the time, python development on Windows is not too far off from
> Linux. But the two systems are quite different, which imply different
> modules (fcntl, pwd, grp modules do not exist in Windows) or different
> implementations of some modules (multiprocessing uses Popen instead of
> os.fork, os module is quite different) or some socket options and
> signals are different in Windows.
> 
> 1.a. As I've said earlier, some modules do not exist in Windows. All, or
> at least most standard modules document the fact that they are strictly
> for Linux. [1][2][3]
> b. At the very least, running the unit tests in a Windows environment
> can at least detect simple problems (e.g. imports). Secondly, there is a
> Hyper-V / Windows CI running on some of the OpenStack projects (nova,
> neutron, networking_hyperv, cinder) that can be checked before merging.
> 
> 2. This is a bit more complicated question. Well, for functions, you
> could have separate modules for Linux specific functions and Windows
> specific functions. This has been done before: [4] As for
> object-oriented implementations, I'd suggest having the system-specific
> calls be done in private methods, which will be overriden by Windows /
> Linux subclasses with their specific implementations. We've done
> something like this before, but solutions were pretty much
> straight-forward; it might not be as simple for oslo_privsep, since it
> is very Linux-specific.
> 
> 3. Typically, the OpenStack services on Hyper-V / Windows are run with
> users that have enough privileges to do their job. For example, the
> nova-compute service is run with a user that has Hyper-V Admin
> privileges and is not necessarily in the "Administrator" user group. We
> haven't used rootwrap in our usecases, it is disabled by default, plus,
> oslo.rootwrap imports pwd, which doesn't exist in Windows.

Right, so to me this seems that privsep just needs a NULL mode, and
we're done. If oslo.rootrwap was never supported on windows, I don't
think privsep really needs to be in a real way.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Thierry Carrez]

Sean Dague wrote:
> Right, so to me this seems that privsep just needs a NULL mode, and
> we're done. If oslo.rootrwap was never supported on windows, I don't
> think privsep really needs to be in a real way.

+1

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Alessandro Pilotti]

Hi Angus,

First thanks for your concern on code portability! It still happens that we 
have to ask to revert patches on Oslo projects due to some Linux specific code 
that we discover only when the actual Oslo modules are used by Nova, Neutron, 
Cinder or other projects. Typically a running Windows CI suddenly turns red on 
every patch and that’s how we get to find out. We’re working on adding many 
more projects under Windows CI tests, so at some point all the relevant Oslo 
ones will be covered and we’ll be able to prevent those situations before the 
code gets merged.

What about preparing some basic integration tests for oslo.privsep that we 
could add to our Windows CI infrastructure? This would give you peace of mind 
during development on Linux without needing to test manually all your patches 
on Windows, knowing that the Windows CI will give an error if the patch 
contains non portable code (or code that doesn’t behave as expected).

Alessandro



From: Angus Lees <g...@inodes.org<mailto:g...@inodes.org>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Thursday 26 November 2015 at 01:56
To: Claudiu Belu 
<cb...@cloudbasesolutions.com<mailto:cb...@cloudbasesolutions.com>>, "OpenStack 
Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

So I spent a day yesterday trying to get to the point where I could just run a 
no-change "tox" successfully on windows.  Unfortunately I gave up when I 
realised I still had several days of downloading+building things ahead of me 
and clearly I was doing it the hard way :(

Could you point me to the "dev environment how-to" doc for hyper-v work?  I'm 
thinking of something like 
http://docs.openstack.org/developer/nova/development.environment.html#setup 
with simple cut+paste instructions for the totally-windows-clueless like me ;)  
Or perhaps a pre-prepared disk image, if the Microsoft license allows such a 
thing.  In particular, the details on https://wiki.openstack.org/wiki/Hyper-V 
look out of date (Grizzly, and several 
docs.openstack.org<http://docs.openstack.org> links are broken), and the links 
to things like 
https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/ look like 
deployment rather than development environments (?).

In particular, I think(?) I should be careful *not* to install too much of a 
cygwin-style environment, since I think(?) that might no longer be 
representative of the environment in which hyper-v is expected to operate.  If 
I'm wrong, and cygwin/msys is ok, then it looks like there are several 
free-software-on-windows "distributions" that would make life a whole lot 
simpler (by frankly being less like Windows).  Some guidance from people who 
understand the issues here would be appreciated.

 - Gus

On Tue, 24 Nov 2015 at 22:01 Claudiu Belu 
<cb...@cloudbasesolutions.com<mailto:cb...@cloudbasesolutions.com>> wrote:
Hello,

Thanks Dims for raising the concern and Angus for reaching out. :)

Most of the time, python development on Windows is not too far off from Linux. 
But the two systems are quite different, which imply different modules (fcntl, 
pwd, grp modules do not exist in Windows) or different implementations of some 
modules (multiprocessing uses Popen instead of os.fork, os module is quite 
different) or some socket options and signals are different in Windows.

1.a. As I've said earlier, some modules do not exist in Windows. All, or at 
least most standard modules document the fact that they are strictly for Linux. 
[1][2][3]
b. At the very least, running the unit tests in a Windows environment can at 
least detect simple problems (e.g. imports). Secondly, there is a Hyper-V / 
Windows CI running on some of the OpenStack projects (nova, neutron, 
networking_hyperv, cinder) that can be checked before merging.

2. This is a bit more complicated question. Well, for functions, you could have 
separate modules for Linux specific functions and Windows specific functions. 
This has been done before: [4] As for object-oriented implementations, I'd 
suggest having the system-specific calls be done in private methods, which will 
be overriden by Windows / Linux subclasses with their specific implementations. 
We've done something like this before, but solutions were pretty much 
straight-forward; it might not be as simple for oslo_privsep, since it is very 
Linux-specific.

3. Typically, the OpenStack services on Hyper-V / Windows are run with users 
that have enough privileges to do their job. For example, the nova-compute 
service is run with a user that has Hyper-V Admin privileges and is not 
necessarily in the "Administrator" use

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

So I spent a day yesterday trying to get to the point where I could just
run a no-change "tox" successfully on windows.  Unfortunately I gave up
when I realised I still had several days of downloading+building things
ahead of me and clearly I was doing it the hard way :(

Could you point me to the "dev environment how-to" doc for hyper-v work?
I'm thinking of something like
http://docs.openstack.org/developer/nova/development.environment.html#setup
with
simple cut+paste instructions for the totally-windows-clueless like me ;)
 Or perhaps a pre-prepared disk image, if the Microsoft license allows such
a thing.  In particular, the details on
https://wiki.openstack.org/wiki/Hyper-V look out of date (Grizzly, and
several docs.openstack.org links are broken), and the links to things like
https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/ look
like deployment rather than development environments (?).

In particular, I think(?) I should be careful *not* to install too much of
a cygwin-style environment, since I think(?) that might no longer be
representative of the environment in which hyper-v is expected to operate.
If I'm wrong, and cygwin/msys is ok, then it looks like there are several
free-software-on-windows "distributions" that would make life a whole lot
simpler (by frankly being less like Windows).  Some guidance from people
who understand the issues here would be appreciated.

 - Gus

On Tue, 24 Nov 2015 at 22:01 Claudiu Belu 
wrote:

> Hello,
>
> Thanks Dims for raising the concern and Angus for reaching out. :)
>
> Most of the time, python development on Windows is not too far off from
> Linux. But the two systems are quite different, which imply different
> modules (fcntl, pwd, grp modules do not exist in Windows) or different
> implementations of some modules (multiprocessing uses Popen instead of
> os.fork, os module is quite different) or some socket options and signals
> are different in Windows.
>
> 1.a. As I've said earlier, some modules do not exist in Windows. All, or
> at least most standard modules document the fact that they are strictly for
> Linux. [1][2][3]
> b. At the very least, running the unit tests in a Windows environment can
> at least detect simple problems (e.g. imports). Secondly, there is a
> Hyper-V / Windows CI running on some of the OpenStack projects (nova,
> neutron, networking_hyperv, cinder) that can be checked before merging.
>
> 2. This is a bit more complicated question. Well, for functions, you could
> have separate modules for Linux specific functions and Windows specific
> functions. This has been done before: [4] As for object-oriented
> implementations, I'd suggest having the system-specific calls be done in
> private methods, which will be overriden by Windows / Linux subclasses with
> their specific implementations. We've done something like this before, but
> solutions were pretty much straight-forward; it might not be as simple for
> oslo_privsep, since it is very Linux-specific.
>
> 3. Typically, the OpenStack services on Hyper-V / Windows are run with
> users that have enough privileges to do their job. For example, the
> nova-compute service is run with a user that has Hyper-V Admin privileges
> and is not necessarily in the "Administrator" user group. We haven't used
> rootwrap in our usecases, it is disabled by default, plus, oslo.rootwrap
> imports pwd, which doesn't exist in Windows.
>
> [1] https://docs.python.org/2/library/fcntl.html
> [2] https://docs.python.org/2/library/pwd.html
> [3] https://docs.python.org/2/library/grp.html
> [4]
> https://github.com/openstack/neutron/blob/master/neutron/agent/common/utils.py
> [5]
> https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/wrapper.py#L19
>
> If you have any further questions, feel free to ask. :)
>
> Best regards,
> Claudiu Belu
>
>
> --
> *From:* Angus Lees [g...@inodes.org]
> *Sent:* Tuesday, November 24, 2015 6:18 AM
> *To:* OpenStack Development Mailing List (not for usage questions);
> Claudiu Belu
> *Subject:* [hyper-v] oslo.privsep vs Windows
>
> Dims has just raised[1] the excellent concern that oslo.privsep will need
> to at least survive on Windows, because hyper-v.  I have no real experience
> coding on windows (I wrote a windows C program once, but I only ever ran it
> under wine ;) and certainly none within an OpenStack/python context:
>
> 1) How can I test whatever I'm working on to see if I have mistakenly
> introduced something Linux-specific?  Surely this is a challenge common
> across every project in the nova/oslo/hyper-v stack.
>
> 2) What predicate should I use to guard the inevitable Linux-specific or
> Windows-specific code branches?
>
> and I guess:
> 3) What does a typical OpenStack/hyper-v install even look like? Do we run
> rootwrap with some sudo-like-thing, or just run everything as the superuser?
> What _should_ oslo.privsep do for this environment?
>
> [1] 

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Alessandro Pilotti]

Angus,

"I'm afraid this has to be easy for me or I'm just not going to be able to 
sustain the effort required :-/ )"

What a tragedy, I’m s sorry that life is so terribly bitter and requires 
some effort to support people with views different from yours! :-)

Just kidding, we’re here to help, but we obviously won’t do your work. Given 
that Oslo code needs to be portable, I was just trying to give you some 
alternative that doesn’t require you to do useless manual work during 
development.

1) Unit tests are not the answer, as hopefully they mock out most significant 
underlying OS features. Sure, you will catch the basic issues, like importing a 
module that doesn’t exist on Windows, but you cannot rely on them as a proof of 
portability.
2) Integration testing on the other side provide by definition a proof of 
reliability on the target system (in the limits of the tests coverage at least) 
and CI testing needs eventually to be there on Windows as well for 
oslo.. This is obvious not a way to develop, but a 
“guard”, against issues. In other words, regardless of your opinion on Windows, 
the Windows CI needs to be there, why not adding it now for this project?

Getting to development practices, when designing features, hopefully 
portability gets evaluated _before_ starting coding, so you should not need too 
many manual runs on every platform before being confident enough to send a 
patch to Gerrit and wait for CI results. And for that you don’t need tox or 
anything fancy, just a Python environment and some very basic OS skills.

When in doubt, an option is to drop by on #openstack-hyper-v and ask. But take 
care, you might find open minded people, make sure you're at ease with this. ;-)

Alessandro


From: Angus Lees <g...@inodes.org<mailto:g...@inodes.org>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Thursday 26 November 2015 at 03:23
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>, 
Claudiu Belu <cb...@cloudbasesolutions.com<mailto:cb...@cloudbasesolutions.com>>
Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

Thanks for the suggestion, and having a CI bot running on oslo.privsep would be 
a good thing once the basic code works on Windows - I would have expected it to 
be already running on all oslo projects (that the hyper-v hypervisor does/might 
depend on) tbh.

But that's a clumsy way to actually develop something.  I _know_ privsep won't 
work currently on windows (imports pwd/grp for starters), and having to add 
print statements + submit + push-to-gerrit + wait for a CI bot + dig through CI 
bot logs + repeat is a pretty terrible workflow ;-)

(I already have very little empathy for anyone choosing to run Windows just to 
provide yet-another-x86-hypervisor when far easier alternatives are available, 
and I won't donate a disproportionate amount of my time to a well-funded 
company who has historically actively undermined the intellectual commons when 
_I_ have far more rewarding alternatives available.  I'm afraid this has to be 
easy for me or I'm just not going to be able to sustain the effort required :-/ 
)


 - Gus

On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti 
<apilo...@cloudbasesolutions.com<mailto:apilo...@cloudbasesolutions.com>> wrote:
Hi Angus,

First thanks for your concern on code portability! It still happens that we 
have to ask to revert patches on Oslo projects due to some Linux specific code 
that we discover only when the actual Oslo modules are used by Nova, Neutron, 
Cinder or other projects. Typically a running Windows CI suddenly turns red on 
every patch and that’s how we get to find out. We’re working on adding many 
more projects under Windows CI tests, so at some point all the relevant Oslo 
ones will be covered and we’ll be able to prevent those situations before the 
code gets merged.

What about preparing some basic integration tests for oslo.privsep that we 
could add to our Windows CI infrastructure? This would give you peace of mind 
during development on Linux without needing to test manually all your patches 
on Windows, knowing that the Windows CI will give an error if the patch 
contains non portable code (or code that doesn’t behave as expected).

Alessandro



From: Angus Lees <g...@inodes.org<mailto:g...@inodes.org>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Thursday 26 November 2015 at 01:56
To: Claudiu Belu 
<cb...@cloudbasesolutions.com<mailto:cb...@cloudbasesolutions.com>>, "OpenStack 
Development Mailing List (not for usage questions)" 
<openstack-dev@list

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Tony Breeds]

On Thu, Nov 26, 2015 at 02:11:01AM +, Alessandro Pilotti wrote:
> Angus,
> 
> "I'm afraid this has to be easy for me or I'm just not going to be able to 
> sustain the effort required :-/ )"
> 
> What a tragedy, I’m s sorry that life is so terribly bitter and requires 
> some effort to support people with views different from yours! :-)



CI has it's place, unit tests have their place, local development has it's 
place.  No on suggested otherwise.

Can you develop in CI, sure is it a great experience? In my opinion no.
 
Anyway, I think this is heading off track and I'd like to try and steer it back
to the problem at hand.

There are 2 simple questions:
 1. How does one setup an python development environment that is as close as
possible to the environment that hyper-v runs in.
 2. Are the details captured in an easy to locate document for developers?

Yours Tony.


pgpKQIwtFpgui.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

So seriously - how do you set up a dev environment for hyper-v?
It doesn't have to be written up all nice and pretty, just a 30s outline in
an email would be really useful :-/

 - Gus

On Thu, 26 Nov 2015 at 13:13 Alessandro Pilotti <
apilo...@cloudbasesolutions.com> wrote:

> Angus,
>
> "I'm afraid this has to be easy for me or I'm just not going to be able to
> sustain the effort required :-/ )"
>
> What a tragedy, I’m s sorry that life is so terribly bitter and
> requires some effort to support people with views different from yours! :-)
>
> Just kidding, we’re here to help, but we obviously won’t do your work.
> Given that Oslo code needs to be portable, I was just trying to give you
> some alternative that doesn’t require you to do useless manual work during
> development.
>
> 1) Unit tests are not the answer, as hopefully they mock out most
> significant underlying OS features. Sure, you will catch the basic issues,
> like importing a module that doesn’t exist on Windows, but you cannot rely
> on them as a proof of portability.
> 2) Integration testing on the other side provide by definition a proof of
> reliability on the target system (in the limits of the tests coverage at
> least) and CI testing needs eventually to be there on Windows as well for
> oslo.. This is obvious not a way to develop, but a
> “guard”, against issues. In other words, regardless of your opinion on
> Windows, the Windows CI needs to be there, why not adding it now for this
> project?
>
> Getting to development practices, when designing features, hopefully
> portability gets evaluated _before_ starting coding, so you should not need
> too many manual runs on every platform before being confident enough to
> send a patch to Gerrit and wait for CI results. And for that you don’t need
> tox or anything fancy, just a Python environment and some very basic OS
> skills.
>
> When in doubt, an option is to drop by on #openstack-hyper-v and ask. But
> take care, you might find open minded people, make sure you're at ease with
> this. ;-)
>
> Alessandro
>
>
> From: Angus Lees <g...@inodes.org>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Thursday 26 November 2015 at 03:23
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>, Claudiu Belu <
> cb...@cloudbasesolutions.com>
>
> Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows
>
> Thanks for the suggestion, and having a CI bot running on oslo.privsep
> would be a good thing once the basic code works on Windows - I would have
> expected it to be already running on all oslo projects (that the hyper-v
> hypervisor does/might depend on) tbh.
>
> But that's a clumsy way to actually develop something.  I _know_ privsep
> won't work currently on windows (imports pwd/grp for starters), and having
> to add print statements + submit + push-to-gerrit + wait for a CI bot + dig
> through CI bot logs + repeat is a pretty terrible workflow ;-)
>
> (I already have very little empathy for anyone choosing to run Windows
> just to provide yet-another-x86-hypervisor when far easier alternatives are
> available, and I won't donate a disproportionate amount of my time to a
> well-funded company who has historically actively undermined the
> intellectual commons when _I_ have far more rewarding alternatives
> available.  I'm afraid this has to be easy for me or I'm just not going to
> be able to sustain the effort required :-/ )
>
>
>  - Gus
>
> On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti <
> apilo...@cloudbasesolutions.com> wrote:
>
>> Hi Angus,
>>
>> First thanks for your concern on code portability! It still happens that
>> we have to ask to revert patches on Oslo projects due to some Linux
>> specific code that we discover only when the actual Oslo modules are used
>> by Nova, Neutron, Cinder or other projects. Typically a running Windows CI
>> suddenly turns red on every patch and that’s how we get to find out. We’re
>> working on adding many more projects under Windows CI tests, so at some
>> point all the relevant Oslo ones will be covered and we’ll be able to
>> prevent those situations before the code gets merged.
>>
>> What about preparing some basic integration tests for oslo.privsep that
>> we could add to our Windows CI infrastructure? This would give you peace of
>> mind during development on Linux without needing to test manually all your
>> patches on Windows, knowing that the Windows CI will give an error if the
>> patch contains non portable code (or code that doesn’t

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Robert Collins]

On 26 November 2015 at 15:54, Alessandro Pilotti
 wrote:
> Basic Python development does not really differ too much on Windows compared
> to Linux.
>
> Let’s start with the Python environment. I’d recommend to install a 2.7
> (x86) one and a 3.4 (x64) one:
>
> https://www.python.org/ftp/python/2.7.10/python-2.7.10.msi
> https://www.python.org/ftp/python/3.4.3/python-3.4.3.amd64.msi
>
> Download also git for Windows: https://git-scm.com/download/win
> Git works in the same way, so nothing particular to add. Just take care of
> the line endings. I usually keep them all in LF, avoiding CRLF, others let
> git do the conversion.
>
> When done, open a PowerShell or command prompt and set your PATH and
> PYTHONPATH e.g.:
>
> $ENV:PATH += ";C:\Python27;C:\Python27\Scripts"
> $ENV:PYTHONPATH = "."
>
> From here is pretty standard Python, e.g.:
>
> pip install “blah>=1.0.0"
> pip install –r requirements.txt
>
> python yourmodule.py
>
> For running tests you can use for example nose since tox / testr don’t
> really work:
>
> pip install mock
> pip install nose
> nosetests .

i haven't seen any bug reports on testrepository vis-a-vis windows;
please do file them, otherwise I'll presume it works.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

Thanks for the suggestion, and having a CI bot running on oslo.privsep
would be a good thing once the basic code works on Windows - I would have
expected it to be already running on all oslo projects (that the hyper-v
hypervisor does/might depend on) tbh.

But that's a clumsy way to actually develop something.  I _know_ privsep
won't work currently on windows (imports pwd/grp for starters), and having
to add print statements + submit + push-to-gerrit + wait for a CI bot + dig
through CI bot logs + repeat is a pretty terrible workflow ;-)

(I already have very little empathy for anyone choosing to run Windows just
to provide yet-another-x86-hypervisor when far easier alternatives are
available, and I won't donate a disproportionate amount of my time to a
well-funded company who has historically actively undermined the
intellectual commons when _I_ have far more rewarding alternatives
available.  I'm afraid this has to be easy for me or I'm just not going to
be able to sustain the effort required :-/ )

 - Gus

On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti <
apilo...@cloudbasesolutions.com> wrote:

> Hi Angus,
>
> First thanks for your concern on code portability! It still happens that
> we have to ask to revert patches on Oslo projects due to some Linux
> specific code that we discover only when the actual Oslo modules are used
> by Nova, Neutron, Cinder or other projects. Typically a running Windows CI
> suddenly turns red on every patch and that’s how we get to find out. We’re
> working on adding many more projects under Windows CI tests, so at some
> point all the relevant Oslo ones will be covered and we’ll be able to
> prevent those situations before the code gets merged.
>
> What about preparing some basic integration tests for oslo.privsep that we
> could add to our Windows CI infrastructure? This would give you peace of
> mind during development on Linux without needing to test manually all your
> patches on Windows, knowing that the Windows CI will give an error if the
> patch contains non portable code (or code that doesn’t behave as expected).
>
> Alessandro
>
>
>
> From: Angus Lees <g...@inodes.org>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Thursday 26 November 2015 at 01:56
> To: Claudiu Belu <cb...@cloudbasesolutions.com>, "OpenStack Development
> Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org
> >
> Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows
>
> So I spent a day yesterday trying to get to the point where I could just
> run a no-change "tox" successfully on windows.  Unfortunately I gave up
> when I realised I still had several days of downloading+building things
> ahead of me and clearly I was doing it the hard way :(
>
> Could you point me to the "dev environment how-to" doc for hyper-v work?
> I'm thinking of something like
> http://docs.openstack.org/developer/nova/development.environment.html#setup 
> with
> simple cut+paste instructions for the totally-windows-clueless like me ;)
>  Or perhaps a pre-prepared disk image, if the Microsoft license allows such
> a thing.  In particular, the details on
> https://wiki.openstack.org/wiki/Hyper-V look out of date (Grizzly, and
> several docs.openstack.org links are broken), and the links to things
> like https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/ 
> look
> like deployment rather than development environments (?).
>
> In particular, I think(?) I should be careful *not* to install too much of
> a cygwin-style environment, since I think(?) that might no longer be
> representative of the environment in which hyper-v is expected to operate.
> If I'm wrong, and cygwin/msys is ok, then it looks like there are several
> free-software-on-windows "distributions" that would make life a whole lot
> simpler (by frankly being less like Windows).  Some guidance from people
> who understand the issues here would be appreciated.
>
>  - Gus
>
> On Tue, 24 Nov 2015 at 22:01 Claudiu Belu <cb...@cloudbasesolutions.com>
> wrote:
>
>> Hello,
>>
>> Thanks Dims for raising the concern and Angus for reaching out. :)
>>
>> Most of the time, python development on Windows is not too far off from
>> Linux. But the two systems are quite different, which imply different
>> modules (fcntl, pwd, grp modules do not exist in Windows) or different
>> implementations of some modules (multiprocessing uses Popen instead of
>> os.fork, os module is quite different) or some socket options and signals
>> are different in Windows.
>>
>> 1.a. As I've said earlier, some modules do not exist in Windows

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Alessandro Pilotti]

Basic Python development does not really differ too much on Windows compared to 
Linux.

Let’s start with the Python environment. I’d recommend to install a 2.7 (x86) 
one and a 3.4 (x64) one:

https://www.python.org/ftp/python/2.7.10/python-2.7.10.msi
https://www.python.org/ftp/python/3.4.3/python-3.4.3.amd64.msi

Download also git for Windows: https://git-scm.com/download/win
Git works in the same way, so nothing particular to add. Just take care of the 
line endings. I usually keep them all in LF, avoiding CRLF, others let git do 
the conversion.

When done, open a PowerShell or command prompt and set your PATH and PYTHONPATH 
e.g.:

$ENV:PATH += ";C:\Python27;C:\Python27\Scripts"
$ENV:PYTHONPATH = "."

From here is pretty standard Python, e.g.:

pip install “blah>=1.0.0"
pip install –r requirements.txt

python yourmodule.py

For running tests you can use for example nose since tox / testr don’t really 
work:

pip install mock
pip install nose
nosetests .

Note: you can also use virtualenvs, exactly like on Linux.

You might incur in occasional native modules that require a C development 
environment. That’s usually more complicated as it requires also dependencies 
and needs C/C++ skills and either Mingw / mingw64 or Visual Studio 2008 (Python 
2.7) or 2010 (Python 3.4).
For those few modules there are typically binary wheels available, either on 
Pypi or provided by 3rd parties. If not, just ask, we compile them from source 
and have wheels in the CI.

What you can optionally do, to get a Python environment pre-loaded with all the 
OpenStack dependencies is to install the OpenStack Hyper-V Compute MSI (see 
http://cloudbase.it) and use the Python env that comes with it.

If you need a basic working OpenStack, just get a standard Devstack running in 
a Linux host or VM plus the above MSI for Hyper-V configured to point to the 
DevStack node. There’s plenty of examples on our blog for similar cases  
(http://cloudbase.it/blog).

It terms of editors it’s a matter of personal taste, but you can find e.g. 
Sublime, Visual Studio Code, Notepad++ or Atom as very good and lightweight 
options.

Let me know if this works for you and if you have any other questions.

Alessandro



From: Angus Lees <g...@inodes.org<mailto:g...@inodes.org>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Date: Thursday 26 November 2015 at 04:22
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>, 
Claudiu Belu <cb...@cloudbasesolutions.com<mailto:cb...@cloudbasesolutions.com>>
Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

So seriously - how do you set up a dev environment for hyper-v?
It doesn't have to be written up all nice and pretty, just a 30s outline in an 
email would be really useful :-/


 - Gus

On Thu, 26 Nov 2015 at 13:13 Alessandro Pilotti 
<apilo...@cloudbasesolutions.com<mailto:apilo...@cloudbasesolutions.com>> wrote:
Angus,

"I'm afraid this has to be easy for me or I'm just not going to be able to 
sustain the effort required :-/ )"

What a tragedy, I’m s sorry that life is so terribly bitter and requires 
some effort to support people with views different from yours! :-)

Just kidding, we’re here to help, but we obviously won’t do your work. Given 
that Oslo code needs to be portable, I was just trying to give you some 
alternative that doesn’t require you to do useless manual work during 
development.

1) Unit tests are not the answer, as hopefully they mock out most significant 
underlying OS features. Sure, you will catch the basic issues, like importing a 
module that doesn’t exist on Windows, but you cannot rely on them as a proof of 
portability.
2) Integration testing on the other side provide by definition a proof of 
reliability on the target system (in the limits of the tests coverage at least) 
and CI testing needs eventually to be there on Windows as well for 
oslo.. This is obvious not a way to develop, but a 
“guard”, against issues. In other words, regardless of your opinion on Windows, 
the Windows CI needs to be there, why not adding it now for this project?

Getting to development practices, when designing features, hopefully 
portability gets evaluated _before_ starting coding, so you should not need too 
many manual runs on every platform before being confident enough to send a 
patch to Gerrit and wait for CI results. And for that you don’t need tox or 
anything fancy, just a Python environment and some very basic OS skills.

When in doubt, an option is to drop by on #openstack-hyper-v and ask. But take 
care, you might find open minded people, make sure you're at ease with this. ;-)

Alessandro


From: Angus Lees <g...@ino

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Angus Lees]

On Thu, 26 Nov 2015 at 14:19 Robert Collins 
wrote:

> On 26 November 2015 at 15:54, Alessandro Pilotti
>  wrote:
> > When done, open a PowerShell or command prompt and set your PATH and
> > PYTHONPATH e.g.:
> >
> > $ENV:PATH += ";C:\Python27;C:\Python27\Scripts"
> > $ENV:PYTHONPATH = "."
>

Wow, this syntax is completely foreign to me.  You have no idea how much of
a newbie I feel, thanks ;)


> > For running tests you can use for example nose since tox / testr don’t
> > really work:
> >
> > pip install mock
> > pip install nose
> > nosetests .
>
> i haven't seen any bug reports on testrepository vis-a-vis windows;
> please do file them, otherwise I'll presume it works.
>

>From my brief day or two of experience, I think the tox executable itself
works fine - it's more the typical tox.ini:
- "python3.5.exe" doesn't typically exist - the regular python 3.5 windows
install seems to just install an unversioned python.exe and breaks the
usual tox basepython values.  Creating a "python3.5" symlink doesn't seem
to be a thing ;)
- Just about every tox.ini we have is full of unixisms - see for example
nova/tox.ini's liberal use of bash scripts and find commands.
- Virtualenv installs things differently on windows.  In particular the
venv has all the executables in $venv\Scripts\ rather than $venv/bin/.

I quickly gave up on using tox, and was running the virtualenv and "pip
install -r requirements.txt" steps by hand.  The wheels (pun unintended)
well and truly fell off once I hit my first C library dependencies (for me,
yaml and netifaces pulled in by oslo.utils), and I was looking at having to
set up a full C toolchain from scratch too.

I didn't get far enough to try testr.


Alessandro: I think the main blocker I've hit so far (other than "how do I
cut+paste" ;) is installing the non-python library dependencies.  Is there
something I can do to steal your pre-built wheels (and dlls?), or use a
pre-built distro like cygwin/msys2[1], or perhaps we should build a conda
repository for openstack/windows like I did once for openstack/linux[2], or
...?

[1] I didn't get an answer to my earlier question, I presume installing
cygwin _is_ going to invalidate my hyper-v-suitability tests, right?  I
presume other "unix-like distros" like msys2 (mingw-based afaict) are
similarly bad?

[2] https://conda.anaconda.org/gus  - It was a (mostly positive) experiment
in trying something other than pip.  I haven't maintained the repo, but it
didn't take that long to package the transitive dependencies of nova.

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

[Claudiu Belu]

Hello,

Thanks Dims for raising the concern and Angus for reaching out. :)

Most of the time, python development on Windows is not too far off from Linux. 
But the two systems are quite different, which imply different modules (fcntl, 
pwd, grp modules do not exist in Windows) or different implementations of some 
modules (multiprocessing uses Popen instead of os.fork, os module is quite 
different) or some socket options and signals are different in Windows.

1.a. As I've said earlier, some modules do not exist in Windows. All, or at 
least most standard modules document the fact that they are strictly for Linux. 
[1][2][3]
b. At the very least, running the unit tests in a Windows environment can at 
least detect simple problems (e.g. imports). Secondly, there is a Hyper-V / 
Windows CI running on some of the OpenStack projects (nova, neutron, 
networking_hyperv, cinder) that can be checked before merging.

2. This is a bit more complicated question. Well, for functions, you could have 
separate modules for Linux specific functions and Windows specific functions. 
This has been done before: [4] As for object-oriented implementations, I'd 
suggest having the system-specific calls be done in private methods, which will 
be overriden by Windows / Linux subclasses with their specific implementations. 
We've done something like this before, but solutions were pretty much 
straight-forward; it might not be as simple for oslo_privsep, since it is very 
Linux-specific.

3. Typically, the OpenStack services on Hyper-V / Windows are run with users 
that have enough privileges to do their job. For example, the nova-compute 
service is run with a user that has Hyper-V Admin privileges and is not 
necessarily in the "Administrator" user group. We haven't used rootwrap in our 
usecases, it is disabled by default, plus, oslo.rootwrap imports pwd, which 
doesn't exist in Windows.

[1] https://docs.python.org/2/library/fcntl.html
[2] https://docs.python.org/2/library/pwd.html
[3] https://docs.python.org/2/library/grp.html
[4] 
https://github.com/openstack/neutron/blob/master/neutron/agent/common/utils.py
[5] 
https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/wrapper.py#L19

If you have any further questions, feel free to ask. :)

Best regards,
Claudiu Belu



From: Angus Lees [g...@inodes.org]
Sent: Tuesday, November 24, 2015 6:18 AM
To: OpenStack Development Mailing List (not for usage questions); Claudiu Belu
Subject: [hyper-v] oslo.privsep vs Windows

Dims has just raised[1] the excellent concern that oslo.privsep will need to at 
least survive on Windows, because hyper-v.  I have no real experience coding on 
windows (I wrote a windows C program once, but I only ever ran it under wine ;) 
and certainly none within an OpenStack/python context:

1) How can I test whatever I'm working on to see if I have mistakenly 
introduced something Linux-specific?  Surely this is a challenge common across 
every project in the nova/oslo/hyper-v stack.

2) What predicate should I use to guard the inevitable Linux-specific or 
Windows-specific code branches?

and I guess:
3) What does a typical OpenStack/hyper-v install even look like? Do we run 
rootwrap with some sudo-like-thing, or just run everything as the superuser?
What _should_ oslo.privsep do for this environment?

[1] 
https://review.openstack.org/#/c/244984

 - Gus
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev