Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-04 Thread Yuriy Taraday
On Wed, Sep 3, 2014 at 7:24 PM, Doug Hellmann d...@doughellmann.com wrote:

 On Sep 3, 2014, at 5:27 AM, Yuriy Taraday yorik@gmail.com wrote:

 On Tue, Sep 2, 2014 at 11:17 PM, Clark Boylan cboy...@sapwetik.org
 wrote:

 It has been pointed out to me that one case where it won't be so easy is
 oslo.messaging and its use of eventlet under python2. Messaging will
 almost certainly need python 2 and python 3 wheels to be separate. I
 think we should continue to use universal wheels where possible and only
 build python2 and python3 wheels in the special cases where necessary.


 We can make eventlet an optional dependency of oslo.messaging (through
 setuptools' extras). In fact I don't quite understand the need for eventlet
 as direct dependency there since we can just write code that uses threading
 library and it'll get monkeypatched if consumer app wants to use eventlet.


 There is code in the messaging library that makes calls directly into
 eventlet now, IIRC. It sounds like that could be changed, but that’s
 something to consider for a future version.


Yes, I hope to see unified threading/eventlet executor there
(futures-based, I guess) some day.

The last time I looked at setuptools extras they were a documented but
 unimplemented specification. Has that changed?


According to docs [1] it works in pip (and has been working in setuptools
for ages), and according to bug [2], it has been working for couple years.

[1] http://pip.readthedocs.org/en/latest/reference/pip_install.html#examples
(#6)
[2] https://github.com/pypa/pip/issues/7

-- 

Kind regards, Yuriy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-04 Thread Yuriy Taraday
On Wed, Sep 3, 2014 at 8:21 PM, Doug Hellmann d...@doughellmann.com wrote:

  On Sep 3, 2014, at 11:57 AM, Clark Boylan cboy...@sapwetik.org wrote:
  On Wed, Sep 3, 2014, at 08:22 AM, Doug Hellmann wrote:
 
  On Sep 2, 2014, at 3:17 PM, Clark Boylan cboy...@sapwetik.org wrote:
  The setup.cfg classifiers should be able to do that for us, though PBR
  may need updating? We will also need to learn to upload potentially 1
 
  How do you see that working? We want all of the Oslo libraries to,
  eventually, support both python 2 and 3. How would we use the
 classifiers
  to tell when to build a universal wheel and when to build separate
  wheels?
 
  The classifiers provide info on the versions of python we support. By
  default we can build python2 wheel if only 2 is supported, build python3
  wheel if only 3 is supported, build a universal wheel if both are
  supported. Then we can add a setup.cfg flag to override the universal
  wheel default to build both a python2 and python3 wheel instead. Dstufft
  and mordred should probably comment on this idea before we implement
  anything.

 OK. I’m not aware of any python-3-only projects, and the flag to override
 the universal wheel is the piece I was missing. I think there’s already a
 setuptools flag related to whether or not we should build universal wheels,
 isn’t there?


I think we should rely on wheel.universal flag from setup.cfg if it's
there. If it's set, we should always build universal wheels. If it's not
set, we should look in specifiers and build wheels for Python versions that
are mentioned there.

-- 

Kind regards, Yuriy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-04 Thread Yuriy Taraday
On Thu, Sep 4, 2014 at 4:47 AM, Jeremy Stanley fu...@yuggoth.org wrote:

 On 2014-09-03 13:27:55 +0400 (+0400), Yuriy Taraday wrote:
 [...]
  May be we should drop 3.3 already?

 It's in progress. Search review.openstack.org for open changes in
 all projects with the topic py34. Shortly I'll also have some
 infra config changes up to switch python33 jobs out for python34,
 ready to drop once the j-3 milestone has been tagged and is finally
 behind us.


Great! Looking forward to purging python 3.3 from my system.

-- 

Kind regards, Yuriy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-04 Thread Doug Hellmann

On Sep 4, 2014, at 3:38 AM, Yuriy Taraday yorik@gmail.com wrote:

 On Wed, Sep 3, 2014 at 7:24 PM, Doug Hellmann d...@doughellmann.com wrote:
 On Sep 3, 2014, at 5:27 AM, Yuriy Taraday yorik@gmail.com wrote:
 On Tue, Sep 2, 2014 at 11:17 PM, Clark Boylan cboy...@sapwetik.org wrote:
 It has been pointed out to me that one case where it won't be so easy is
 oslo.messaging and its use of eventlet under python2. Messaging will
 almost certainly need python 2 and python 3 wheels to be separate. I
 think we should continue to use universal wheels where possible and only
 build python2 and python3 wheels in the special cases where necessary.
 
 We can make eventlet an optional dependency of oslo.messaging (through 
 setuptools' extras). In fact I don't quite understand the need for eventlet 
 as direct dependency there since we can just write code that uses threading 
 library and it'll get monkeypatched if consumer app wants to use eventlet.
 
 There is code in the messaging library that makes calls directly into 
 eventlet now, IIRC. It sounds like that could be changed, but that’s 
 something to consider for a future version.
 
 Yes, I hope to see unified threading/eventlet executor there (futures-based, 
 I guess) some day.
 
 The last time I looked at setuptools extras they were a documented but 
 unimplemented specification. Has that changed?
 
 According to docs [1] it works in pip (and has been working in setuptools for 
 ages), and according to bug [2], it has been working for couple years.

Wow, apparently it has been a while since I looked at it. :-)

We should look into whether it would make sense to add optional dependencies 
for some of our drivers. I don’t know how the distros interpret the 
requirements.txt files and if they can filter out optional items or if it would 
end up confusing their tool chain.

Doug

 
 [1] http://pip.readthedocs.org/en/latest/reference/pip_install.html#examples 
 (#6)
 [2] https://github.com/pypa/pip/issues/7
 
 -- 
 
 Kind regards, Yuriy.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-03 Thread Yuriy Taraday
On Tue, Sep 2, 2014 at 11:17 PM, Clark Boylan cboy...@sapwetik.org wrote:

 On Tue, Sep 2, 2014, at 11:30 AM, Yuriy Taraday wrote:
  Hello.
 
  Currently for alpha releases of oslo libraries we generate either
  universal
  or Python 2.x-only wheels. This presents a problem: we can't adopt alpha
  releases in projects where Python 3.x is supported and verified in the
  gate. I've ran into this in change request [1] generated after
  global-requirements change [2]. There we have oslotest library that can't
  be built as a universal wheel because of different requirements (mox vs
  mox3 as I understand is the main difference). Because of that py33 job in
  [1] failed and we can't bump oslotest version in requirements.
 
  I propose to change infra scripts that generate and upload wheels to
  create
  py3 wheels as well as py2 wheels for projects that support Python 3.x (we
  can use setup.cfg classifiers to find that out) but don't support
  universal
  wheels. What do you think about that?
 
  [1] https://review.openstack.org/117940
  [2] https://review.openstack.org/115643
 
  --
 
  Kind regards, Yuriy.
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 We may find that we will need to have py3k wheels in addition to the
 existing wheels at some point, but I don't think this use case requires
 it. If oslo.test needs to support python2 and python3 it should use mox3
 in both cases which claims to support python2.6, 2.7 and 3.2. Then you
 can ship a universal wheel. This should solve the immediate problem.


Yes, I think, it's the way to go with oslotest specifically. Created a
change request for this: https://review.openstack.org/118551

It has been pointed out to me that one case where it won't be so easy is
 oslo.messaging and its use of eventlet under python2. Messaging will
 almost certainly need python 2 and python 3 wheels to be separate. I
 think we should continue to use universal wheels where possible and only
 build python2 and python3 wheels in the special cases where necessary.


We can make eventlet an optional dependency of oslo.messaging (through
setuptools' extras). In fact I don't quite understand the need for eventlet
as direct dependency there since we can just write code that uses threading
library and it'll get monkeypatched if consumer app wants to use eventlet.

The setup.cfg classifiers should be able to do that for us, though PBR
 may need updating?


I don't think so - it loads all classifiers from setup.cfg, they should be
available through some distutils machinery.

We will also need to learn to upload potentially 1
 wheel in our wheel jobs. That bit is likely straight foward. The last
 thing that we need to make sure we do is that we have some testing in
 place for the special wheels. We currently have the requirements
 integration test which runs under python2 checking that we can actually
 install all the things together. This ends up exercising our wheels and
 checking that they actually work. We don't have a python3 equivalent for
 that job. It may be better to work out some explicit checking of the
 wheels we produce that applies to both versions of python. I am not
 quite sure how we should approach that yet.


I guess we can just repeat that check with Python 3.x. If I see it right,
all we need is to repeat loop in pbr/tools/integration.sh with different
Python version. The problem might occur that we'll be running this test
with Python 3.4 that is default on trusty but all our unittests jobs run on
3.3 instead. May be we should drop 3.3 already?

-- 

Kind regards, Yuriy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-03 Thread Doug Hellmann

On Sep 2, 2014, at 3:17 PM, Clark Boylan cboy...@sapwetik.org wrote:

 On Tue, Sep 2, 2014, at 11:30 AM, Yuriy Taraday wrote:
 Hello.
 
 Currently for alpha releases of oslo libraries we generate either
 universal
 or Python 2.x-only wheels. This presents a problem: we can't adopt alpha
 releases in projects where Python 3.x is supported and verified in the
 gate. I've ran into this in change request [1] generated after
 global-requirements change [2]. There we have oslotest library that can't
 be built as a universal wheel because of different requirements (mox vs
 mox3 as I understand is the main difference). Because of that py33 job in
 [1] failed and we can't bump oslotest version in requirements.
 
 I propose to change infra scripts that generate and upload wheels to
 create
 py3 wheels as well as py2 wheels for projects that support Python 3.x (we
 can use setup.cfg classifiers to find that out) but don't support
 universal
 wheels. What do you think about that?
 
 [1] https://review.openstack.org/117940
 [2] https://review.openstack.org/115643
 
 -- 
 
 Kind regards, Yuriy.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 We may find that we will need to have py3k wheels in addition to the
 existing wheels at some point, but I don't think this use case requires
 it. If oslo.test needs to support python2 and python3 it should use mox3
 in both cases which claims to support python2.6, 2.7 and 3.2. Then you
 can ship a universal wheel. This should solve the immediate problem.

That sounds like a good solution to that specific issue. It may also require 
changes in the application test suites, but those changes can be made as we 
move them to use oslotest.

 
 It has been pointed out to me that one case where it won't be so easy is
 oslo.messaging and its use of eventlet under python2. Messaging will
 almost certainly need python 2 and python 3 wheels to be separate. I
 think we should continue to use universal wheels where possible and only
 build python2 and python3 wheels in the special cases where necessary.
 
 The setup.cfg classifiers should be able to do that for us, though PBR
 may need updating? We will also need to learn to upload potentially 1

How do you see that working? We want all of the Oslo libraries to, eventually, 
support both python 2 and 3. How would we use the classifiers to tell when to 
build a universal wheel and when to build separate wheels?

 wheel in our wheel jobs. That bit is likely straight foward. The last
 thing that we need to make sure we do is that we have some testing in
 place for the special wheels. We currently have the requirements
 integration test which runs under python2 checking that we can actually
 install all the things together. This ends up exercising our wheels and
 checking that they actually work. We don't have a python3 equivalent for

We only know the wheels can be installed. We don’t actually have a test that 
installs our code and runs it any more (devstack uses “develop” mode which 
bypasses some of the installation steps, as we found while fixing the recent 
neutron/pbr issue with a missing config file in their packaging instructions).

 that job. It may be better to work out some explicit checking of the
 wheels we produce that applies to both versions of python. I am not
 quite sure how we should approach that yet.

To fix the asymmetric gating we have between pbr and everything else, Robert 
suggested setting up some sort of job to install pbr and then build and install 
a package for the project being tested. We already, as you point out, have a 
job that does this for all of the projects to test changes to pbr itself. Maybe 
we can run the same test under python 2 and 3 as part of the same job?

Doug

 
 Clark
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-03 Thread Doug Hellmann

On Sep 3, 2014, at 5:27 AM, Yuriy Taraday yorik@gmail.com wrote:

 On Tue, Sep 2, 2014 at 11:17 PM, Clark Boylan cboy...@sapwetik.org wrote:
 On Tue, Sep 2, 2014, at 11:30 AM, Yuriy Taraday wrote:
  Hello.
 
  Currently for alpha releases of oslo libraries we generate either
  universal
  or Python 2.x-only wheels. This presents a problem: we can't adopt alpha
  releases in projects where Python 3.x is supported and verified in the
  gate. I've ran into this in change request [1] generated after
  global-requirements change [2]. There we have oslotest library that can't
  be built as a universal wheel because of different requirements (mox vs
  mox3 as I understand is the main difference). Because of that py33 job in
  [1] failed and we can't bump oslotest version in requirements.
 
  I propose to change infra scripts that generate and upload wheels to
  create
  py3 wheels as well as py2 wheels for projects that support Python 3.x (we
  can use setup.cfg classifiers to find that out) but don't support
  universal
  wheels. What do you think about that?
 
  [1] https://review.openstack.org/117940
  [2] https://review.openstack.org/115643
 
  --
 
  Kind regards, Yuriy.
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 We may find that we will need to have py3k wheels in addition to the
 existing wheels at some point, but I don't think this use case requires
 it. If oslo.test needs to support python2 and python3 it should use mox3
 in both cases which claims to support python2.6, 2.7 and 3.2. Then you
 can ship a universal wheel. This should solve the immediate problem.
 
 Yes, I think, it's the way to go with oslotest specifically. Created a change 
 request for this: https://review.openstack.org/118551
 
 It has been pointed out to me that one case where it won't be so easy is
 oslo.messaging and its use of eventlet under python2. Messaging will
 almost certainly need python 2 and python 3 wheels to be separate. I
 think we should continue to use universal wheels where possible and only
 build python2 and python3 wheels in the special cases where necessary.
 
 We can make eventlet an optional dependency of oslo.messaging (through 
 setuptools' extras). In fact I don't quite understand the need for eventlet 
 as direct dependency there since we can just write code that uses threading 
 library and it'll get monkeypatched if consumer app wants to use eventlet.

There is code in the messaging library that makes calls directly into eventlet 
now, IIRC. It sounds like that could be changed, but that’s something to 
consider for a future version.

The last time I looked at setuptools extras they were a documented but 
unimplemented specification. Has that changed?

 
 The setup.cfg classifiers should be able to do that for us, though PBR
 may need updating?
 
 I don't think so - it loads all classifiers from setup.cfg, they should be 
 available through some distutils machinery.
 
 We will also need to learn to upload potentially 1
 wheel in our wheel jobs. That bit is likely straight foward. The last
 thing that we need to make sure we do is that we have some testing in
 place for the special wheels. We currently have the requirements
 integration test which runs under python2 checking that we can actually
 install all the things together. This ends up exercising our wheels and
 checking that they actually work. We don't have a python3 equivalent for
 that job. It may be better to work out some explicit checking of the
 wheels we produce that applies to both versions of python. I am not
 quite sure how we should approach that yet.
 
 I guess we can just repeat that check with Python 3.x. If I see it right, all 
 we need is to repeat loop in pbr/tools/integration.sh with different Python 
 version. The problem might occur that we'll be running this test with Python 
 3.4 that is default on trusty but all our unittests jobs run on 3.3 instead. 
 May be we should drop 3.3 already?
 
 -- 
 
 Kind regards, Yuriy.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-03 Thread Clark Boylan
On Wed, Sep 3, 2014, at 08:22 AM, Doug Hellmann wrote:
 
 On Sep 2, 2014, at 3:17 PM, Clark Boylan cboy...@sapwetik.org wrote:
 
  On Tue, Sep 2, 2014, at 11:30 AM, Yuriy Taraday wrote:
  Hello.
  
  Currently for alpha releases of oslo libraries we generate either
  universal
  or Python 2.x-only wheels. This presents a problem: we can't adopt alpha
  releases in projects where Python 3.x is supported and verified in the
  gate. I've ran into this in change request [1] generated after
  global-requirements change [2]. There we have oslotest library that can't
  be built as a universal wheel because of different requirements (mox vs
  mox3 as I understand is the main difference). Because of that py33 job in
  [1] failed and we can't bump oslotest version in requirements.
  
  I propose to change infra scripts that generate and upload wheels to
  create
  py3 wheels as well as py2 wheels for projects that support Python 3.x (we
  can use setup.cfg classifiers to find that out) but don't support
  universal
  wheels. What do you think about that?
  
  [1] https://review.openstack.org/117940
  [2] https://review.openstack.org/115643
  
  -- 
  
  Kind regards, Yuriy.
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  
  We may find that we will need to have py3k wheels in addition to the
  existing wheels at some point, but I don't think this use case requires
  it. If oslo.test needs to support python2 and python3 it should use mox3
  in both cases which claims to support python2.6, 2.7 and 3.2. Then you
  can ship a universal wheel. This should solve the immediate problem.
 
 That sounds like a good solution to that specific issue. It may also
 require changes in the application test suites, but those changes can be
 made as we move them to use oslotest.
 
  
  It has been pointed out to me that one case where it won't be so easy is
  oslo.messaging and its use of eventlet under python2. Messaging will
  almost certainly need python 2 and python 3 wheels to be separate. I
  think we should continue to use universal wheels where possible and only
  build python2 and python3 wheels in the special cases where necessary.
  
  The setup.cfg classifiers should be able to do that for us, though PBR
  may need updating? We will also need to learn to upload potentially 1
 
 How do you see that working? We want all of the Oslo libraries to,
 eventually, support both python 2 and 3. How would we use the classifiers
 to tell when to build a universal wheel and when to build separate
 wheels?
 
The classifiers provide info on the versions of python we support. By
default we can build python2 wheel if only 2 is supported, build python3
wheel if only 3 is supported, build a universal wheel if both are
supported. Then we can add a setup.cfg flag to override the universal
wheel default to build both a python2 and python3 wheel instead. Dstufft
and mordred should probably comment on this idea before we implement
anything.

  wheel in our wheel jobs. That bit is likely straight foward. The last
  thing that we need to make sure we do is that we have some testing in
  place for the special wheels. We currently have the requirements
  integration test which runs under python2 checking that we can actually
  install all the things together. This ends up exercising our wheels and
  checking that they actually work. We don't have a python3 equivalent for
 
 We only know the wheels can be installed. We don’t actually have a test
 that installs our code and runs it any more (devstack uses “develop” mode
 which bypasses some of the installation steps, as we found while fixing
 the recent neutron/pbr issue with a missing config file in their
 packaging instructions).
 
Yup, this would be a second level of testing that we should consider as
well. The post install can it do stuff test. We moved our unittesting
away from using sdist installs in preference for setup.py develop
equivalent which means that our unittests no longer cover some of this
stuff for us. Sorry if I hijacked this thread into a how do we test
release artifacts? thread.

  that job. It may be better to work out some explicit checking of the
  wheels we produce that applies to both versions of python. I am not
  quite sure how we should approach that yet.
 
 To fix the asymmetric gating we have between pbr and everything else,
 Robert suggested setting up some sort of job to install pbr and then
 build and install a package for the project being tested. We already, as
 you point out, have a job that does this for all of the projects to test
 changes to pbr itself. Maybe we can run the same test under python 2 and
 3 as part of the same job?
 
And to get that extra level of tested we can run the unittests in that
package install. We won't be able to run everything against python3
because of missing support (well we can test installs but 

Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-03 Thread Doug Hellmann

On Sep 3, 2014, at 11:57 AM, Clark Boylan cboy...@sapwetik.org wrote:

 On Wed, Sep 3, 2014, at 08:22 AM, Doug Hellmann wrote:
 
 On Sep 2, 2014, at 3:17 PM, Clark Boylan cboy...@sapwetik.org wrote:
 
 On Tue, Sep 2, 2014, at 11:30 AM, Yuriy Taraday wrote:
 Hello.
 
 Currently for alpha releases of oslo libraries we generate either
 universal
 or Python 2.x-only wheels. This presents a problem: we can't adopt alpha
 releases in projects where Python 3.x is supported and verified in the
 gate. I've ran into this in change request [1] generated after
 global-requirements change [2]. There we have oslotest library that can't
 be built as a universal wheel because of different requirements (mox vs
 mox3 as I understand is the main difference). Because of that py33 job in
 [1] failed and we can't bump oslotest version in requirements.
 
 I propose to change infra scripts that generate and upload wheels to
 create
 py3 wheels as well as py2 wheels for projects that support Python 3.x (we
 can use setup.cfg classifiers to find that out) but don't support
 universal
 wheels. What do you think about that?
 
 [1] https://review.openstack.org/117940
 [2] https://review.openstack.org/115643
 
 -- 
 
 Kind regards, Yuriy.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 We may find that we will need to have py3k wheels in addition to the
 existing wheels at some point, but I don't think this use case requires
 it. If oslo.test needs to support python2 and python3 it should use mox3
 in both cases which claims to support python2.6, 2.7 and 3.2. Then you
 can ship a universal wheel. This should solve the immediate problem.
 
 That sounds like a good solution to that specific issue. It may also
 require changes in the application test suites, but those changes can be
 made as we move them to use oslotest.
 
 
 It has been pointed out to me that one case where it won't be so easy is
 oslo.messaging and its use of eventlet under python2. Messaging will
 almost certainly need python 2 and python 3 wheels to be separate. I
 think we should continue to use universal wheels where possible and only
 build python2 and python3 wheels in the special cases where necessary.
 
 The setup.cfg classifiers should be able to do that for us, though PBR
 may need updating? We will also need to learn to upload potentially 1
 
 How do you see that working? We want all of the Oslo libraries to,
 eventually, support both python 2 and 3. How would we use the classifiers
 to tell when to build a universal wheel and when to build separate
 wheels?
 
 The classifiers provide info on the versions of python we support. By
 default we can build python2 wheel if only 2 is supported, build python3
 wheel if only 3 is supported, build a universal wheel if both are
 supported. Then we can add a setup.cfg flag to override the universal
 wheel default to build both a python2 and python3 wheel instead. Dstufft
 and mordred should probably comment on this idea before we implement
 anything.

OK. I’m not aware of any python-3-only projects, and the flag to override the 
universal wheel is the piece I was missing. I think there’s already a 
setuptools flag related to whether or not we should build universal wheels, 
isn’t there?

 
 wheel in our wheel jobs. That bit is likely straight foward. The last
 thing that we need to make sure we do is that we have some testing in
 place for the special wheels. We currently have the requirements
 integration test which runs under python2 checking that we can actually
 install all the things together. This ends up exercising our wheels and
 checking that they actually work. We don't have a python3 equivalent for
 
 We only know the wheels can be installed. We don’t actually have a test
 that installs our code and runs it any more (devstack uses “develop” mode
 which bypasses some of the installation steps, as we found while fixing
 the recent neutron/pbr issue with a missing config file in their
 packaging instructions).
 
 Yup, this would be a second level of testing that we should consider as
 well. The post install can it do stuff test. We moved our unittesting
 away from using sdist installs in preference for setup.py develop
 equivalent which means that our unittests no longer cover some of this
 stuff for us. Sorry if I hijacked this thread into a how do we test
 release artifacts? thread.

I wonder if we should have a flag in devstack to control whether we install in 
develop or “regular” mode? That would let us test real installations, but still 
have editable versions for local developer systems.

 
 that job. It may be better to work out some explicit checking of the
 wheels we produce that applies to both versions of python. I am not
 quite sure how we should approach that yet.
 
 To fix the asymmetric gating we have between pbr and everything else,
 Robert suggested setting up 

Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-03 Thread Jeremy Stanley
On 2014-09-03 13:27:55 +0400 (+0400), Yuriy Taraday wrote:
[...]
 May be we should drop 3.3 already?

It's in progress. Search review.openstack.org for open changes in
all projects with the topic py34. Shortly I'll also have some
infra config changes up to switch python33 jobs out for python34,
ready to drop once the j-3 milestone has been tagged and is finally
behind us.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] [infra] Alpha wheels for Python 3.x

2014-09-02 Thread Clark Boylan
On Tue, Sep 2, 2014, at 11:30 AM, Yuriy Taraday wrote:
 Hello.
 
 Currently for alpha releases of oslo libraries we generate either
 universal
 or Python 2.x-only wheels. This presents a problem: we can't adopt alpha
 releases in projects where Python 3.x is supported and verified in the
 gate. I've ran into this in change request [1] generated after
 global-requirements change [2]. There we have oslotest library that can't
 be built as a universal wheel because of different requirements (mox vs
 mox3 as I understand is the main difference). Because of that py33 job in
 [1] failed and we can't bump oslotest version in requirements.
 
 I propose to change infra scripts that generate and upload wheels to
 create
 py3 wheels as well as py2 wheels for projects that support Python 3.x (we
 can use setup.cfg classifiers to find that out) but don't support
 universal
 wheels. What do you think about that?
 
 [1] https://review.openstack.org/117940
 [2] https://review.openstack.org/115643
 
 -- 
 
 Kind regards, Yuriy.
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

We may find that we will need to have py3k wheels in addition to the
existing wheels at some point, but I don't think this use case requires
it. If oslo.test needs to support python2 and python3 it should use mox3
in both cases which claims to support python2.6, 2.7 and 3.2. Then you
can ship a universal wheel. This should solve the immediate problem.

It has been pointed out to me that one case where it won't be so easy is
oslo.messaging and its use of eventlet under python2. Messaging will
almost certainly need python 2 and python 3 wheels to be separate. I
think we should continue to use universal wheels where possible and only
build python2 and python3 wheels in the special cases where necessary.

The setup.cfg classifiers should be able to do that for us, though PBR
may need updating? We will also need to learn to upload potentially 1
wheel in our wheel jobs. That bit is likely straight foward. The last
thing that we need to make sure we do is that we have some testing in
place for the special wheels. We currently have the requirements
integration test which runs under python2 checking that we can actually
install all the things together. This ends up exercising our wheels and
checking that they actually work. We don't have a python3 equivalent for
that job. It may be better to work out some explicit checking of the
wheels we produce that applies to both versions of python. I am not
quite sure how we should approach that yet.

Clark

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev