Re: [openstack-dev] [tripleo] Recap of Python 3 testing session at PTG

[Victor Stinner]

2018-04-13 15:07 GMT+02:00 Thomas Goirand :
> BTW, any progress on upstream Swift WRT Py3 support?

There is a voting Python 3.4 gate which runs 902 unit tests. The
Python 2.7 gate runs 5,902 unit tests. I compute that 15% of unit
tests pass on Python 3.4.

I tested locally with Python 3.5 (tox -e py35): 876 tests pass on
Python 3.5, 57 skipped and 1 failure:
"FAIL: test_get_logger_sysloghandler_plumbing
(test.unit.common.test_utils.TestUtils)"

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Recap of Python 3 testing session at PTG

[Victor Stinner]

2018-04-13 14:48 GMT+02:00 Thomas Goirand :
> On 03/17/2018 09:34 AM, Emilien Macchi wrote:
>> ## Challenges
>>
>> - Some services aren't fully Python 3
>
> To my experience switching everything to Py3 in Debian, the only issues
> were:
>
> - manila-ui
> - networking-mlnx

What about swift?

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] broken python35 job due to webob compatibility issues

[Victor Stinner]

Adding the charset sounds like a good practice, especially in Keystone 
which is security sensitive. See this old Python vulnerability:


http://python-security.readthedocs.io/vuln/cve-2011-4940_simplehttpserver_utf-7.html

"The list_directory() function in Lib/SimpleHTTPServer.py in 
SimpleHTTPServer in Python before 2.5.6c1, 2.6.x before 2.6.7 rc2, and 
2.7.x before 2.7.2 does not place a charset parameter in the 
Content-Type HTTP header, which makes it easier for remote attackers to 
conduct cross-site scripting (XSS) attacks against Internet Explorer 7 
via UTF-7 encoding."


Maybe in 2017, browsers don't have issues with encodings anymore, right? ;-)

I don't know the WebOb module, but I'm not surprised that it doesn't 
already add charset='utf-8' *by default*.


Victor


Le 29/03/2017 à 23:54, Lance Bragstad a écrit :

The keystone gate is currently broken [0]. This seems related to a
previous change we made to be compatible with webob 1.7 [1]. Looks like
we missed a couple spots in the original patch that are failing now that
we're using a newer version of webob.

There is a solution up for review [2] that should unblock the gate.

[0] 
http://logs.openstack.org/44/443344/6/gate/gate-keystone-python35/e162b3d/testr_results.html.gz
[1] https://review.openstack.org/#/c/422234/
[2] https://review.openstack.org/#/c/451559/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] issues with requiring python3 only tool?

[Victor Stinner]

Le 17/01/2017 à 17:36, Sean Dague a écrit :

When putting the cli interface on it, I discovered python3's argparse
has subparsers built in. This makes building up the cli much easier, and
removes pulling in a dependency for that. (Currently the only item in
requirements.txt is pbr). This is useful both from an ease to install,
as well as overall runtime.


Do you mean the argparse module of the Python standard library? It is 
available on Python 2.7. Subparsers are also supported on Python 2.7, no?

https://docs.python.org/2/library/argparse.html#sub-commands

If you need a more recent version of argparse on Python 2.7, you might try:
https://pypi.python.org/pypi/argparse

But I'm not sure that this third-party module is used on Python 2.7, 
since import checks the stdlib before checking site-packages.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding gdavoian to oslo-core

[Victor Stinner]

+1 from me.

Victor

Le 03/10/2016 à 19:40, Joshua Harlow a écrit :

Greetings all stackers,

I propose that we add Gevorg Davoian[1] to the oslo-core[2] team.

Gevorg has been actively contributing to oslo for a while now, both in
helping make oslo better via code contribution(s) and by helping with
the review load when he can. He has provided quality reviews and is
doing an awesome job with the various oslo concepts and helping make
oslo the best it can be!

Overall I think he would make a great addition to the core review team.

Please respond with +1/-1.

Thanks much!

- Joshua Harlow

[1] https://launchpad.net/~gdavoian
[2] https://launchpad.net/oslo

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding ozamiatin to oslo-core

[Victor Stinner]

+1 from me.

Victor

Le 03/10/2016 à 19:42, Joshua Harlow a écrit :

Greetings all stackers,

I propose that we add Oleksii Zamiatin[1] to the oslo-core[2] team.

Oleksii has been actively contributing to oslo for a while now, both in
helping make oslo better via code contribution(s) and by helping with
the review load when he can. He has provided quality reviews and is
doing an awesome job with the various oslo concepts and helping make
oslo the best it can be!

Overall I think he would make a great addition to the core review team.

Please respond with +1/-1.

Thanks much!

- Joshua Harlow

[1] https://launchpad.net/~ozamiatin
[2] https://launchpad.net/oslo

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kuryr] Python 3 usage and kuryr-kubernetes

[Victor Stinner]

Hi,

Le 30/08/2016 à 16:39, Antoni Segura Puimedon a écrit :

a) Work with the OpenStack distributions to ensure python3.5 support is
reached soon for Kuryr and its dependencies (some listed below):

  * babel
  * oslo.concurrency
  * oslo.config
  * oslo.log
  * oslo.utils
  * pbr
  * pyroute2
  * python-neutronclient
  * python-keystoneclient


I don't know pyroute2, but except of this one, other listed dependencies 
work on Python 3. Except pyroute2 and babel, all other dependencies have 
a voting CI running tests on Python 3. According to their PyPI page, 
babel and pyroute2 work on Python 3.


In term of packaging, Debian already provides OpenStack packages for 
Python 3. Just one example:


   https://packages.debian.org/sid/python3-neutronclient


This also implies that distros should adopt a policy about having
OpenStack services running in Python2, some in Python3, as I think the
best is to have each project move at its own speed (within reason).


"each project move at its own speed" -> exactly!

Moreover, there is no need to take any decision about Python 2 only or 
Python 3 only. The plan is to support Python 2 *and* Python 3 in the 
near future.


   https://wiki.openstack.org/wiki/Python3

I don't care of the long term plan for Python 2 at this point, since 
Python 3 is not fully supported yet ;-)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Status of the OpenStack port to Python 3

[Victor Stinner]

Hi,

Le 13/07/2016 à 21:17, Denis Makogon a écrit :

I have free capacity to work on porting code to Py3. So, if any PTL is
running out of team capacity i can help to work on project to enable Py3
support.


If you would like to help, you can try to run functional tests on 
DevStack. Begin with projects which have their unit tests already ported 
to Python 3.

https://wiki.openstack.org/wiki/Python3#Functional_and_Integration_Tests

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] New Python35 Jobs coming

[Victor Stinner]

Le 04/07/2016 à 07:59, Denis Makogon a écrit :

Then let it run for a day or two in our CI, discuss with neutron team,
and send a patch for project-config to change the setup,

Can confirm that nova, glance, cinder, heat clients are py35 compatible.


tox.ini of Nova, Swift and Trove need to be modified to copy/paste the 
whitelist of tests running on Python 3. Fix for Nova:


   https://review.openstack.org/#/c/336432/

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Need help to review patches porting Nova (unit tests) to Python 3

[Victor Stinner]

Hi,

I'm working on a new Python 3 patches to port the "last" Nova unit 
tests. For example, my current patch serie ports 1,107 more unit tests 
(76% => 84%). Changes should be "quite simple" but I need your help to 
review them! My serie of 10 patches starts at:

https://review.openstack.org/#/c/332701/

Please review also other Python 3 changes:
https://review.openstack.org/#/q/topic:bp/nova-python3-newton+status:open
("topic:bp/nova-python3-newton status:open")

If you need more information on Python 3, the reference is the wiki page:
https://wiki.openstack.org/wiki/Python3

Or come talk on the #openstack-python3 channel!

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Status of the OpenStack port to Python 3

[Victor Stinner]

Le 22/06/2016 à 09:18, Victor Stinner a écrit :

Current status: only 3 projects are not ported yet to Python 3:

* nova (76% done)
* trove (42%)
* swift (0%)


If you want to help, please review my patches! I sent patches to these 
projects.


Nova:
https://review.openstack.org/#/c/332686/

Swift:
https://review.openstack.org/#/c/333297/

Trove:
https://review.openstack.org/#/c/333262/
https://review.openstack.org/#/c/333267/
https://review.openstack.org/#/c/184400/ PyMySQL!
(Since december 2015, Trove already merged 17 of my patches for py3!)


Others wrote Python 3 patches, please review these patches too!

Nova:
https://review.openstack.org/#/q/topic:bp/nova-python3-newton+status:open

Swift:
https://review.openstack.org/#/q/project:openstack/swift+branch:master+topic:py3+status:open


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Status of the OpenStack port to Python 3

[Victor Stinner]

Le 23/06/2016 à 23:04, Thomas Goirand a écrit :

Just think about it for a while. If we get Nova to work with Py3, and
everything else is working, including all functional tests in Tempest,


Hum, that's a long list of expectations :-)



then after Otaca, we could even start to *REMOVE* Py2 support after
Otaca+1. That would be really awesome to stop all the compat layer
madness and use the new features available in Py3.


Even if everything works well, I disagree that we must drop immediately 
Python 2 support. We should give time to deployers to prepare their 
migration to Python 3. We need at least have one cycle *fully* 
compatible with Python 3 to be able to *prepare* dropping Python 2 support.


In practice, we already support Python 2 and Python 3 in the same code 
base. We already paid the price of supporting both major versions. Once 
all code is ported, it shouldn't be too expensive to maintain both versions.


One concrete issue is that running tests on Python 2 *and* Python 3 
multiply the risk of random failures by 2. It's just math... In my 
experience, fixing random failures is not the most favorite task of 
developers...


If you want to deprecate Python 2, the first question is *when* we can 
start to make Python 2 checks non-voting.




However, for this, it'd be super helful to have as much visibility as
possible. Are we setting a hard deadline for the Otaca release? Or is
this just a goal we only "would like" to reach, but it's not really a
big deal if we don't reach it?


In my experience, Python 3 work is seen as background refactoring work, 
and it is rarely seen as high-priority. You should be prepared to have 
to wait :-)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Status of the OpenStack port to Python 3

[Victor Stinner]

Le 23/06/2016 à 18:11, Doug Hellmann a écrit :

I'd like for the community to set a goal for Ocata to have Python
3 functional tests running for all projects.


We can already experiment functional tests right now ;-) I expect to get 
many small issues in all projects, but it may take time to fix all these 
small issues, before being to fix the real (Python 3) bugs in applications.


According to a recent discussion on #openstack-nova, I don't think that 
it will be possible to have Nova ported at the end of the Newton cycle.


So I guess that we will run nova and swift on Python 2 for these 
functional tests at the beginning, right?


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Status of the OpenStack port to Python 3

[Victor Stinner]

Le 22/06/2016 à 10:49, Thomas Goirand a écrit :

Do you think it looks like possible to have Nova ported to Py3 during
the Newton cycle?


It doesn't depend on me: I'm sending patches, and then I have to wait 
for reviews. The question is more how to accelerate reviews.


Right now, I'm splitting the great work made by dims during the previous 
cycle, 2 giant patches for nova, into smaller patches which should be 
easier to review.


When I will have enough changes, I will try to create to taskforce to 
review these changes. I don't expect that nova core reviewers have 
enough time to review all these changes. It would prefer to have a first 
review step made by contributors interested by Python 3.


My current Python 3 patches for nova:

https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/nova-python3

dims patches:

* (1/2) https://review.openstack.org/262083
* (2/2) https://review.openstack.org/261045

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Status of the OpenStack port to Python 3

[Victor Stinner]

Hi,

Current status: only 3 projects are not ported yet to Python 3:

* nova (76% done)
* trove (42%)
* swift (0%)

   https://wiki.openstack.org/wiki/Python3

Number of projects already ported:

* 19 Oslo Libraries
* 4 Development Tools
* 22 OpenStack Clients
* 6 OpenStack Libraries (os-brick, taskflow, glance_store, ...)
* 12 OpenStack services approved by the TC
* 17 OpenStack services (not approved by the TC)

Raw total: 80 projects. In fact, 3 remaining projects on 83 is only 4%, 
we are so close! ;-)


The next steps are to port the 3 remaining projects and work on 
functional and integration tests on Python 3.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Welcome Keystone to the World of Python 3

[Victor Stinner]

Le 21/05/2016 à 05:58, Morgan Fainberg a écrit :

We've gone through all of our test cases and all of our code base. At
this point Keystone is no longer skipping any of the tests (which do
tend to test the entire request stack) and we are properly gating on
being Python3.4 compatible.


By the way, *all* openstack projects now have a *voting* Python 3 check job!
https://wiki.openstack.org/wiki/Python3#OpenStack_applications_.28tc-approved.29

At least, the Python 3 support cannot regress ;-)

It's now time to work on functional tests and integration tests!

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db

[Victor Stinner]

Le 16/05/2016 16:12, Peter Stachowski a écrit :

We're aware that there are ways to mock (and un-mock) correctly.
We're trying to make sure that all our new test code follows those
patterns.  We also decided that it wouldn't make sense to change all
the working tests to use the 'right' methods as that could have the
short-term effect of destabilizing the tests considerably (plus it
would be a fair bit of effort with very little actual gain). As a
compromise we added this code to check that things weren't left
mocked (just in case someone copy-pasted the old style, did it wrong
and no-one noticed - we're still trying to maintain consistency
wherever possible).


Thanks for the explanation. But I don't understand something.

The purpose of the code detecting dangling mocks is to find bugs in 
"legacy" tests which don't use correctly the mock module. I read the 
code and I saw that it makes the unit test failing in such case. All 
unit tests pass on Python 2. Does it mean that all the legacy code is 
gone and all unit tests use correctly the mock module?


If I'm wrong, how can we detect remaining "dangling mocks"? Would it be 
possible to fix them at once to be able to remove the code detecting 
dangling mocks?


I proposed a patch to remove the code detecting dangling mocks: it makes 
unit tests 13x faster on Python 3 (183.6 sec => 13.8 sec):


   https://review.openstack.org/#/c/316955/

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db

[Victor Stinner]

Le 16/05/2016 16:12, Peter Stachowski a écrit :

Amrith is right though - python34 is *much* slower running this check
(and somewhat in general) than python27.  Maybe that doesn't
translate into real-world performance data, but it has made me a bit
nervous nonetheless.


IMHO we should look more closely to the output. See my comments on the 
bug report:


   https://bugs.launchpad.net/trove/+bug/1582257

Currently, running unit tests on Python 3 runs 686 tests in 15.971s, 
whereas Python 2 runs 1495 tests in 22.595s. Python 3 is "faster" :-) 
(16 sec < 23 sec)


The whole Python 3 job takes 10 minutes, but in these 10 minutes, almost 
8 minutes are taken just to build binary wheel packages. On Python 2, 
prebuilt packages are used and so creating the virtual environment is 
much faster (around one minute). I proposed a change to prebuild also 
wheel packages on Python 3:


   https://review.openstack.org/#/c/316890/

Ok, but sometimes we get timeout, right? Again, if you look closely to 
the output, between two runs on Python 2, the timing also changes a lot:


(a) Ran 1495 tests in 22.595s
(b) Ran 1495 tests in 393.364s => 17x slower!

The issue is not specific to Python 3. But it looks like the random 
slowdown is much larger on Python 3. Duration of Python 3 unit tests:


(a) "Ran 686 tests in 15.971s"
(b) "Ran 686 tests in 1581.090s" => 100x slower!

I guess that the difference is that Python 3 unit tests are currently 
run sequentially (1 process), whereas Python 2 unit tests are run in 
parallel (8 processes).


*Again* please check the code detecting dangling mocks. For example, 
"tox -e py34" takes 14 seconds without this code, 177 seconds with the 
code: 12x slower with the code.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db

[Victor Stinner]

Le 09/05/2016 16:16, Peter Stachowski a écrit :

Hi Amrith,

We’ve had a lot of ‘new’ tests enabled for py34 in the last week – from
your results you can see it’s running 6x the number of tests (although
it’s taking 8.5x as long).  It looks like python3 isn’t as fast as
python2.7?  If so, we may have to do something wrt the ‘dangling-mock’
detector code so that the tests run faster (I believe we can optimize it
now that all the tests have moved over to the new paradigm).


I opened a bug to collect information about this issue:
https://bugs.launchpad.net/trove/+bug/1582257

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db

[Victor Stinner]

Le 16/05/2016 13:52, Amrith Kumar a écrit :

IMHO the strange mock detector code must be removed. It is very
slow and I don't understand its purpose.


[amrith] It serves and has served a very useful purpose and that is
to detect bad tests where code has (and we've had lots of trouble
with this) established a Mock() but failed to delete it.


There are many options to disable (unregister, stop, call it as you 
want) mocks automatically. As I wrote, fixtures & oslotest give you 
tools to do that automatically. It's also a base feature of the mock 
module: "with mock.patch(...): ...". Sorry, I don't know enough the 
Trove code base (code of the unit tests) to say which option is the best.




I'd rather figure out why it is slower in Python3 because it may be
indicative of something that may impact other parts of the code as
well. We're having this whole discussion about Python performance and
the Go language, I think it is not a good idea to delete code which
is performing poorly because it is performing slowly.


Sorry but Go doesn't solve badly designed functions :-) It's an 
algorithmic complexity issue: the function has to iterate on *all* alive 
Python objects: complexity of O(n). I propose to remove to code to 
simplify the complexity to O(1) :-)


Ok, let's take an example with Python 2.7: the command "python -bb -m 
testtools.run trove/tests/unittests/common/test_exception.py" takes 972 
ms. Remove the mock workaround, the command now taks 1 ms: it's now 972x 
faster.


=> removing the slow workaround makes test ~1000x faster!

IMHO it's worth to start here, instead of starting to investigate why 
running tests on Python 3 seems slower.


I'm not aware of such huge performance difference between Python 2 and 
Python 3 when running unit tests on other OpenStack services. The issue 
is specific to Trove, and IMHO it comes from the mock workaround.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db

[Victor Stinner]

Hi,

Le 09/05/2016 16:38, Amrith Kumar a écrit :

So I’m able to run the py34 tests in about 10s if I run the dangling
mock detector with a depth of 0 or 1.


IMHO the strange mock detector code must be removed. It is very slow and 
I don't understand its purpose.


The mock module has a nice stop_all() method, why not using it?
https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.patch.stopall

The fixtures and oslotest modules also help to stop mocks.

(I don't know if the "mock detector" thing is related to the slow tests 
on Python 3.)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Removing Nova specifics from oslo.log

[Victor Stinner]

Le 13/04/2016 22:54, Julien Danjou a écrit :

There's a bunch of projects that have no intention of using
oslo.context, so depending and referring to it by default is something
I'd love to fade away.


It looks like Oslo has an identity crisis :-)

Basically the question looks like: should we make Oslo easier to use 
outside "OpenStack"? If I summarized correctly the question, my answer 
is YES!


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [All][Neutron] Improving development and review velocity - Is it time for a drastic change?

[Victor Stinner]

Le 01/04/2016 03:12, Assaf Muller a écrit :

2) Now, transform yourself six to twelve months in the future. We now
face a new problem. (...) I hereby propose that we remove the
tests. (...) Needless to say, my
proposal keeps pep8 in place. We all know how important a consistent
style is.


The good news is that it fits well with the next Python version, Python 
8, which will run pep8 checks for you when you import a module!

https://mail.python.org/pipermail/python-dev/2016-March/143603.html

It makes pep8 jobs useless.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [api] Reminder: WSME is not being actively maintained

[Victor Stinner]

Le 08/03/2016 22:22, gordon chung a écrit :

it seems like the main reason there are so many projects leveraging WSME
is because everyone is just using Ironic or Ceilometer as the starting
point for their APIs. can we officially say to all new projects who plan
on copying API logic of Ironic et al.: 'stop'.


Or maybe start to upgrade these two projects to use the state of the 
art, since they look to be used as example :-)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Status of Python 3 in OpenStack Mitaka

[Victor Stinner]

Hi,

I just wrote an article "Status of Python 3 in OpenStack Mitaka":
http://blogs.rdoproject.org/7894/status-of-python-3-in-openstack-mitaka

Summary:

* 13 services were ported to Python 3 during the Mitaka cycle: Cinder, 
Glance, Heat, Horizon, etc.

* 9 services still need to be ported
* Next Milestone: Functional and integration tests

“Ported to Python 3” means that all unit tests pass on Python 3.4 which 
is verified by a voting gate job. It is not enough to run applications 
in production with Python 3. Integration and functional tests are not 
run on Python 3 yet.


Join us in the #openstack-python3 IRC channel on Freenode to discuss 
Python 3! ;-)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] PTL for Newton and beyond

[Victor Stinner]

Le 03/03/2016 13:05, Flavio Percoco a écrit :

Thanks for all your hard work as an Oslo PTL. You did amazing and I
think you'd
do an awesome mentor for other folks as well. It was an honor to have
you as a
PTL and I look forward to keep working with you as an Oslo contributor.


I concur with Flavio, dims, you are really amazing! IMHO dims doesn't 
exist and is in fact a robot. I don't understand how he can handle so 
many projects in parallel and be good in what he does :-)


Dims is a meta liaison for the whole OpenStack project.

You put expectations on the Oslo PTL very high!

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][stable][oslo][all] oslo.service 0.9.1 release (liberty)

[Victor Stinner]

Hi,

Oh, I noticed myself the bug 
https://bugs.launchpad.net/nova/+bug/1538204 (nova failed to stop) on 
Grenade tests of one of my Cinder changes. Grenade started on Liberty 
with oslo.service 1.1.0, a version which didn't have my fixes for signal 
handling.


Good news: since upper-constraints.txt of Liberty was updated to use 
oslo.service 0.9.1, the random bug should now be gone (with my fixes for 
signal handling).


I hope that it will help to make Grenade tests more reliable!

Please keep me in touch if you still see the failure, especially this 
message in logs:

""AssertionError: Cannot switch to MAINLOOP from MAINLOOP" error"

Victor

Le 18/02/2016 11:05, Victor Stinner a écrit :

Hi,


Le 17/02/2016 19:29, no-re...@openstack.org a écrit :
 > We are chuffed to announce the release of:
 >
 > oslo.service 0.9.1: oslo.service library
 > (...)
 >
 > Changes in oslo.service 0.9.0..0.9.1
 > 
 >
 > 8b6e2f6 Fix race condition on handling signals
 > eb1a4aa Fix a race condition in signal handlers

This release contains two major changes to fix race conditions in signal
handling. Related bugs:

"Race condition in SIGTERM signal handler"
https://bugs.launchpad.net/oslo.service/+bug/1524907
=> "AssertionError: Cannot switch to MAINLOOP from MAINLOOP" error

"Failed to stop nova-api in grenade tests"
https://bugs.launchpad.net/nova/+bug/1538204
=> "oslo_service.threadgroup RuntimeError: dictionary changed size
during iteration"

oslo.service 0.9.1 is now in upper-contraints.txt and so will be
deployed on Liberty CIs:
https://review.openstack.org/#/c/280934/

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] Start to port Trove to Python 3 in Mitaka cycle?

[Victor Stinner]

Le 18/02/2016 14:15, Amrith Kumar a écrit :

Let's definitely discuss this again once you have all the changes that you feel 
should be merged for Mitaka ready.


I don't like working on long patch series. In my experience, after more 
than 4 patches, it's more expensive to maintain the patch serie than to 
write patches. So I prefer to work on few patches, wait until they are 
merged, and then write following patches.


I'm not going to write dozens of patches. I suggest to do as I done in 
the paste, make progress with baby steps :-)


For example, my first change only changes the py34 test environment in 
tox.ini, it cannot break anything on Python 2, and it's enough to fix 
"tox -e py34". It is not in conflict with any other pending change.

https://review.openstack.org/#/c/279098/

From this point, we can add a voting gate to be able to validate 
following Python 3 changes.



> What I would like to avoid is a dribble of changes where we don't 
know how much more we have coming down the pike.


You have to be prepared for dozens of small patches. It only depends on 
the size of your project (number of code line numbers) :-)


To have an idea, you can see the Cinder blueprint which has an 
exhaustive list of all changes made for Python 3:

https://blueprints.launchpad.net/cinder/+spec/cinder-python3

I counted 100 patches between June 2015 and February 2016.

FYI with all my pending patches for Cinder (only 4 changes remain), all 
unit tests will pass on Python 3!


It also gives you an idea of the time frame: it took me 9 months to port 
Cinder unit tests to Python 3. So more than a single OpenStack cycle (6 
months).


Since the port is long and painful, I would like to start as soon as 
possible :-)



> And while your changes may be "low risk", it does mean that if they 
merge now, the large feature sets that we have committed for this 
release will have to go through the cycle of merge conflicts, rebasing, 
code review, gate ... and so on.


The principle of technical debt is that the price only is only 
increasing if you wait longer :-) Merging Python 3 today or tomorrow 
doesn't solve the problem of merge conflicts :-)


It's really up to you to decide to "open the gate" for the flow of 
Python 3 patches, it's also up to you to control how much Python 3 
changes will merged. I can only offer my help to port code. I don't feel 
able to decide when it's the best time to start porting Trove ;-)


By the way, Gerrit provides a great "Conflicts With" information! It 
also helps to decide if it's ok to merge a Python 3 change, or if it's 
better to focus on the other changes in conflict.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [trove] Start to port Trove to Python 3 in Mitaka cycle?

[Victor Stinner]

Hi,

When I began to work on porting Trove to Python 3, I was blocked by 
MySQL-Python which is not compatible with Python 3. I tried a big change 
replacing MySQL-Python with PyMySQL, since other OpenStack services also 
moved to PyMySQL. But tests fail and I'm unable to fix them :-/

https://review.openstack.org/#/c/225915/

Recently, I noticed that the dependency is now skipped on Python 3 
(thanks to env markers in requirements.txt), and so "tox -e py34" is 
able to create the test environment.


So I abandoned my PyMySQL change (I will reopen it later) and started 
new simpler patches following the plan of my Python 3 blueprint for Trove:

https://blueprints.launchpad.net/trove/+spec/trove-python3

In short:

(1) fix the Python 3 gate
(2) make the Python 3 gate voting
(3) port more and more unit tests

My patches:

trove: "Add a minimal py34 test environment"
https://review.openstack.org/#/c/279098/
=> fix "tox -e py34", start with a whitelist of the 3 most basic unit tests

trove: "Port test_template unit test to Python 3"
https://review.openstack.org/#/c/279119/
=> port another unit test

openstack-infra/project-config: "Add non-voting gate-trove-python34 check"
https://review.openstack.org/#/c/279108/


IMHO these changes are simple and the risk of regression is low, but 
amrith wrote me "thanks for your change set but per last trove meeting, 
I think this should wait till mitaka is done, and we can pick it up 
early in newton."


I discussed with some Trove developers who are interested to start the 
Python 3 port right now. What do you think?


Maybe we can discuss that in the next Trove meeting?
https://wiki.openstack.org/wiki/Meetings/TroveMeeting
(Wednesdays at 18:00 UTC in #openstack-meeting-alt)

Oops, I just missed the meeting yesterday. I was too slow to write this 
email :-)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][stable][oslo] oslo.service 0.9.1 release (liberty)

[Victor Stinner]

Hi,


Le 17/02/2016 19:29, no-re...@openstack.org a écrit :
> We are chuffed to announce the release of:
>
> oslo.service 0.9.1: oslo.service library
> (...)
>
> Changes in oslo.service 0.9.0..0.9.1
> 
>
> 8b6e2f6 Fix race condition on handling signals
> eb1a4aa Fix a race condition in signal handlers

This release contains two major changes to fix race conditions in signal 
handling. Related bugs:


"Race condition in SIGTERM signal handler"
https://bugs.launchpad.net/oslo.service/+bug/1524907
=> "AssertionError: Cannot switch to MAINLOOP from MAINLOOP" error

"Failed to stop nova-api in grenade tests"
https://bugs.launchpad.net/nova/+bug/1538204
=> "oslo_service.threadgroup RuntimeError: dictionary changed size 
during iteration"


oslo.service 0.9.1 is now in upper-contraints.txt and so will be 
deployed on Liberty CIs:

https://review.openstack.org/#/c/280934/

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] eventlet 0.18.1 not on PyPi anymore

[Victor Stinner]

Le 17/02/2016 13:43, Henry Gessau a écrit :

And it looks like eventlet 0.18.3 breaks neutron:
https://bugs.launchpad.net/neutron/+bug/1546506


2 releases, 2 regressions in OpenStack. Should we cap eventlet version? 
The requirement bot can produce patches to update eventlet, patches 
which would run integration tests using Nova, Keystone, Neutron on the 
new eventlet version.


eventlet 0.18.2 broke OpenStack Keystone and OpenStack Nova
https://github.com/eventlet/eventlet/issues/296
https://github.com/eventlet/eventlet/issues/299
https://review.openstack.org/#/c/278147/
https://bugs.launchpad.net/nova/+bug/1544801

eventlet 0.18.3 broke OpenStack Neutron
https://github.com/eventlet/eventlet/issues/301
https://bugs.launchpad.net/neutron/+bug/1546506

FYI eventlet 0.18.0 broke WSGI servers:
https://github.com/eventlet/eventlet/issues/295

It was followed quickly by eventlet 0.18.2 to fix this issue.

Sadly, it looks like bugfix releases of eventlet don't include a single 
bugfix, but include also other changes. For example, 0.18.3 fixed the 
bug #296 but introduced "wsgi: TCP_NODELAY enabled by default" optimization.


IMHO the problem is not the release manager of eventlet, but more the 
lack of tests on eventlet, especially on OpenStack services.


Current "Continious Delivery"-like with gates do detect bugs, yeah, but 
also block a lot of developers when the gates are broken. It doesn't 
seem trivial to investigate and fix eventlet issues.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift

[Victor Stinner]

Hi,

Le 08/02/2016 14:42, Victor Stinner a écrit :
> https://review.openstack.org/#/c/237019/

Nice, this one was merged!



https://review.openstack.org/#/c/237027/
https://review.openstack.org/#/c/236998/


These two changes look to be blocked by encodings issues. Tim Burke, 
Samuel Merritt and Christian Schwede have concerns about the exact 
behaviour on Python 3.


Let me explain how I'm working. I spend between 10 minutes and one day 
to port a single unit test and then I submit a patch (or patches if the 
change is big). I prefer to work incrementally: first fix the unit test, 
and then enhance the code if needed.


Right now, unit tests don't pass, there are still serious bytes vs 
Unicode issues. I would prefer to reach a first milestone where unit 
tests pass and then discuss how to fix complex encoding issues.


Change 236998: For the hmac patch, supporting arbitrary hash prefix or 
suffix requires to implement a new feature. Swift parses the 
configuration files using the ConfigParser which doesn't support 
arbitrary bytes. I understand the use case and I agree that something 
should be done, but I suggest to do that later.


Change 237027: For the encoding of HTTP headers, it looks like Swift 
doesn't respect HTTP RFCs. The HTTP requires headers to be encoded to 
Latin1, but Swift (server or client, sorry I don't know) encode headers 
to UTF-8. Something should be do too, but it will require a deep 
analysis, prepare a transition period, etc. This problem is complex and 
cannot be fixed right now.


Just to be clear: Swift does not support Python 3, so we are still free 
to change completely the code for Python 3, until Swift is fully 
compatible with Python 3.


Swift is not my main target, so I cannot spend too much time on it. If 
you expect me to solve all issues at once, I'm sorry, I cannot help :-/


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] tenant vs. project

[Victor Stinner]

Le 12/02/2016 13:01, Sean Dague a écrit :

OpenStack is wildly inconsistent in it's use of tenant vs. project.


Yeah, it's time to find a 3rd term to avoid confusion! ;-)

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift

[Victor Stinner]

Le 12/02/2016 15:12, Ian Cordasco a écrit :

Just to interject here, RFC 2616 is the one you're talking about. That
encoding requirement was dropped when HTTP/1.1 was updated in the
7230-7235 RFCs. (...)
Where VCHAR is any visible US ASCII character. So while UTF-8 is still
a bad idea for the header value (and in fact, http.client on Python 3
will auto-encode headers to Latin 1) Latin 1 is no longer the
requirement.

For those interested, you can read up on headers in HTTP/1.1 here:
https://tools.ietf.org/html/rfc7230#section-3.2


Oh thanks, it looks like my HTTP skills are rusty :-)

For Swift, it's maybe better to always try to use UTF-8, but fallback to 
Latin1 if an HTTP cannot be decoded from UTF-8. Swift has many clients 
implemented in various programming languages, I'm not sure that all 
clients use UTF-8.


By the way, https://review.openstack.org/#/c/237027/ was merged, cool.

I fixed the third patch mentioned in my previous email to support 
arbitrary byte strings for hash prefix and suffix in the configuration file:

https://review.openstack.org/#/c/236998/

I also updated my HTTP parser patch for Python 3. With these two 
patches, test_utils now pass on Python 3.

https://review.openstack.org/#/c/237042/

For me, it's a nice milestone to have test_utils working on Python 3. It 
allows to port more interesting stuff and start the real portage work ;-)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [swift] Plan to add Python 3 support to Swift

[Victor Stinner]

Hi,

The Swift port was blocked during 6 months by a complex issue related to 
PyEClib. Good news: this issue was fixed 2 months ago. After the Liberty 
Summit, a plan was defined to port Swift to Python 3 (end of october), 
and I understood that the whole Swift team agreed on it.


Some Python 3 changes were merged, but less than I expected. My 3 latest 
patches are waiting for reviews since october 19, 2015:


https://review.openstack.org/#/c/237027/
https://review.openstack.org/#/c/236998/
https://review.openstack.org/#/c/237019/

I can write patches, but it takes time to rebase them and to regularly 
try various ways to try to get a review (like ping on IRC).


Because of the general lack of interest of Swift developers for Python 
3, I think that I will simply abandon my patches and let others port Swift.


Victor


Le 30/10/2015 06:54, John Dickinson a écrit :

Thanks for the update. This seems like a reasonable way forward, but also one 
that will take a long time. Thank you for your work.

I think this will result in larger and larger chunks of work, and so it will 
eventually result in large patches to move different components to py3. So 
you'll be able to start small, but the work will get larger as you go.

You're right about needing the voting gate job. That should be the first 
priority for py3 work.

--John




On 30 Oct 2015, at 12:47, Victor Stinner wrote:


Hi,

We talked about Python 3 with Christian Schwede, Alistair Coles, Samuel Meritt, 
Jaivish Kothari and others (sorry, I don't recall all names :-/) during Swift 
contributor meetup. It looks like we had an agreement on how to add Python 3 
support to Swift. The plan is:

1) Fix the gate-swift-python34 check job

2) Make the gate-swift-python34 check job voting

3) Port remaining code step by step (incremental development)

Python 3 issues had been fixed in the past in Swift, but came back. So it's 
important to not reintroduce such regressions by making the gate voting.

Christian said that he will explain the plan at the next Swift meeting 
(Wednesday). I don't think that I will be able to attend this meeting, I have 
another one at the same time with my team :-/

I can put this plan in a blueprint if you want. So we can refer to the 
blueprint in Python 3 changes. It's up to you.


Plan in detail.

(1) To fix the Python 3 job, the idea is to only run a subset of tests on 
Python 3. For example, if we fix Python 3 issues with dnspython (dnspython3) 
and PyEClib dependencies, we can run
"nosetests test/unit/common/test_exceptions.py" on Python 3 (the test pass on 
Python 3).

We need these two changes:

* "py3: Update pbr and dnspython requirements"
https://review.openstack.org/#/c/217423/

* "py3: Add py34 test environment to tox"
https://review.openstack.org/#/c/199034/


(2) When the gate-swift-python34 check job will pass and we waited long enough 
to consider that it's stable, we can make it voting. At this point, we cannot 
introduced Python 3 regressions on the code tested on Python 3. Then the idea 
is to run more and more tests on Python 3.


(3) Ok, now the interesting part. To port remaining code, following changes 
will enlarge the code coverage of Python 3 tests by adding new tests to 
tox.ini. For example, port utils.py to Python 3 and add test_utils.py to 
tox.ini.


Misc questions.

Q: "Is it possible to port Swift to Python 3 in a single patch?"

A: Yes, it is technically possible. But it would be one unique giant patch 
simply impossible to review and that will conflict at each merged change. Some 
changes required by Python 3 need discussions and to make technical choices.  
It's more convenient to work on smaller patches.

Q: "How much changes do we need to port Swift to Python ?"

A: Sorry, I don't know. Since we cannot run all tests on Python 3 right now, we 
cannot see all issues. It's really hard to estimate the number of required 
changes. Anyway, the plan is to port the code step by step.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][oslo][all] oslo.context 2.0.0 release (mitaka)

[Victor Stinner]

(I added [all] to the topic)

Hi,

Le 02/02/2016 17:20, dava...@gmail.com a écrit :

We are thrilled to announce the release of:
oslo.context 2.0.0: Oslo Context library
(...)
4a8a1df Fix request_id type on Python 3: use text (Unicode)


This change is a deliberate incompatible change in the API, but only on 
Python 3. If you notice any recent failure related to oslo.context (2.0) 
on Python 3, please bug me! I will investigate the issue.


I analyzed the code of all OpenStack applications and prepared the code 
of some applications in master and liberty to avoid any gate failure. 
But shit happens ;-)


Hopefully, this change should fix oslo.log on Python 3.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adding Dmitry Ukhlov to Oslo-Messaging-Core

[Victor Stinner]

+1 for me

Le 28/01/2016 16:25, Davanum Srinivas a écrit :

Team,

Dmitry has been focused solely on the Pika Driver this cycle:
http://stackalytics.com/?user_id=dukhlov=commits

Now that we have Pika driver in master, i'd like to invite Dmitry to
continue his work on all of Oslo.Messaging in addition to Pika.
Clearly over time he will expand to other Oslo stuff (hopefully!).
Let's please make add him to the Oslo-Messaging-Core in the meantime.

Thanks,
Dims



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] It's better to ask forgiveness than permission

[Victor Stinner]

(I changed the title to stop hijacking the Oslo thread.)

Hi,

Le 30/01/2016 22:25, Julien Danjou a écrit :
> (...) And it's easier/faster to fix with a larger team than a few.
> Which mean inclusion. Which mean openness.

While I think that Julien is a little bit rude and his email is stongly 
opinionated, I have to agree with his global idea of openness.


IMHO some groups in OpenStack are too conservative which makes the 
review process slower and slower every day and can easily discourage 
motivated contributors. I understand that changing core parts of a 
project require a long analysis, but it's sad that simple fixes, cleanup 
changes, etc. can sometimes be stuck for many months before being 
abandoned :-/


A side effect is that it became hard to reduce the technical debt in 
some projects, or said differently: the technical debt became high in 
some projects, and no solution was found to reduce it.


I prefer to trust developers. Everyone knows the impact of changes in 
OpenStack. I'm sure that developers understand that they are supposed to 
only modify some parts of a project and need more skills to remove the 
tricky parts of the core.


I'm a strong supporter of "It's better to ask forgiveness than permission".

Hopefully, as dims wrote, each group is free to choose its own internal 
policy for contributions ;-)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][osdk] PrettyTable needs a home in OpenStack

[Victor Stinner]

Thanks for investigating the tabulate option :-)

Victor

Le 26/01/2016 02:25, Joshua Harlow a écrit :

As far as the other option (using tabulate):

https://bitbucket.org/astanin/python-tabulate/pull-requests/25/

That was a (very very basic) POC for a potential compatibility layer,

The author (of tabulate) though thinks that a
'tabulate-prettytable-compat' library might be the best option, which
seems to make sense. So if the jazzband thing doesn't work out we can
work with the tabulate author to create such a thing (and then as time
goes on just move to tabulate).

-Josh

Flavio Percoco wrote:

On 08/01/16 08:36 -0430, Flavio Percoco wrote:

Greetings,

As some of you know already, google code is going to be shutdown. Some
projects we're using are hosted and, unfortunately, some of them are
unmaintained and perhaps going away.

One of these projects is PrettyTable. This point was raised by Erno in
this patch[0] from jd__. PrettyTable is not just being used in several
openstack specific projects but it's also a transitive dependency for
all client libraries using cliff.

With all that in mind, I've contacted the author of the library and
asked him if it'd be ok for us (OpenStack) to adopt this library. The
author accepted and granted me access to the project on pypi.

I'm saying all the above because we now need to find a home for it in
OpenStack.

I've identified 2 possible places:

1) Oslo, as we maintaing cross-project libraries and some of them are
not in the oslo namespace

2) OpenStack Client team as they maintain cliff already and it'd
perhaps make more sense to have this library there.

One thing to note is that this library has been quite stable, which
means it won't, hopefully, add too much work to the team.

Thoughts?

[0] https://review.openstack.org/#/c/234340/


FWIW, we've decided to give jazzband[0] a try and host prettytable
there. If
that doesn't work out well, we might revisit this option (unless the
tabulate
migration happens before that).

[0] https://jazzband.co

Thanks everyone and Doug for suggesting jazzband!
Flavio



__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] gate blockage due to eventlet 0.18

[Victor Stinner]

Hi,

Oops, I feel guilty, I wrote the patch which introduced the regression 
:-/ My patch fixes a bug on Python 3: "sock.makefile('rb').readline() 
doesn't handle blocking errors correctly"

https://github.com/eventlet/eventlet/issues/274

I was relying on the eventlet test suite, but it looks like it lacks an 
unit test on sendto(). FYI I sent a pull request sunday, but just after 
that I noticed that Jakub Stasiak already fixed the regression. My new 
pull request adds an unit test on sendto() and recvfrom() (UDP socket) 
which should help to avoid similar regression:

https://github.com/eventlet/eventlet/pull/292

The best would be to run some OpenStack tests on the development branch 
of eventlet in a CI, or at least run OpenStack tests on new eventlet 
release. It became common to see regression on eventlet releases, it 
looks like eventlet test suite is too small.


Victor

Le 24/01/2016 20:48, Andreas Jaeger a écrit :

On 01/24/2016 08:01 PM, Jeremy Stanley wrote:

On 2016-01-24 13:39:38 -0500 (-0500), Sean Dague wrote:

Something about the eventlet 0.18 release is failing the cloudpipe
functional tests, as well as our docs job (which is really really odd).

An eventlet pin has been posted - https://review.openstack.org/271809 -
once landed it should let the spice flow again. If someone could look
into it deeper it would be appreciated. I know a lot of us are traveling
over the next 24 hours, so not sure who is going to have time to dig in.
But it will be massively appreciated.


Dims reported a related bug upstream:

 https://github.com/eventlet/eventlet/issues/290



0.18.1 was just released and should fix this.

I could reproduce the failure building the docs with 0.18 but with
0.18.1 it works again for me,

Andreas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Python 3.5 is now the default Py3 in Debian Sid

[Victor Stinner]

Hi,

What are the issues? Is there a list of issues somewhere?

Victor

Le 13/01/2016 03:07, Thomas Goirand a écrit :

Hi,

Just a quick notice for those not following developments in Debian.
Python 3.5 is now the default Python 3 interpreter in Debian Unstable,
and when the transition will be finished [1], it's going to be the one
in Testing (aka: Stretch) as well. The current plan is to have Python
3.5 only in Stretch (ie: get rid of Python 3.4 completely).

Stretch is to be frozen in late 2016, so there's a lot of chances that
it's going to include Mitaka.

In other words, any Python 3.5 problem in Olso, clients and so on will
be considered a Debian RC bug and shall be addressed ASAP there.

Cheers,

Thomas Goirand (zigo)

[1] https://release.debian.org/transitions/html/python3.5.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][osdk] PrettyTable needs a home in OpenStack

[Victor Stinner]

Le 11/01/2016 10:37, Thierry Carrez a écrit :

Joshua Harlow wrote:

[...]
So I'd def help keep prettytable going, of course another option is to
move to https://pypi.python.org/pypi/tabulate (which does seem active
and/or maintained); tabulate provides pretty much the same thing
(actually more table formats @
https://pypi.python.org/pypi/tabulate#table-format ) than prettytable
and the api is pretty much the same (or nearly).

So that's another way to handle this (just to move off prettytable
entirely).


This sounds like a reasonable alternative...



IMHO contributing to an actively developped library (tabulate) seems 
more productive than starting to maintain a second library which is 
currently no more maintained.


Does anyone know how much code should be modified to replace prettytable 
with tabulate on the whole OpenStack project?


--

I don't like the global trend in OpenStack to create a new community 
separated from the Python community. In general, OpenStack libraries 
have too many dependencies and it's harder to contribute to other 
projects. Gerrit is less popular than Github, and OpenStack requires to 
sign a contributor agreement. I would prefer to stop moving things into 
OpenStack and continue to contribute to existing projects, as we already do.


(I also know why projects are moved into OpenStack "big tent", they are 
some good arguments.)


Well, that's my feeling that OpenStack and Python communities are 
splitted, maybe I'm wrong ;-) I just want to avoid what happened in 
Zope: a lot of great code and great libraries, but too many dependencies 
and at the end a different community.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][keystone] Move oslo.policy from oslo to keystone

[Victor Stinner]

Le 16/12/2015 20:33, Davanum Srinivas a écrit :

Brant,

I am ok either way, guess the alternative was to add keystone-core
directly to the oslo.policy core group (can't check right now).

The name is very possibly going to create confusion

-- Dims


I heard some people consider that "OpenStack" and "Oslo" are two 
projects with different goal and two separated developer groups.


It's sad that the purpose of Oslo is misundestood :-/ Oslo project is 
working for OpenStack. Oslo developers collaborate closely with all 
OpenStack subprojects, they take care of backward compatibility, they 
work to reduce the technical debt, and they regulary help to fix Oslo 
issues in other projects. So Oslo doesn't only produce code and break 
code for free, Oslo developers are also consumers of Oslo code and make 
sure that everything works fine.


Sorry, this email is not really related to the Keystone discussion, it's 
just to share my feelings on Oslo.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Congress] Python 3 ready

[Victor Stinner]

Hi,

Le 04/12/2015 05:12, Eric K a écrit :

Congress can finally pass python34 gating.

Here’s the last patch needed to make it happen.
https://review.openstack.org/#/c/253298/


Great job :-) Congrats.

I see that antlr3 was ported to Python 3 in the commit 
0576d774a49dd310970974d0c881e8bd4915c2eb. This part blocked me, I don't 
know Java and ANTLR upstream development is now focused in the version 4.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][oslo] On Python 3, request_id must by Unicode, not bytes

[Victor Stinner]

Hi,

The next oslo.context release including the following change (still 
under review) might break the voting Python 3 gate of your project:


   https://review.openstack.org/#/c/250731/

Please try to run Python 3 tests of your project with this change. I 
already ran the tests on Python 3 of the following projects:


- ceilometer
- cinder
- heat
- neutron
- nova


The type of the request_id attribute of oslo_context.RequestContext was 
changed from Unicode (str) to bytes on Python 3 in April "to fix an unit 
test". According to the author of the change, it was a mistake.


The request_id is a string looks like 'req-83a6...', 'req-' followed by 
an UUID. On Python 3, it's annoying to manipulate a bytes string. For 
example, print(request_id) writes b'req-83a6...' instead of req-83a6..., 
request_id.startswith('req-83a6...') raises a TypeError, etc.


I propose to modify request_id type again to make it a Unicode string to 
fix oslo.log (don't log b'req-...' anymore, but req-...):


   https://review.openstack.org/#/c/250731/

It looks like it doesn't break services, but only one specific unit test 
duplicated in some projects. The unit test relies on the exact 
request_id type, it looks like: request_id.startswith(b'req-'). I fixed 
this unit test in Glance, Neutron and oslo.middlware to accept bytes and 
Unicode request_id.


I also searched for b'req-' in http://codesearch.openstack.org/ to find 
all projects relying on the exact request_id type.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Thoughts on python-future?

[Victor Stinner]

Hi,

I'm porting OpenStack code to Python 3 for two years. I'm happy with 
six. It has been adopted by all OpenStack projects which are being 
ported to (or have been ported to) Python 3. It was discussed to use 
python-future in Swift, but Swift is now using six too.


I wrote a tool to port an OpenStack project to six:
https://pypi.python.org/pypi/sixer

It does at least half of the port for you. I wrote the tool to be able 
to produce patches reviewable by a human: you can easily select which 
operations are enabled or not to produce smaller patches, and you can 
rerun the tool multiple times.


FYI I wrote an article to explain why I wrote a new tool for OpenStack:
https://haypo.github.io/python3-sixer.html

Victor

Le 25/11/2015 03:33, Eric Kao a écrit :

Hi all,

I’ve been using the python-future library for Python 3 porting and want
to see what people think of it.
http://python-future.org/overview.html#features

The end result is standard Python3 code made compatible with Python2
through library imports. The great thing is that Python 3 execution is
mostly independent of the library, so once Python 2 support is dropped,
the use of the library can be dropped too.

Anyone know why it’s not used in OpenStack perhaps alongside six? Thanks!


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Thoughts on python-future?

[Victor Stinner]

All informations on Python 3 are on the wiki:
https://wiki.openstack.org/wiki/Python3

You may also join the IRC channel on Freenode: #openstack-python3

Victor

Le 25/11/2015 03:33, Eric Kao a écrit :

Hi all,

I’ve been using the python-future library for Python 3 porting and want
to see what people think of it.
http://python-future.org/overview.html#features

The end result is standard Python3 code made compatible with Python2
through library imports. The great thing is that Python 3 execution is
mostly independent of the library, so once Python 2 support is dropped,
the use of the library can be dropped too.

Anyone know why it’s not used in OpenStack perhaps alongside six? Thanks!


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] Proposal for new core reviewer: ChangBo Guo

[Victor Stinner]

Hi,

I noticed that ChangBo Guo (aka gcb) is very active in various Oslo 
projects, to write patches but also to review patches written by others. 
He attend Oslo meetings, welcome reviews, etc. It's a pleasure to work 
with him.


To accelerate the Oslo development, I propose to invite ChangBo to 
become an Oslo core reviewer. I asked him privately and he already 
replied that it would be a honor for him.


What do you think?

ChangBo Guo's open changes:

https://review.openstack.org/#/q/owner:glongw...@gmail.com+status:open,n,z

ChangBo Guo's merged changes:

https://review.openstack.org/#/q/owner:glongw...@gmail.com+status:merged,n,z

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Should we make qpid/proton optional dependencies? (please say yes)

[Victor Stinner]

Hi,

Would it be possible to use the "extra dependencies" thing in setup.cfg? 
Robert Collins started a similar change for Oslo DB for make DB drivers 
optional:

https://review.openstack.org/#/c/184328/

Sadly, this change was not merged yet.

@Robert: any progress on this change?

It would allow to ask for "Oslo Messaging with Qpid support" in service 
dependencies, without knowning the exact required Python packages for Qpid.


Victor

Le 19/11/2015 19:22, Matt Riedemann a écrit :

I want to fix a thing in oslo.messaging but to my surprise I can't run
pep8. This is because python-qpid-proton tries to install qpid-proton
and it can't get that because the apache site where it's hosted is down.
[1]

The apache site for qpid has actually been down a lot lately, I
coincidentally know that because of trying to access QPID security
issues for backports to things like Grizzly - yay for me!

Anyway, depending on a 3rd party repo like this to get packages just to
run pep8/unit tests in oslo.messaging is the suck.  Can we make this
optional?  Isn't the qpid stuff in tree deprecated anyway?

[1] http://paste.openstack.org/show/479460/



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Graduate cliutils.py into oslo.utils

[Victor Stinner]

Le 16/11/2015 08:33, Kekane, Abhishek a écrit :

Hi,

As apiclient is now removed from oslo-incubator, to proceed with
request-id spec [1] I have two options in mind,

1.Use keystoneauth1 + cliff in all python-clients (add request-id
support in cliff library)

2.apiclient code is available in all python-*clients, modify this code
in individual clients and add support to return request-id.


(1) is the obvious choice for me. Modifying each old copy of apiclient 
would take a lot of time and we don't want to maintain this code anymore.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] XxxOpt classes "kept for backward-compatibility" in oslo.config

[Victor Stinner]

Le 10/11/2015 15:33, Davanum Srinivas a écrit :

+1 Victor.


Ok thanks for the confirmation. The comment disturbed me a lot :-D

Here is a simple change to remove the comment:
https://review.openstack.org/243630

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] Graduate cliutils.py into oslo.utils

[Victor Stinner]

Hi,

At Tokyo, we discussed quickly what to do with cliutils.py of 
oslo-incubator, but for me the plan is not really concrete.


https://etherpad.openstack.org/p/mitaka-oslo-incubator-cleanup

I don't like dims' suggestion to move code inside clients, it means 
duplicating 280 lines of python in 11 clients.


It was also proposed to reuse openstackclient or the openstack SDK. I 
don't understand how novaclient can use openstackclient, since 
openstackclient depends on python-novaclient... The OpenStack SDK 
project is also a large library no? (I don't know well the OpenStack SDK 
project.)


I would prefer to move cliutils.py to oslo.utils. Only 3 clients use 
cliutils.py but don't depend on oslo.utils yet. The 8 remaining clients 
using cliutils.py already depends on oslo.utils.


An alternative is to create yet another oslo library. But it means 
creating a library for a single file of 280 lines.


What do you think?

--

On 29 "python-*client" projects, I found 11 clients using cliutils.py:

python-ceilometerclient
python-cloudkittyclient
python-heatclient
python-ironicclient
python-magnumclient
python-manilaclient
python-mistralclient
python-novaclient
python-saharaclient
python-solumclient
python-tuskarclient

9 clients are almost synchronized with oslo-incubator: only the new 
optional 'dict_value' parameter of the print_list() function is missing, 
not a big deal (since the parameter is optional, it doesn't break the API).


magnumclient and mistralclient have larger changes.

magnumclient adds a recursive keys_and_vals_to_strs() function which 
casts all dictionary keys and values to string. We may copy this feature 
in the upstream code (oslo incubator), it makes sense to me.


cliutils.py of mistralclient looks outdated, but a find_resource() 
function was also added. We can move find_resource() out of cliutils.py, 
to put it somewhere in mistralclient.


On the 11 clients using cliutils.py, 8 depend oslo.utils, whereas 3 don't.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Graduate apiclient library

[Victor Stinner]

Hi,

Le 10/11/2015 07:29, Kekane, Abhishek a écrit :

We have done extensive analysis to figure out the differences of usages of
different oslo-incubator/openstack/common/apiclient modules used in the
respective Openstack python client libraries and added all our
observations in the google spreadsheet [2].


Good job! I added my +2 to your spec.
https://review.openstack.org/#/c/235200/



Possible resolutions:-
1) Remove auth_plugin.py from respective python client libraries and add
all required common functionality in auth.py. Add auth.py module into the
new apiclient library.
2) Remove auth.py completely and add auth_plugins.py module in the
respective python client libraries. No need to add auth.py into the new
apiclient library.
3) Check if keystoneauth library has all needed functionality present in
auth.py and auth_plugin.py. If present, don¹t include auth.py in the new
client library and eliminate auth_plugin.py from python client libraries
and instead use keystoneauth wherever needed.


I don't like (2): security matters, it's better to use the same code in 
all clients.


IMHO the best would be (3): put required functions directly in keystoneauth.

If it's not possible for whatever reason, (1) is my second favorite choice.



exceptions.py
===

Please refer to the exception classes present in the respective python
client libraries in the google spreadsheet ³exception_details².
All common exceptions from respective python client libraries should be
moved to the exception.py module and this module should be part of the new
apiclient library.


Agreed, but only common exceptions. For example, only solumclient 
defines and uses NotUnique, there is no need to put it the oslo.apiclient.




fake_client.py
===

Retain this module as it is for unit testing purpose.


It might be moved to oslo_apiclient/tests/ directory.



Possible resolutions:
1. Move utils.py to the new apiclient library and delete find_resource
method from all python client libraries and make them use it from the
apiclient library
2. Simply not include utils.py to the new apiclient library and implement
find_resource method in the manila client.
We prefer to implement option #1.


Since all clients need this function, I would prefer to put it in the 
new library. I hope that it's not to hard to refactorize the code to 
have one unique implementation.



Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] XxxOpt classes "kept for backward-compatibility" in oslo.config

[Victor Stinner]

Hi,

In oslo_config/cfg.py, there is a main Opt class which takes an optional 
type parameter (default is oslo_config.types.String). But there is also 
a long list of XxxOpt classes: StrOpt, BoolOpt, FloatOpt, ListOpt, 
IPOpt, etc. Recently I saw *new* classes added with the "kept for 
backward-compatibility" comment, comment copied from existing classes.


What is the plan for these XxxOpt classes? Is there a clear plan to 
deprecate them and remove them? Or can we agree on keeping them and so 
removing the comment?


It looks like oslo_config.cfg.Opt is *never* used directly in OpenStack. 
Only specialized options StrOpt, IntOpt etc. are used.


So the obvious choice is to remove the comment, right?

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Graduate cliutils.py into oslo.utils

[Victor Stinner]

Le 10/11/2015 14:01, Davanum Srinivas a écrit :

That's exactly what i want as well. We should just let whoever has
copies to do whatever they want with them and drop our oversight of
it. Since they are mature and haven't been touched for a while, i
really don't see the need to take care of them.


Ok, let's say that cliff+keystoneauth is the new way to write clients. 
13 clients already use cliff.


On these 13 clients, 5 are still using clituils.py:

* python-heatclient
* python-ironicclient
* python-mistralclient
* python-saharaclient
* python-tuskarclient

and 6 are still using apiclient/:

* python-congressclient
* python-heatclient
* python-ironicclient
* python-mistralclient
* python-saharaclient
* python-tuskarclient

Should I understand that cliff and keystoneauth doesn't contain all 
functions requires by these clients?


If yes, does it make sense to move some code from cliutils.py and 
apiclient to cliff / keystoneauth?


If no, we can just kill cliutils.py and apiclient in Oslo Incubator, and 
help clients to upgrade their code to cliff and keystoneauth.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][bandit] Handling bandit configuration files in Oslo.

[Victor Stinner]

Le 02/11/2015 19:40, Brant Knudson a écrit :

(...) by typing something like:

$ bandit-conf-generator --disable try_except_pass --out bandit.yaml
oslo.messaging ~/openstack/bandit/bandit/config/bandit.yaml


(...) we should have a config file for bandit-conf-generator...
but then why not just have bandit know how to read the
bandit-conf-generator config file and skip the extra step?


Hi,

I don't like very long command lines, it's hard to document them or 
comment them. I prefer configuration files. But bandit.yaml, the 
"template", is already a configuration file!?


As Brant wrote, we should enhance Bandit to use a simpler configuration 
file. Or maybe we should have our own configuration file which on ly 
contains "differences" between the YAML template and the expected YAML 
output configuration file. Basically, the "differences" is what you 
wrote on the command line.


Anyway, it would be better to add this new bandit-conf-generator tool 
(or making config simpler) directly in Bandit. What do you think Cyril?


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [swift] Plan to add Python 3 support to Swift

[Victor Stinner]

Hi,

We talked about Python 3 with Christian Schwede, Alistair Coles, Samuel 
Meritt, Jaivish Kothari and others (sorry, I don't recall all names :-/) 
during Swift contributor meetup. It looks like we had an agreement on 
how to add Python 3 support to Swift. The plan is:


1) Fix the gate-swift-python34 check job

2) Make the gate-swift-python34 check job voting

3) Port remaining code step by step (incremental development)

Python 3 issues had been fixed in the past in Swift, but came back. So 
it's important to not reintroduce such regressions by making the gate 
voting.


Christian said that he will explain the plan at the next Swift meeting 
(Wednesday). I don't think that I will be able to attend this meeting, I 
have another one at the same time with my team :-/


I can put this plan in a blueprint if you want. So we can refer to the 
blueprint in Python 3 changes. It's up to you.



Plan in detail.

(1) To fix the Python 3 job, the idea is to only run a subset of tests 
on Python 3. For example, if we fix Python 3 issues with dnspython 
(dnspython3) and PyEClib dependencies, we can run
"nosetests test/unit/common/test_exceptions.py" on Python 3 (the test 
pass on Python 3).


We need these two changes:

* "py3: Update pbr and dnspython requirements"
  https://review.openstack.org/#/c/217423/

* "py3: Add py34 test environment to tox"
  https://review.openstack.org/#/c/199034/


(2) When the gate-swift-python34 check job will pass and we waited long 
enough to consider that it's stable, we can make it voting. At this 
point, we cannot introduced Python 3 regressions on the code tested on 
Python 3. Then the idea is to run more and more tests on Python 3.



(3) Ok, now the interesting part. To port remaining code, following 
changes will enlarge the code coverage of Python 3 tests by adding new 
tests to tox.ini. For example, port utils.py to Python 3 and add 
test_utils.py to tox.ini.



Misc questions.

Q: "Is it possible to port Swift to Python 3 in a single patch?"

A: Yes, it is technically possible. But it would be one unique giant 
patch simply impossible to review and that will conflict at each merged 
change. Some changes required by Python 3 need discussions and to make 
technical choices.  It's more convenient to work on smaller patches.


Q: "How much changes do we need to port Swift to Python ?"

A: Sorry, I don't know. Since we cannot run all tests on Python 3 right 
now, we cannot see all issues. It's really hard to estimate the number 
of required changes. Anyway, the plan is to port the code step by step.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Require documenting changes with versionadded and versionchanged

[Victor Stinner]

Le 15/10/2015 17:54, Joshua Harlow a écrit :

I had this problem with deprecation versioning (the debtcollector
library functions take a version="XYZ", removal_version="ABC" params,
see
http://docs.openstack.org/developer/debtcollector/examples.html#further-customizing-the-emitted-messages)
and it is pretty hard to get those two numbers right, especially with
weekly releases and not knowing when a review will merge... I'm not
saying we shouldn't try to do this, but we just have to figure out how
to do it in a smart way.


I hope that we will not release *too* frequently. Oslo libraries are 
supposed to be somehow "stable" :-) Past history showed that any minor 
change has major impact on the OpenStack CI ;-)


If we have to choose between "documenting changes with the wrong version 
number" and "don't document changes", I pick the wrong version. IHMO 
it's better than doc at all.


As I replied to Brant, I don't expect so much issues between doc and 
version numbers.


In a few weeks, if we consider that we have too many issues between doc 
and versions, it would maybe be time to rethink how we release Oslo 
libraries? Maybe our release process should be enhanced to review more 
carefully changes before a release?


By the way, Sphinx has a great tool to extract *all* versionchanged and 
versionadded. It's super useful to prepare a change log for humans. But 
it can also help to check the documentation of changes before a release, 
maybe manual check at the beginning?


The most basic check is to compute the diff since the previous release 
and check manually all versionadded and versionchanged.



Perhaps there need to be a gerrit based way to add these, so for example
a review submitter would write:

".. versionadded:: $FILL_ME_IN_WHEN_MERGED" and ".. versionchanged::
$FILL_ME_IN_WHEN_MERGED" or something, and gerrit when merging code
would change those into real numbers...


Hey, it remembers me CVS :-D It may work, but such tool has to be 
written first ;-) It may also make the release process more complex. 
Right now, it's quite simple: pick any Git SHA1 and this hash becomes 
your release. No *any* change between this revision and the final release.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] Require documenting changes with versionadded and versionchanged

[Victor Stinner]

Hi,

I propose that changes must now be documented in Oslo libraries. If a 
change is not documented, it must *not* be approved.


IMHO it's very important to document all changes. Otherwise, it becomes 
really hard to guess if a specific parameter or a specific function can 
be used just by reading the doc :-/ And we should not force users to 
always upgrading the Oslo libraries to the latest versions. It doesn't 
work on stable branches :-)


Currently, ".. versionadded:: x.y" and ".. versionchanged:: x.y" are not 
(widely) used in Oslo libraries. A good start would be to dig the Git 
history to add these tags. I started to do this for the oslo.config library:

https://review.openstack.org/#/c/235232/

I'm interested to write similar changes for other Oslo libraries.

Because I created this change, I started to vote -1 on all patches which 
oslo.config changes the API without documenting the doc.


What do you think?

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Require documenting changes with versionadded and versionchanged

[Victor Stinner]

Le 15/10/2015 12:52, Victor Stinner a écrit :

I started to do this for the oslo.config library:
https://review.openstack.org/#/c/235232/


More changes.

oslo.concurrency:
https://review.openstack.org/#/c/235416/

oslo.serialization:
https://review.openstack.org/#/c/235297/

oslo.utils:
https://review.openstack.org/#/c/235386/

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Require documenting changes with versionadded and versionchanged

[Victor Stinner]

Le 15/10/2015 16:34, Brant Knudson a écrit :

Submitters don't know what release their change is going to get into.
They might submit the review when version 1.1.0 is current so they mark
it with added in 1.2.0, but then the change doesn't get merged until
after 1.4.0 is tagged. Also, the submitter doesn't know what the next
release is going to be tagged as, since it might be 1.2.0 or it might be
2.0.0.


I propose to expect that the next release is X.Y+1 where X.Y is the 
current release. If the release manager wants to increase the major 
version number, it's very easy to fix all documentation at once. Example 
with sed in oslo.concurrency to replace 2.7 with 3.0:


   sed -i -e 's/\(versionadded\|versionchanged\):: 2.7/\1:: 3.0/g' \
  $(find oslo_concurrency/ -name "*.py")


So this will create extra churn as reviews have to be updated with the
release #, and the docs will likely be wrong when reviewers forget to
check it (unless this can be automated).


If a the version X.Y+1 is released before the change is reviewed, the 
reviewer is responsible to complain about outdated 
versionadded/versionchanged markups.


I know that it adds more work, but I consider that it's worth it.


I don't think it's worth it to document in which release something is
added in the oslo libs. We're not charging for this stuff so if you want
the online docs to match your code use the latest. Or check out the tag
and generate the docs for the release you're on to look at to see if the
feature is there.


It's maybe time to think outside OpenStack. Oslo libraries can be used 
outside OpenStack and API stability and documenting changes matters even 
more outside OpenStack.


I wrote 4 patches to document all changes of 4 Oslo Libraries 
(concurrency, config, serialization, utils). See my patches, they are 
quite small. I took me 2 hours to dig the whole Git history since the 
creation of each library. Only some patches changes the API. For 
example, bug fixes don't have to be documented.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] gates broken by WebOb 1.5 release

[Victor Stinner]

Hi,

WebOb 1.5 was released yesterday. test_misc of Cinder starts failing 
with this release. I wrote this simple fix which should be enough to 
repair it:


https://review.openstack.org/233528
"Fix test_misc for WebOb 1.5"

 class ConvertedException(webob.exc.WSGIHTTPException):
-def __init__(self, code=0, title="", explanation=""):
+def __init__(self, code=500, title="", explanation=""):

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] gates broken by WebOb 1.5 release

[Victor Stinner]

Le 12/10/2015 10:43, Victor Stinner a écrit :

WebOb 1.5 was released yesterday. test_misc of Cinder starts failing
with this release. I wrote this simple fix which should be enough to
repair it:

https://review.openstack.org/233528
"Fix test_misc for WebOb 1.5"


FYI my fix was merged in Cinder. You may have to recheck your changes.

The bug report: https://bugs.launchpad.net/bugs/1505153

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [swift] Plan to port Swift to Python 3

[Victor Stinner]

Hi,

Le 09/10/2015 12:12, vishal yadav a écrit :

However I was just checking if you considered using 2to3. I can
understand that translation using this tool might not cover every area
in the code more specifically custom/3rd party libraries (non-standard
python libraries) but IMO it can do fixer translations to much extent.


I tried 2to3, modernize and 2to6 tools in the past, but they produce a 
single giant patch with unwanted changes. These tools are written to add 
compatibility for all Python versions including Python 2.6 and Python 
3.0. In OpenStack, we only care of Python 2.7 and 3.4, so the code can 
be simpler. For example, we can simply write u"unicode" instead of 
six.b("unicode").


I wrote the sixer tool for OpenStack. Basically, it's the same than 
2to6, except that:


- sixer respects OpenStack Coding Style on imports: it groups and sorts 
imports. It avoids to have to manually fix individual modified import 
which takes a lot of time


- sixer can produce a patch for a single pattern. For example, replace 
all unicode with six.text_type but nothing else. Since all changes are 
reviewed carefully in OpenStack, it's important to produce "reviewable" 
(small) changes.


See also my blog article which explains the full rationale:
http://haypo.github.io/python3-sixer.html

My patch serie of 6 changes to fix most Python 3 issues was almost fully 
generated by sixer. Sometimes, I had to manually fix a few lines because 
no tool is perfect ;-) The patch serie:


* "py3: Replace unicode with six.text_type"
  https://review.openstack.org/#/c/232476/

* "py3: Replace urllib imports with six.moves.urllib"
  https://review.openstack.org/#/c/232536/

* "py3: Use six.reraise() to reraise an exception"
  https://review.openstack.org/#/c/232537/

* "py3: Replace gen.next() with next(gen)"
  https://review.openstack.org/#/c/232538/

* "Replace itertools.ifilter with six.moves.filter"
  https://review.openstack.org/#/c/232539/

* "py3: Replace basestring with six.string_types"
  https://review.openstack.org/#/c/232540/

Then I will then use sixer on individual files to fix all Python 3 at once.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [swift] Plan to port Swift to Python 3

[Victor Stinner]

Hi,

Good news, we made good progress last weeks on porting Swift to Python 
3, a few changes were merged and all dependencies now work on Python 3. 
We only need two more simple changes to have a working pyhon34 check job:


* "py3: Update pbr and dnspython requirements"
  https://review.openstack.org/#/c/217423/
* "py3: Add py34 test environment to tox"
  https://review.openstack.org/#/c/199034/

With these changes, it will be possible to make the python34 check job 
voting to avoid Python 3 regressions. It's very important to avoid 
regressions, so we cannot go backward again in Python 3 support.


On IRC, it was said that it's better to merge Python 3 changes at the 
beginning of the Mitaka cycle, because Python 3 requires a lot of small 
changes which can likely introduce (subtle) bugs, and it's better to 
catch them early during the development cycle.


John Dickinson prefers incremental and small changes, whereas clayg 
looks to like giant patches to fix all Python 3 issues at once to avoid 
conflicts in other (non-Python3) changes. (Sorry, if I didn't summarized 
correctly the discussion we had yesterday.)


The problem is that it's hard to fix "all" Python 3 issues in a single 
patch, the patch would be super giant and just impossible to review. 
It's also annoying to have to write dozens of small patches: we loose 
time on merge conflicts, rebasing, random gate failures, etc.


I proposed a first patch serie of 6 changes to fix a lot of simple 
Python 3 issues "at once":


* "py3: Replace unicode with six.text_type"
  https://review.openstack.org/#/c/232476/

* "py3: Replace urllib imports with six.moves.urllib"
  https://review.openstack.org/#/c/232536/

* "py3: Use six.reraise() to reraise an exception"
  https://review.openstack.org/#/c/232537/

* "py3: Replace gen.next() with next(gen)"
  https://review.openstack.org/#/c/232538/

* "Replace itertools.ifilter with six.moves.filter"
  https://review.openstack.org/#/c/232539/

* "py3: Replace basestring with six.string_types"
  https://review.openstack.org/#/c/232540/

The overall diff is impressive: "61 files changed, 233 insertions(+), 
189 deletions(-)" ... but each change is quite simple. It's only one 
pattern replaced with a different pattern. For example, replace 
"unicode" with "six.text_type" (and add "import six" if needed). So 
these changes should be easy to review.


With a working (and voting?) python34 check job and these 6 changes, it 
will be (much) easier to work on porting Swift to Python 3. Following 
patches will be validated by the python34 check job, shorter and 
restricted to a few files.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Process to clean up the review queue from non-active patches

[Victor Stinner]

Hi,

Le 06/10/2015 16:36, Flavio Percoco a écrit :

Not so long ago, Erno started a thread[0] in this list to discuss the
abandon policies for patches that haven't been updated in Glance.
(...)
1) Lets do this on patches that haven't had any activity in the last 2
months. This adds one more month to Erno's proposal. The reason being
that during the lat cycle, there were some ups and downs in the review
flow that caused some patches to get stuck.


Please don't do that. I sent a patch in June (20) and it was only 
reviewed in October (4)... There was no activity simply because I had 
nothing to add, everything was explained in the commit message, I was 
only waiting for a review...


I came on #openstack-glance to ask for review several time between 
August and September but nobody reviewed by patches (there was al.


Example of patch: https://review.openstack.org/#/c/193786/ (now merged)

It would be very frustrating to have to resend the same patch over and over.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [pecan] [mistral] Pecan python3 compatibility

[Victor Stinner]

Hi,

Le 01/10/2015 15:50, Nikolay Makhotkin a écrit :

In Mistral, we are trying to fix the codebase for supporting python3,
but we are not able to do this since we faced issue with pecan library.
If you want to see the details, see this traceback - [1].


I tried to run Mistral tests on Python 3 and I got the same error. For 
me, it's an obvious Python 3 bug in Pecan. Maybe Pecan has only a 
partial Python 3 support?


If value is a bounded method, you can replace value.im_class with 
type(value.__self__).


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] nominating Brant Knudson for Oslo core

[Victor Stinner]

+1 for Brant

Victor

Le 24/09/2015 19:12, Doug Hellmann a écrit :

Oslo team,

I am nominating Brant Knudson for Oslo core.

As liaison from the Keystone team Brant has participated in meetings,
summit sessions, and other discussions at a level higher than some
of our own core team members.  He is already core on oslo.policy
and oslo.cache, and given his track record I am confident that he would
make a good addition to the team.

Please indicate your opinion by responding with +1/-1 as usual.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] -1 due to line length violation in commit messages

[Victor Stinner]

Le 25/09/2015 17:00, Julien Danjou a écrit :

If we wanted to enforce that, we would just have to write a bot setting
-1 automatically. I'm getting tired of seeing people doing bots' jobs in
Gerrit.


It may also help to enhance "git review -s" to install a local 
post-commit hook to warn if the commit message doesn't respect OpenStack 
coding style.


(Like the evil "final point", haha. I don't recall if it was mandatory 
or illegal.)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] How to link a change to the completed cinder-python3 blueprint?

[Victor Stinner]

Hi,

I'm porting Cinder to Python 3. There are discussions on my patches to 
decide how to link my changes to the cinder-python3 blueprint (syntax of 
the "Blueprint cinder-python3" line).


Mike Perez completed the blueprint on 2015-08-15. I don't understand 
why, the port is incomplete. I'm still writing patches to continue the port.


Because the blueprint is completed, launchpad is unable to find the 
blueprint when you click on the blueprint line from gerrit. Some people 
asked me to write the URL to the blueprint instead. The problem is that 
"git review" uses the topic "bp/https" instead of "bp/cinder-python3" in 
this case. I have to specify manually the topic ("git review -t 
bp/cinder-python3). The URL may also change in the future, using a name 
is more future proof.


Another question is how to mention the blueprint in the commit message. 
I already wrote 35 changes which were merged into Cinder using the 
syntax "Blueprint cinder-python3". But some people are now asking me to 
use the syntax "Implements: blueprint cinder-python3", "Partially 
implements: blueprint cinder-python3", "Implements: blueprint " or 
something else.


I suggest to continue to use "Blueprint cinder-python3", and maybe 
change the status of the blueprint to be able to find it again in launchpad.



cinder-python3 blueprint:
https://blueprints.launchpad.net/cinder/+spec/cinder-python3

My Python 3 patches for Cinder:
https://review.openstack.org/#/q/status:open+project:openstack/cinder+branch:master+topic:bp/cinder-python3,n,z


Note: I wrote an email because people started to vote -1 on my changes 
because of the syntax of the "blueprint" line in my commit message.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [packaging][all] setting minimum version of setuptools in setup.py

[Victor Stinner]

Le 29/07/2015 10:27, Robert Collins a écrit :

Similar to pbr, we have a minimum version of setuptools required to
consistently install things in OpenStack. Right now thats 17.1.

However, we don't declare a setup_requires version for it.

I think we should.


If it's possible to explicit the minimum version of setuptools required 
to install an application in setup.py, yes, we should do that. I like 
being explicit to avoid bad surprises.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [packaging][all] setting minimum version of setuptools in setup.py

[Victor Stinner]

Le 29/07/2015 18:12, Clay Gerrard a écrit :

I agree an error message is better than breaking for insane reasons.

But... maybe as an aside... what about not breaking?


Well, yes, but *who* wants to do that?

Old versions of setuptoolsl, pip and pbr have a lot of issues. It will 
be a nightmare to workaround them.


For example, it was really difficult to handle dependencies which depend 
on the Python version (for python = 2.6, for python = 3.0, etc.). 
Environment markers solve a real concrete issue. Trying to working 
around environment markers is like reinventing the wheel, it doesn't 
sound like a good idea to me.


For me, it's a better investment to work upstream on setuptools, pip and 
pbr, and require recent versions.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Python 3: 5 more projects with a py34 voting gate, only 4 remaing

[Victor Stinner]

Hi,

On 27/07/2015 12:35, Roman Vasilets wrote:

Hi, just what to share with you. Rally project also have voting py34
jobs. Thank you.


Cool! I don't know if Rally port the Python 3 is complete or not, so I 
wrote work in progress. Please update the wiki page if the port is done:

https://wiki.openstack.org/wiki/Python3#OpenStack_applications

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Python 3: 5 more projects with a py34 voting gate, only 4 remaing

[Victor Stinner]

Hi,

Could you please modify the wiki page yourself to add Designate? I don't want 
to be the only one maintaining this wiki page ;-)

https://wiki.openstack.org/wiki/Python3

Victor

- Original Message -
 From: Kiall Mac Innes ki...@macinnes.ie
 To: openstack-dev@lists.openstack.org
 Sent: Tuesday, July 21, 2015 12:51:03 PM
 Subject: Re: [openstack-dev] Python 3: 5 more projects with a py34 voting 
 gate, only 4 remaing
 
 You can (nearly) add Designate to this list :)
 
 Pradeep has been doing a great job getting the codebase py3 compatible!
 
 Thanks,
 Kiall
 
 On 17/07/15 12:32, Victor Stinner wrote:
  Hi,
  
  We are close to having a voting py34 gate on all OpenStack libraries and
  applications. I just made the py34 gate voting for the 5 following
  projects:
  
  * keystone
  * heat
  * glance_store: Glance library (py34 is already voting in Glance)
  * os-brick: Cinder library (py34 is already voting in Cinder)
  * sqlalchemy-migrate
  
  
  A voting py34 gate means that we cannot reintroduce Python 3 regressions
  in the code tested by tox -e py34. Currently, only a small subset of
  test suites is executed on Python 3.4, but the subset is growing
  constantly and it already helps to detect various kinds of Python 3 issues.
  
  Sirushti Murugesan (who is porting Heat to Python 3) and me proposed a
  talk Python 3 is coming! to the next OpenStack Summit at Tokyo. We
  will explain the plan to port OpenStack to Python in depth.
  
  
  There are only 4 remaining projects without py34 voting gate:
  
  (1) swift: I sent patches, see the Fix tox -e py34 patch:
  
  https://review.openstack.org/#/q/project:openstack/swift+branch:master+topic:py3,n,z
  
  
  
  (2) horizon: I sent patches:
  
  https://review.openstack.org/#/q/topic:bp/porting-python3+project:openstack/horizon,n,z
  
  
  
  (3) keystonemiddleware: blocked by python-memcached, I sent a pull
  request 3 months ago and I'm still waiting...
  
  https://github.com/linsomniac/python-memcached/pull/67
  
  I may fork the project if the maintainer never reply. Read the current
  thread [all] Non-responsive upstream libraries (python34 specifically)
  on openstack-dev.
  
  
  (4) python-savannaaclient: We haven't enough tests to ensure that
  savanna client works correctly on py33, so, it's kind of premature step.
  We already have py33 and pypy jobs in experimental pipeline. This
  client can be ported later.
  
  
  Note: The py34 gate of oslo.messaging is currently non voting because of
  a bug in Python 3.4.0, fix not backported to Ubuntu Trusty LTS yet:
  
  https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1367907
  
  The bug was fixed in Python 3.4 in May 2014 and was reported to Ubuntu
  in September 2014.
  
  Victor
  
  __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Python 3: 5 more projects with a py34 voting gate, only 4 remaing

[Victor Stinner]

Hi,

We are close to having a voting py34 gate on all OpenStack libraries and 
applications. I just made the py34 gate voting for the 5 following projects:


* keystone
* heat
* glance_store: Glance library (py34 is already voting in Glance)
* os-brick: Cinder library (py34 is already voting in Cinder)
* sqlalchemy-migrate


A voting py34 gate means that we cannot reintroduce Python 3 regressions 
in the code tested by tox -e py34. Currently, only a small subset of 
test suites is executed on Python 3.4, but the subset is growing 
constantly and it already helps to detect various kinds of Python 3 issues.


Sirushti Murugesan (who is porting Heat to Python 3) and me proposed a 
talk Python 3 is coming! to the next OpenStack Summit at Tokyo. We 
will explain the plan to port OpenStack to Python in depth.



There are only 4 remaining projects without py34 voting gate:

(1) swift: I sent patches, see the Fix tox -e py34 patch:

https://review.openstack.org/#/q/project:openstack/swift+branch:master+topic:py3,n,z


(2) horizon: I sent patches:

https://review.openstack.org/#/q/topic:bp/porting-python3+project:openstack/horizon,n,z


(3) keystonemiddleware: blocked by python-memcached, I sent a pull 
request 3 months ago and I'm still waiting...


https://github.com/linsomniac/python-memcached/pull/67

I may fork the project if the maintainer never reply. Read the current 
thread [all] Non-responsive upstream libraries (python34 specifically) 
on openstack-dev.



(4) python-savannaaclient: We haven't enough tests to ensure that 
savanna client works correctly on py33, so, it's kind of premature step. 
We already have py33 and pypy jobs in experimental pipeline. This 
client can be ported later.



Note: The py34 gate of oslo.messaging is currently non voting because of 
a bug in Python 3.4.0, fix not backported to Ubuntu Trusty LTS yet:


https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1367907

The bug was fixed in Python 3.4 in May 2014 and was reported to Ubuntu 
in September 2014.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Non-responsive upstream libraries (python34 specifically)

[Victor Stinner]

Le 16/07/2015 22:28, Thomas Goirand a écrit :

IMO, we should, for the Liberty release, get rid of:
- suds  suds-jurko


suds was replaced with suds-jurko (in oslo.vmware, cinder, nova): I 
kicked suds off of global requirements.



- memcached (in the favor of pymemcache)


As I wrote, I may fork python-memcached. It's just not in my priority 
list right now.


FYI some colleagues just forked python-ldap in 
https://github.com/pyldap/pyldap/ to add Python 3 support. Stay tuned ;-)



- mysqldb (this has been discussed at large already)


MySQL-Python was replaced with PyMySQL in almost all projects, no? Even 
nova is now using PyMySQL ;-)



- cliff-tablib and tablib (not ported to Py3, used only for testing)


tablib works on Python 3, but it has a strange way of using 
dependencies. You can try to remove tablib/packages/markup.py when 
building the package for Python 3 and it should just work.


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Non-responsive upstream libraries (python34 specifically)

[Victor Stinner]

Hi,

Le 16/07/2015 17:00, Davanum Srinivas a écrit :

I ended up here:
https://github.com/linsomniac/python-memcached/issues/54
https://github.com/linsomniac/python-memcached/pull/67


Oh, that's my pull request :-) Multiple peoples asked to merge my pull 
request, and I pinged explicitly the maintainer without _any_ kind of 
feedback. He's away or just don't care anymore since april (2015).


For me, they are two options:

* Fork python-memcached, but keep the same Python module name. Similar 
approach than suds/suds-jurko and pam/pam3


* Switch to pymemcache which is already compatible with Python 3

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] Need a new release of os-brick for Python 3

[Victor Stinner]

Hi,

The latest release of os-brick is not compatible with Python 3. 
Different patches were merged to fix Python 3 support. tox -e py34 now 
executes all tests and all tests pass on Python 3.4.


I need a release of os-brick to port Cinder to Python 3. A Python 3 
issue in os-brick blocks my 4 pending Cinder patches for Python 3.


Tell me if I can help to get this release.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] Progress of the Python 3 port

[Victor Stinner]

Hi,

On 09/07/2015 16:29, Ian Cordasco wrote:

Thanks for your hard work Victor! I'll bring up a new release today in the
Glance meeting (currently running) and see if Nik and I can bug the
release managers for a new release today.


Any update on this release?

In the meeting report, I read:
14:39:59 jokke_ also queued up bunch of stable stuff ... glance_store 
gotta wait until Cinder gets their client requirements fixed, but 
glanceclient backports would need some love


What is this Cinder client requirements issue? Is it fixed now? The 
stable stuff are required for the new glance_store release?


http://eavesdrop.openstack.org/meetings/glance/2015/glance.2015-07-09-14.00.log.html

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Time to remove Python2.6 jobs from master

[Victor Stinner]

On 13/07/2015 13:57, Jeremy Stanley wrote:

people on 2.6-only platforms who want to install and run
python-novaclient from PyPI can use the stable/juno version (though
I'll admit that finding which version works with 2.6 may be a tricky
proposition for a consumer who is unaware of this situation).


Hopefully, some Linux distro package python-novaclient and so user don't 
have to guess the version compatible with their OS.


https://packages.debian.org/jessie/python-novaclient
http://packages.ubuntu.com/fr/precise/python/python-novaclient
https://admin.fedoraproject.org/pkgdb/package/python-novaclient/
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/6/html/Command-Line_Interface_Reference_Guide/install_clients.html
etc.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tests][swift] Fix it friday! [mock failure in CI] - fix for Swift

[Victor Stinner]

Here is a fix for Swift:
https://review.openstack.org/200474

Victor

On 10/07/2015 09:45, Robert Collins wrote:

Good news everybody, mock 1.1.0 is now out. This backports all the
improvements over the last couple of years, making it fully
synchronised with cPython master. Yay.

Bad news. Lots of unit tests jobs have suffered falled from this.

But - none of the things I've looked into so far are bugs in mock 1.1.0.

One of the improvements in mock is to fail when a bad method is called.

Consider this: 
https://review.openstack.org/#/c/200384/1/taskflow/tests/unit/test_engine_helpers.py

Note the old method: mock_import.assert_called_onec_with(name)

That method never existed. onec is a typo :).

mock 1.0.1 silently accepts that - thats part of its job. But, its a
very fragile API.

1.1.0 makes that an error, for methods with assert prefixes - unless
unsafe is specifically requested. So a big chunk of the failing tests
are tests that were not testing anything *at all*.
One common exampled of that is 'assert_called' - another method that
never ever existed. All our tests using that were testing nothing at
all.


Neutron is failing on a bunch of tests that accessed a private
function inside mock. I'm surprised reviewers didn't spot this, but
_patch isn't part of the public API, and never was.

Tempest seems to be failing due to a different object being returned -
I haven't dug deep enough to describe the cause in more detail.

We can probably pin mock back to 1.0.1 locally within projects to gain
breathing room, but given the API improvements in 1.1.0 (see below[1]
as readthedocs is failing to build it due to having an old pip in
their virtualenvs :/) - I think we'll be much better off adopting it
as quickly as we can. NB: with this release we don't need to use
'unittest.mock' for Python3.4 - we can just standardise on using
'mock' across the board, which is much simpler and easier to reason
about (same codebase everywhere[2])

[2]: Python 2.6 support was dropped in 1.1.0, so we need to use
markers to select 1.0.1 for the remaining 2.6 gate jobs. (We should
kill those of asap).
[1]:
- Issue #23310: Fix MagicMock's initializer to work with __methods__, just
   like configure_mock().  Patch by Kasia Jachim.

- Issue #23568: Add rdivmod support to MagicMock() objects.
   Patch by Håkan Lövdahl.

- Issue #23581: Add matmul support to MagicMock. Patch by Håkan Lövdahl.

- Issue #23326: Removed __ne__ implementations.  Since fixing default __ne__
   implementation in issue #21408 they are redundant. *** NOT BACKPORTED ***

- Issue #21270: We now override tuple methods in mock.call objects so that
   they can be used as normal call attributes.

- Issue #21256: Printout of keyword args should be in deterministic order in
   a mock function call. This will help to write better doctests.

- Issue #21262: New method assert_not_called for Mock.
   It raises AssertionError if the mock has been called.

- Issue #21238: New keyword argument `unsafe` to Mock. It raises
   `AttributeError` incase of an attribute startswith assert or assret.

- Issue #21239: patch.stopall() didn't work deterministically when the same
   name was patched more than once.

- Issue #21222: Passing name keyword argument to mock.create_autospec now
   works.

- Issue #17826: setting an iterable side_effect on a mock function created by
   create_autospec now works. Patch by Kushal Das.

- Issue #17826: setting an iterable side_effect on a mock function created by
   create_autospec now works. Patch by Kushal Das.

- Issue #20968: unittest.mock.MagicMock now supports division.
   Patch by Johannes Baiter.

- Issue #20189: unittest.mock now no longer assumes that any object for
   which it could get an inspect.Signature is a callable written in Python.
   Fix courtesy of Michael Foord.

- Issue #17467: add readline and readlines support to mock_open in
   unittest.mock.

- Issue #17015: When it has a spec, a Mock object now inspects its signature
   when matching calls, so that arguments can be matched positionally or
   by name.

- Issue #15323: improve failure message of Mock.assert_called_once_with

- Issue #14857: fix regression in references to PEP 3135 implicit __class__
   closure variable (Reopens issue #12370)

- Issue #14295: Add unittest.mock


-Rob





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tests] Fix it friday! [mock failure in CI]

[Victor Stinner]

On 10/07/2015 10:42, Robert Collins wrote:

Releasing this on a Friday sounds like bad timing, especially without
advance notice that we'd have to rush to fix the dozens of new issues it
would expose.


There would never be a good time to release such things. The whole
point of the constraints system we're rolling out is to make us not
sensitive to the release schedule of upstreams: we can't dictate the
release schedule of the world to be convenient to our cycles.


I see mock===1.0.1 in upper-constraints.txt, but the python27 check 
job of Swift was broken by the release of mock 1.1.


Can someone please explain me why Swift check job failed?

Patch were I noticed the regression of mock 1.1, on the patch set 6:
https://review.openstack.org/#/c/199034/

(test_wsgi started to fail because the attribute assert_called doesn't 
exist.)


My fix for mock 1.1 in Swift:
https://review.openstack.org/#/c/200474/

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tests] Fix it friday! [mock failure in CI]

[Victor Stinner]

Le 10/07/2015 13:39, Jeremy Stanley a écrit :

On 2015-07-10 13:21:56 +0200 (+0200), Victor Stinner wrote:

I see mock===1.0.1 in upper-constraints.txt, but the python27
check job of Swift was broken by the release of mock 1.1.

Can someone please explain me why Swift check job failed?


As far as I'm aware, only DevStack is making use of the constraints
file currently. Unit tests are still relying on pip to install
working versions of packages from the requirements.txt and
test-requirements.txt within each repo.


Is there a plan to use pinned versions on other gates to avoid similar 
issues in the future? (Decide when we upgrade a dependency)


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [swift] Stategy to port Swift to Python 3

[Victor Stinner]

Hi,

I have 9 pending patches to fix Python 3 issues in Swift, but they 
didn't get much attention yet. Most of these patches replace a pattern 
with a new pattern to add Python 3 support in addition to Python 2 support.


https://review.openstack.org/#/q/owner:%22Victor+Stinner%22+status:open+project:openstack/swift,n,z

The problem is that these patches are long, and so it's common to get 
conflicts. It takes me a lot of time just to rebase these patches.

Only Replace dict.iteritems() with dict.items() got a +2 yet.

I hesitate to simply give up on porting Swift to Python 3, to focus on 
other projects which are faster to review my Python 3 patches 
(ceilometer, cinder, glance, keystone, nova).


Maybe I took the wrong strategy for Swift. Instead of replacing a 
pattern in the whole Swift project, I should maybe try to port tests one 
by one to have shorter patches?


My last try fix tox -e py34 in a single patch:
https://review.openstack.org/#/c/199034/

Practical issue: it depends on my pyeclib pull request that I sent 3 
months ago...


If this Swift Fix tox -e py34 patch is merged, my pyeclib pull request 
is merged, and a new version of pyeclib including my fix is released, it 
will become possible to make the gate-swift-python34 voting to avoid 
Python 3 regressions. It should be nice milestone to start with shorter 
patches.


What do you think?

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] gate is wedged

[Victor Stinner]

Hi,

Le 30/06/2015 05:49, Matt Riedemann a écrit :

Alternatively, oslo.versionedobjects 0.5.1 is cut after
https://review.openstack.org/#/c/196926/ is merged and then we just need
haypo's test_db_api fix for the oslo.db 2.0.0 change:

https://review.openstack.org/#/c/196719/


I updated my patch Update test_db_api for oslo.db 2.0 (1) to not 
depend on my Fix Python 3 issues in nova.db.sqlalchemy patch. I should 
now be easier to fix gates.


(1) https://review.openstack.org/#/c/196719/
(2) https://review.openstack.org/#/c/195191/

I suggest to block my Python 3 patch (2) until gates are fixed.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] gate is wedged

[Victor Stinner]

Le 30/06/2015 10:32, Victor Stinner a écrit :

I updated my patch Update test_db_api for oslo.db 2.0 (1) ...

(1) https://review.openstack.org/#/c/196719/


I updated my patch again to also block oslo.versionedobjects 0.5.0 in 
requirements. So we can have two patches to fix the bug ;) The patch (1) 
only fixes the bug, whereas the Python 3 patch fixes many other things 
(Python 3 support, remove test-requirements-py3.txt, etc.).


At the same time, we have issues on the OpenStack infra :-/

10:48 -!- ChanServ changed the topic of #openstack-infra to: OpenStack 
CI is down due to hard drive failures


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] Ideas to detect Oslo regressions earlier

[Victor Stinner]

Hi,

Unfortunately, it looks like each regressions introduced by releases of 
Oslo libraries are still common :-( We already have tools to detect 
regressions, but they are run manually. It would be nice to automate 
these tests and run them more frequently (at least one per week, or even 
daily?).


There are tools to run unit tests of projects like neutronclient or nova 
on the development version of oslo.* libraries. They take up to 12 hours 
to run all tests. Example of tools:


- tox -e venv -- oslo_run_pre_release_tests in each Oslo project
- tools/oslo_run_cross_tests in oslotest

To prepare the latest bunch of releases, dims wrote two patches to run 
nova unit tests and tempest:


* https://review.openstack.org/#/c/186413/
* https://review.openstack.org/#/c/186418/

Unfortunately, the stable version of some oslo.* projects were used 
instead of the development version, and 2 regressions were missed :-/


It would nice to automate a job running cinder, nova and neutron unit 
tests and tempest on the development versions (git master branch) of all 
oslo.* projects. We can start with something simple: run 
tools/oslo_run_cross_tests and send the result by email every day to 
some people interested by the result (ex: Oslo liaisons, or just me if 
nobody cares). It would be nice to have a dedicated resource in the 
OpenStack infra for such job.


I proposed to run all tests using Gerrit for each patch send to an 
Oslo project, but it would use too much resources of the OpenStack infra.


Another idea would be to write a check job dedicated to Oslo releases 
and use Gerrit to prepare a release (at least to tag a version in Git).


Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Let's get rid of tablib and cliff-tablib

[Victor Stinner]

Hi,

Le 29/06/2015 11:03, Thomas Goirand a écrit :

cliff-tablib is used for the unit tests of things like
python-neutronclient. The annoying bit is that cliff-tablib depends on
tablib, which itself is a huge mess. It has loads of 3rd party embedded
packages and most of them aren't Python 3.x compatible.


tablib includes copies of various dependencies in its tablib/packages/ 
directory. Some of them are for Python 2, others are for Python 3.


It would be better to use dependencies (requirements in setup.py), not 
copies. Do you try to contact tablib authors to ask them to remove 
completly tablib/packages/?


setup.py uses a different list of packages on Python 2 and Python 3.

I tried python3 setup.py install: the bytecode compilation of 
markup.py fails with an obvious SyntaxError, the code is for Python 2. 
But there is also markup3.py which is compiled successfully.


Even if the compilation of the markup.py fails, python setup.py 
install succeed with the exit code 0. What is your problem?


setup.py should be fixed to skip markup.py on Python 3, and skip 
markup3.py on Python 2. A workaround is to remove manually the file 
depending on the Python major version.


Note: pip install tablib works on Python 3 (pip uses the binary wheel 
package).




I've seen that for python-openstackclient, recently, cliff-tablib was
added. Let's do the reverse, and remove cliff-tablib whenever possible.

If we really want to keep using cliff-tablib, then someone has to do the
work to port tablib to Python3 (good luck with that...).


cliff-tablib is used in tests. If you remove the cliff-tablib 
dependency, tests will obviously fail.


What do you propose? Modify tests to reimplement cliff-tablib? Remove tests?

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder][glance][nova] Python 3 coming!

[Victor Stinner]

Hi,

The Python 3.4 gate (py34) of cinder, glance and nova just became 
voting. You now have to write Python code compatible with Python 2.7 and 
3.4.


If the py34 check job fails on your patch, you may try to rebase it to 
get the new tox.ini and updated requirements.


You can find information on Python 3 in OpenStack in the following wiki 
page:


   https://wiki.openstack.org/wiki/Python3

If you have issues with Python 3 or the py34 gates, reply to this email 
or contact me privately.


The Cinder blueprint and Nova spec for Python 3 explain the overall plan 
to add Python 3 support to these projects. I have the same plan for Glance.


   https://blueprints.launchpad.net/cinder/+spec/cinder-python3


http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/adding-python34-support-to-nova.html

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Jenkins check failed on gate-glance_store-python34

[Victor Stinner]

Hi,

Le 26/06/2015 09:17, 孙明达 a écrit :

I am a new contributor for openstack.
the following url is my code review for glance_store

 https://review.openstack.org/#/c/193403/

Jenkins check failed on gate-glance_store-python34
Referring to the console info, I have never changed the code mentioned.


Please rebase your patch to get the new tox.ini. py34 should pass with 
the new tox.ini.


The py34 gate of glance just became voting!

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [glance] Progress of the Python 3 port

[Victor Stinner]

Hi,

We made great progress on porting Glance to Python 3. tox -e py34 now 
pass with a subset of tests and there is a non-voting py34 check job. I 
propose to make the py34 gate voting to avoid Python 3 regressions:


https://review.openstack.org/194048
Make gate-glance-python34 voting

When the py34 will become voting, it will no more possible to introduce 
code incompatible with Python 3, at least in the code tested by tox -e py34.




Currently, py34 uses the development branch of glance_store because 
there is no release including my Python 3 fixes. For glance_store, it 
would be nice if someone can review my last patches:


https://review.openstack.org/#/q/status:open+project:openstack/glance_store+branch:master+topic:py3,n,z

With these 3 patches to port drivers, it becomes to possible to run all 
tests in tox -e py34, not only a subset of tests. So glance_store will 
be fully compatible with Python 3. My current patch tox -e py34: use a 
subset of working tests may be updated to run all tests, not only a 
subset, when other py3 patches will be merged.




Glance currently only uses PyMySQL on Python 3. For Python 2, there is a 
pending patch. It was blocked by oslo.db which didn't use PyMySQL. It 
changed: oslo.db now also uses PyMySQL driver by default for MySQL (but 
there is no release yet including this change). Glance patch:


https://review.openstack.org/#/c/184373/
Switch from MySQL-python to PyMySQL

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Debian already using Python 3.5: please gate on that

[Victor Stinner]

Hi,

Le 18/06/2015 15:44, Thomas Goirand a écrit :

As per the subject line, we already have Python 3.5 in Debian (AFAICT,
from Debian Experimental, in version beta 2). As a consequence, we're
already running (unit) tests using Python 3.5. Some have failures: I
could see issues in ceilometerclient, keystoneclient, glanceclient and
more (yes, I am planning to report these issues, and we already started
doing so).


Can you please share the list of issues? I may take a look.

Python 3.5 is not supposed to break backward compatibility.

Sorry, I don't remember which one, but I remember that a project is 
running tests using the default (development) branch of CPython, and it 
already helped us (CPython) to fix issues before the final release.


I don't think that OpenStack has resources (CPU and manpower) to run 
OpenStack tests on top of CPython default, nor on a beta version.


But as I wrote, I'm interested to take a look at issues with Python 3.5 ;-)

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][python3] use of six.iteritems()

[Victor Stinner]

Hi,

Le 10/06/2015 02:15, Robert Collins a écrit :

python2.7 -m timeit -s 'd=dict(enumerate(range(100)))' 'for i in
d.items(): pass'
10 loops, best of 3: 76.6 msec per loop



python2.7 -m timeit -s 'd=dict(enumerate(range(100)))' 'for i in
d.iteritems(): pass'
100 loops, best of 3: 22.6 msec per loop


.items() is 3x as slow as .iteritems(). Hum, I don't have the same 
results. Try attached benchmark. I'm using my own wrapper on top of 
timeit, because timeit is bad at calibrating the benchmark :-/ timeit 
gives unreliable results.


Results on with CPU model: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz:

[ 10 keys ]
713 ns: iteritems
922 ns (+29%): items

[ 10^3 keys ]
42.1 us: iteritems
59.4 us (+41%): items


[ 10^6 keys (1 million) ]
89.3 ms: iteritems
442 ms (+395%): items

In my benchmark, .items() is 5x as slow as .iteritems(). The code to 
iterate on 1 million items takes almost an half second. IMO adding 300 
ms to each request is not negligible on an application. If this delay is 
added multiple times (multiple loops iterating on 1 million items), we 
may reach up to 1 second on an user request :-/


Anyway, when I write patches to port a project to Python 3, I don't want 
to touch *anything* to Python 2. The API, the performances, the 
behaviour, etc. must not change.


I don't want to be responsible of a slow down, and I don't feel able to 
estimate if replacing dict.iteritems() with dict.items() has a cost on a 
real application.


As Ihar wrote: it must be done in a separated patch, by developers 
knowning well the project.


Currently, most developers writing Python 3 patches are not heavily 
involved in each ported project.


There is also dict.itervalues(), not only dict.iteritems().

for key in dict.iterkeys() can simply be written for key in dict:.

There is also xrange() vs range(), the debate is similar:
https://review.openstack.org/#/c/185418/

For Python 3, I suggest to use from six.moves import range to get the 
Python 3 behaviour  on Python 2: range() always create an iterator, it 
doesn't create a temporary list. IMO it makes the code more readable 
because for i in xrange(n): becomes for i in range(n):. six is not 
written outside imports and range() is better than xrange() for 
developers starting to learn Python.


Victor

Micro-benchmark for the Python operation key in dict. Run it with:

./python.orig benchmark.py script bench_str.py --file=orig
./python.patched benchmark.py script bench_str.py --file=patched
./python.patched benchmark.py compare_to orig patched

Download benchmark.py from:

https://bitbucket.org/haypo/misc/raw/tip/python/benchmark.py

import gc

def consume_items(dico):
for key, value in dico.items():
pass


def consume_iteritems(dico):
for key, value in dico.iteritems():
pass


def run_benchmark(bench):
for nkeys in (10, 10**3, 10**6):
bench.start_group('%s keys' % nkeys)
dico = {str(index): index for index in range(nkeys)}

bench.compare_functions(
('iteritems', consume_iteritems, dico),
('items', consume_items, dico),
)
dico = None
gc.collect()
gc.collect()

if __name__ == __main__:
import benchmark
benchmark.main()
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Adopt mox3

[Victor Stinner]

Hi,

Le 10/06/2015 22:17, Davanum Srinivas a écrit :

Oslo folks, everyone,

mox3 needs to be maintained since some of our projects use it and we
have it in our global requirements.

Here's the proposal from Doug - https://review.openstack.org/#/c/190330/


Why not only creating a project on Github? Do we need all OpenStack 
tools for mox3? I don't expect much enhancements in the library. For me, 
OpenStack looks more restrictive than a classic Github project, it's 
more heavy to contribute (sign a CLA, use review.openstack.org, etc.).


I prefer mock over mox, the API is very different. mock is now part of 
Python stdlib (unittest.mock since Python 3.3). In the past, I ported 
some mox tests to mock, but it takes a lot of time :-(


https://docs.python.org/dev/library/unittest.mock.html

Did anyone try to contact original mox authors? smiddlek is the last 
active contributor.


I see changes in 2012 in the Subversion repository of mox, whereas the 
latest mox release was in 2010 (mox 0.5.3).


It would be nice to merge back enhancements of mox3 (Python 3 support) 
into mox. It's annoying to have a specific project to get Python 3 support.


mox is hosted on code.google.com which is closing!

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] oslo releasing is noisy...

[Victor Stinner]

Le 09/06/2015 18:31, Thierry Carrez a écrit :

We are currently exploring the option to repurpose the
openstack-announce ML to be the extensive record of release
announcements. It's part of a larger plan to streamline library
releases, about which Doug should start a discussion pretty soon now.


In the Python community, there are Special Interest Groups (SIG) which 
have their mailing lists: distutils-sig, import-sig, mobile-sig, etc.


https://www.python.org/community/sigs/

Maybe we can move some traffic to new mailing lists, like a new 
openstack-oslo@ list, instead of using tags in the subject?


Anyone would be able to choose to subscribe or not to openstack-oslo.

Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [release][oslo] oslotest release 1.7.0 (liberty)

[Victor Stinner]

Good news: your fix is already merged into oslo.messaging ;-) 
oslo.messaging now uses directly mox3, on Python 2 and Python 3.


Victor

Le 10/06/2015 14:04, Victor Sergeyev a écrit :

Hi All,

this oslotest release break oslo.messaging gate - unittest fails
with ImportError, on import mox from six.moves - [0].
It caused by commit [1], which removed adding mox to six moves.

There is a fix for oslo.messaging - [2] - please, help to merge it.

[0] - http://paste.openstack.org/show/281091/
[1] -
https://github.com/openstack/oslotest/commit/9e0c8ad2c251274128499a7fcfb591c488d27d2b
[2] - https://review.openstack.org/190136

Thanks,
Victor

On Tue, Jun 9, 2015 at 6:14 PM, d...@doughellmann.com
mailto:d...@doughellmann.com wrote:

We are content to announce the release of:

oslotest 1.7.0: Oslo test framework

This release is part of the liberty release series.

With source available at:

http://git.openstack.org/cgit/openstack/oslotest

For more details, please see the git log history below and:

http://launchpad.net/oslotest/+milestone/1.7.0

Please report issues through launchpad:

http://bugs.launchpad.net/oslotest

Changes in oslotest 1.6.0..1.7.0


5bd9b31 Updated from global requirements
ff9b38e Fix argument handling in oslo_run_cross_tests
960e444 Add class to deal with clouds.yaml support
1c0863f Remove unneeded runtime pbr dep
5f2e635 Updated from global requirements
f8a5a3c Advertise support for Python3.4 / Remove support for Python 3.3
a063f25 Do not sync run_cross_tests.sh
0782ab7 Remove unused discover dependency
9e0c8ad Remove six.moves call

Diffstat (except docs and test files)
-

openstack-common.conf  |  7 --
oslotest/__init__.py   |  1 -
oslotest/functional.py | 59
++
oslotest/moxstubout.py |  2 +-
requirements.txt   |  3 +--
setup.cfg  |  6 +
tox.ini|  3 +--
8 files changed, 83 insertions(+), 30 deletions(-)


Requirements updates


diff --git a/requirements.txt b/requirements.txt
index 674130c..1486ede 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -5,2 +4,0 @@
-pbr=0.6,!=0.7,1.0
-discover
@@ -14,0 +13 @@ mox3=0.7.0
+os-client-config=1.2.0



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Port Cinder to Python 3

[Victor Stinner]

Hi,

Le 26/05/2015 11:51, Duncan Thomas a écrit :

Thanks Victor, that is pretty much exactly what I wanted to hear -
there's a known, finite plan to get us to fully working with python3.
I'll go change my reviews now.


As expected, two weeks later, all my patches are in conflict :-) I 
rebased my Python 3 patches for Cinder:


https://review.openstack.org/#/q/status:open+project:openstack/cinder+branch:master+topic:py3,n,z

In the meantime, I enhanced my sixer tool, so the generated patches are 
larger but fixes more code. I splitted the six.moves patch into 3 
patches: six.moves, StringIO and urllib.


Can someone please review them to not have to rebase them every 2 weeks? ;-)

Thanks,
Victor

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

[Victor Stinner]

Oh, on the wiki page I read that The liaison should be a core reviewer 
for the project, but does not need to be the PTL.. I'm not a core 
reviewer for nova. Is it an issue?


On the wiki page, I see that John Villalovos (happycamp) is the Nova 
liaison for Oslo, not Joe Goron. I don't understand.


Victor

Le 27/05/2015 20:44, Joe Gordon a écrit :



On Wed, May 27, 2015 at 3:20 AM, Davanum Srinivas dava...@gmail.com
mailto:dava...@gmail.com wrote:

Victor,

Nice, yes, Joe was the liaison with Nova so far. Yes, please go ahead
and add your name in the wiki for Nova as i believe Joe is winding
down the oslo liaison as well.
https://wiki.openstack.org/wiki/CrossProjectLiaisons#Oslo



Yup, thank you Victor!



thanks,
dims

On Wed, May 27, 2015 at 5:12 AM, Victor Stinner vstin...@redhat.com
mailto:vstin...@redhat.com wrote:
  Hi,
 
  By the way, who is the oslo liaison for nova? If there is
nobody, I would
  like to take this position.
 
  Victor
 
  Le 25/05/2015 18:45, Ghe Rivero a écrit :
 
  My focus on the Ironic project has been decreasing in the last
cycles,
  so it's about time to relinquish my position as a oslo-ironic
liaison so
  new contributors can take over it and help ironic to be the vibrant
  project it is.
 
  So long, and thanks for all the fish,
 
  Ghe Rivero
  --
  Pinky: Gee, Brain, what do you want to do tonight?
  The Brain: The same thing we do every night, Pinky—try to take
over the
  world!
 
.''`.  Pienso, Luego Incordio
  : :' :
  `. `'
 `- www.debian.org http://www.debian.org
http://www.debian.org www.openstack.com http://www.openstack.com
  http://www.openstack.com
 
  GPG Key: 26F020F7
  GPG fingerprint: 4986 39DA D152 050B 4699  9A71 66DB 5A36 26F0 20F7
 
 
 
__
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
__
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

[Victor Stinner]

Hi,

By the way, who is the oslo liaison for nova? If there is nobody, I 
would like to take this position.


Victor

Le 25/05/2015 18:45, Ghe Rivero a écrit :

My focus on the Ironic project has been decreasing in the last cycles,
so it's about time to relinquish my position as a oslo-ironic liaison so
new contributors can take over it and help ironic to be the vibrant
project it is.

So long, and thanks for all the fish,

Ghe Rivero
--
Pinky: Gee, Brain, what do you want to do tonight?
The Brain: The same thing we do every night, Pinky—try to take over the
world!

  .''`.  Pienso, Luego Incordio
: :' :
`. `'
   `- www.debian.org http://www.debian.org www.openstack.com
http://www.openstack.com

GPG Key: 26F020F7
GPG fingerprint: 4986 39DA D152 050B 4699  9A71 66DB 5A36 26F0 20F7


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] Port Cinder to Python 3

[Victor Stinner]

(Oops, I sent a draft by mistake, sorry about that.)

Hi,

There is an ongoing effort to port OpenStack applications to Python 3 during 
the Liberty cycle. The following table gives the status of each application and 
links to patches:
https://wiki.openstack.org/wiki/Python3#OpenStack_applications

Specs were accepted for:

* heat: https://review.openstack.org/#/c/175340/
* keystone: https://review.openstack.org/#/c/177380/
* neutron: https://review.openstack.org/#/c/172962/
* nova: https://review.openstack.org/176868

For cinder, I ported the last blocking dependency to Python 3, rtslib-fb:
https://github.com/agrover/rtslib-fb/pull/62
(There is not release including the fix yet.)

MySQL-python should be replaced with PyMySQL to run Python 3 tests, as done in 
Ironic, Nova and other applications.

I started to write patches to port Cinder code to Python 3:
https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:py3,n,z

Duncan Thomas rejected one of my patch:
As commented on https://review.openstack.org/#/c/185411/ I'd really like to be 
convinced that there's an end in sight for this python3 work, or I'm going to 
start rejecting it. This change is making the code noticeably harder to read, 
and has no checks to stop it recurring in future, and is lacking substantial 
justification as to it's benefits.

In the other issue, he wrote:

Is there a master list of remaining python3 issues anywhere? At the moment we 
are introducing more and more little changes into the code, including some 
really ugly six stuff, without any idea if we are even close to actually being 
able to work with python3.

To be honest, I think that I'd like to see a working python3 tree before we 
merge any more - we can then pull in the fixes in a clean manner, knowing the 
end-goal is possible.

Thoughts appreciated, otherwise I'm tempted to start hitting -2 on python3 
stuff.


Ok, I checked the status of Cinder port to Python 3.

I merged my pending patches into a local branch and I fixed manually 
dependencies for tox -e py34 (replace MySQL-python with PyMySQL, install the 
development version of oslo.vmware). With 4 minor changes, I'm able to run one 
cinder unit test (cinder/tests/unit/test_quota.py). Cool!

The next step will be to run a subset of tests in tox -e py34 to make it 
pass, and then add a checking job. When the job becomes stable, make it voting 
to avoid further Python 3 regressions.

Then more code should be ported to Python 3 to slowly port the whole Cinder 
code base.

Well, that's exactly the plan approved for Nova, and other OpenStack 
applications have the same (or a similar) plan for Python 3.

About the usage of six: we chose six to add Python 3 support to all other 
OpenStack libraries and applications. Sorry if it makes the code more ugly. 
More information on porting code to Python 3:
https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3

Victor Stinner aka haypo

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev