[openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-04 Thread victor stinner
Hi, I would like to replace eventlet with asyncio in OpenStack for the asynchronous programming. The new asyncio module has a better design and is less magical. It is now part of python 3.4 arguably becoming the de-facto standard for asynchronous programming in Python world. The asyncio

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-04 Thread victor stinner
Hi, Thanks for taking this on, Victor. Do you have a small eventlet program example that you can show the transformation to asyncio? Example getting the body of an HTML page and display it to the terminal. The HTTP query is invalid, the Host header is missing, but supporting HTTP/1.0 and

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-04 Thread victor stinner
Hi Joshua, Why not just create a eventlet micro version that underneath uses asyncio? Then code isn't riddled with yield from or Return() or similar. It seems possible (maybe?) to replace the eventlet hub with one that uses asyncio? This is then a nice change for all those who are using

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-05 Thread victor stinner
Hi, Chris Behrens wrote: Interesting thread. I have been working on a side project that is a gevent/eventlet replacement [1] that focuses on thread-safety and performance. This came about because of an outstanding bug we have with eventlet not being Thread safe. (We cannot safely enable

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-05 Thread victor stinner
Hi, Thierry Carrez wrote: The problem is that the asyncio module was written for Python 3.3, whereas OpenStack is not fully Python 3 compatible (yet). To easy the transition I have ported asyncio on Python 2, it's the new Trollis project which supports Python 2.6-3.4:

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-06 Thread victor stinner
Hi, Joshua Harlow: Any mysql DB drivers (I think the majority of openstack deployments use mysql?). I don't know. Here are some asynchronous clients for MySQL: https://github.com/PyMySQL/PyMySQL/ https://launchpad.net/myconnpy https://github.com/hybridlogic/txMySQL

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-06 Thread victor stinner
Sean Dague wrote: First, very cool! Thanks. This is very promising work. It might be really interesting to figure out if there was a smaller project inside of OpenStack that could be test ported over to this (even as a stackforge project), and something we could run in the gate. Oslo

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-07 Thread Victor Stinner
Le mardi 4 février 2014, 12:53:24 Kevin Conway a écrit : One of the great benefits of using a green thread abstraction, like eventlet or gevent, is that it lets you write normal Python code and slap your concurrency management over the top. It even lets us program in familiar models by

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-07 Thread Victor Stinner
Le jeudi 6 février 2014, 22:32:26 Zane Bitter a écrit : [BTW am I right in thinking that in trollius you _have_ to drive your co-routines from a top-level Task? That's not the case in Heat (although we do anyway), or IIUC in asyncio - I was expecting e.g. the exception handling for Return to

Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio

2014-02-07 Thread Victor Stinner
Le vendredi 7 février 2014, 09:37:27 Vishvananda Ishaya a écrit : To be clear, since many people weren’t around in ye olde days, nova started using tornado. We exchanged tornado for twisted, and finally moved to eventlet. People have suggested gevent and threads in the past, and now asyncio.

Re: [openstack-dev] [solum] async / threading for python 2 and 3

2014-02-21 Thread Victor Stinner
Le vendredi 21 février 2014, 09:27:49 Angus Salkeld a écrit : Honestly, I have no answer to your question right now (How useful is trollius ...). (...) I asked your question on Tulip mailing list to see how a single code base could support Tulip (yield from) and Trollius (yield). At least

Re: [openstack-dev] [qa] Python 3.3 patches (using six)

2014-03-14 Thread victor stinner
Hi, I'm working for eNovance and we are working on porting OpenStack to Python 3. Status of the port: http://techs.enovance.com/6722/status-of-the-openstack-port-to-python-3-2 https://wiki.openstack.org/wiki/Python3 I understand that it becomes late for Python 3 changes before Icehouse

Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?

2014-03-18 Thread victor stinner
Hi, My patches for Oslo Messaging are naive and inefficient. They are just a first step to prepare OpenStack for Trollius. I chose to run asyncio event loop in its own dedicated thread and dispatch messages in a pool of threads. It would be nice to run asyncio event loop in the main thread, as

[openstack-dev] WSGI and Python paste

2014-03-19 Thread victor stinner
Hi, According to the following table, paste is blocking many OpenStack servers to be ported to Python 3: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects The author of paste, Ian Bicking, gave me the commit permission to paste. I integrated patches from Debian and my

[openstack-dev] Sprint at Pycon: Port OpenStack to Python 3

2014-04-01 Thread victor stinner
Hi, I will organize a sprint to Port OpenStack to Python 3 during 4 days at Montreal (Canada) during Pycon Montreal 2014, between April, 14 (Monday) and April, 17 (Thursday). The goal of the sprint is to port OpenStack components and OpenStack dependencies to Python 3, send patches to port as

Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-04-01 Thread Victor Stinner
Hi, Le lundi 17 mars 2014, 09:19:03 John Eckersberg a écrit : There are a couple of known bugs that can prevent rabbitmq-server from starting on F20. First one (same bug, two BZs) is related to SELinux and port probing: https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8

Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3

2014-04-01 Thread Victor Stinner
Hi, Le mardi 1 avril 2014, 08:12:20 Doug Hellmann a écrit : I won't be attending PyCon this year, but count me in as a remote reviewer. Cool :-) Will you be setting up an IRC channel we should join? Hum, let's say #openstack-pycon on the Freenode server. Victor

Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3

2014-04-01 Thread Victor Stinner
Hi, Le mardi 1 avril 2014, 09:11:52 John Dennis a écrit : What are the plans for python-ldap? Only a small part of python-ldap is pure python, are you also planning on tackling the CPython code? Oh, python-ldap was just an example, I don't have concrete plan for each dependency. We are

Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3

2014-04-01 Thread Victor Stinner
Le mardi 1 avril 2014, 09:44:11 John Dennis a écrit : The goal of the sprint is to port OpenStack components and OpenStack dependencies to Python 3, This is a great goal, thank you! But I'm concerned it might be premature. The portage is already in progress. There are many components (8

Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3

2014-04-01 Thread Victor Stinner
Hi, Le mardi 1 avril 2014, 10:48:21 John Dennis a écrit : Oh, almost forgot. One of the significant issues in Py3 string handling occurs when dealing with the underlying OS, specifically Posix, the interaction with Posix objects such as pathnames, hostnames, environment values, etc. Virtually

Re: [openstack-dev] devstack: Unable to restart rabbitmq-server

2014-04-01 Thread Victor Stinner
Le mardi 1 avril 2014, 12:17:39 John Eckersberg a écrit : An update has been submitted for Fedora 20 which corrects this issue: https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-4.fc20 Please give it a try and if it works for you, be sure to leave karma on the update so it gets

[openstack-dev] Issues when running unit tests in OpenStack

2014-04-08 Thread victor stinner
Hi, I have some issues when running unit tests in OpenStack. I would like to help, but I don't know where I should start and how I can fix these bugs. My use case is to run unit tests and rerun a single test if one or more tests failed. Well, it should be the most basic use case, no? (1)

Re: [openstack-dev] Issues when running unit tests in OpenStack

2014-04-08 Thread victor stinner
Oh, I already got feedback from colleagues :-) Thanks. (1) First problem: if a Python module cannot be loaded, my terminal is flooded with a binary stream which looks like: ...

Re: [openstack-dev] Issues when running unit tests in OpenStack

2014-04-08 Thread victor stinner
Sylvain Bauza wrote: (2) When a test fails, it's hard to find the command to rerun a single failing test. (...) See the wiki page I gave to you. Some helpful tricks are there. That said, I never had the issue you mentioned related to only checking one unittest by providing the path. When

Re: [openstack-dev] [oslo] use of the oslo namespace package

2014-04-08 Thread Victor Stinner
Hi, Le mardi 8 avril 2014, 10:54:24 Julien Danjou a écrit : On Mon, Apr 07 2014, Doug Hellmann wrote: We can avoid adding to the problem by putting each new library in its own package. We still want the Oslo name attached for libraries that are really only meant to be used by OpenStack

[openstack-dev] [oslo] Split Oslo Incubator?

2014-04-08 Thread Victor Stinner
(Follow-up of the [olso] use of the oslo namespace package thread) Hi, The openstack.common module also known as Oslo Incubator or OpenStack Common Libraries has 44 dependencies. IMO we reach a point where it became too huge. Would it be possible to split it into smaller parts and distribute

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-04 Thread victor stinner
Hi, I promise a status of my work on Trollius and greenio to Mark, but it's not easy to summarize it because there are still a few pending patches to implement the final greenio executor. There are different parts: asyncio, Trollius, greenio, Olso Messaging. The design of the asyncio is the

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-07 Thread Victor Stinner
Hi, Le lundi 7 juillet 2014, 12:48:59 Nikola Đipanov a écrit : When I read all of this stuff and got my head around it (took some time :) ), a glaring drawback of such an approach, and as I mentioned on the spec proposing it [1] is that we would not really doing asyncio, we would just be

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-07 Thread Victor Stinner
Le lundi 7 juillet 2014, 11:26:27 Gordon Sim a écrit : When we switch to asyncio's event loop, all of this code needs to be ported to asyncio's explicitly asynchronous approach. We might do: @asyncio.coroutine def foo(self): result = yield from some_async_op(...)

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-08 Thread Victor Stinner
Le lundi 7 juillet 2014, 19:14:45 Angus Salkeld a écrit : 4) retraining of OpenStack developers/reviews to understand the new event loop. (eventlet has warts, but a lot of devs know about them). Wait, what? Sorry if it was not clear, but the *whole* point of replacing eventlet with asyncio

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-10 Thread Victor Stinner
Le jeudi 10 juillet 2014, 14:48:04 Yuriy Taraday a écrit : I'm not suggesting that taskflow is useless and asyncio is better (apple vs oranges). I'm saying that using coroutines (asyncio) can improve ways we can use taskflow and provide clearer method of developing these flows. This was mostly

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-11 Thread Victor Stinner
Hi, Le lundi 7 juillet 2014, 19:18:38 Mark McLoughlin a écrit : I'd expect us to add e.g. @asyncio.coroutine def call_async(self, ctxt, method, **kwargs): ... to RPCClient. Perhaps we'd need to add an AsyncRPCClient in a separate module and only add the method there - I don't

[openstack-dev] [oslo] run tests using testr on Python 3

2014-04-08 Thread victor stinner
Hi, Olso Incubator runs tests using testr on Python 2, but it uses nosetests on Python 3 to only run a subset of the test suite (modules and tests ported to Python 3). In my latest patch for Oslo Incubator (gettext), Ben Nemec wrote: I think we could get around the nose issue by using a testr

Re: [openstack-dev] [oslo] Split Oslo Incubator?

2014-04-08 Thread Victor Stinner
Hi, Le mardi 8 avril 2014, 10:05:31 Doug Hellmann a écrit : The openstack.common module also known as Oslo Incubator or OpenStack Common Libraries has 44 dependencies. IMO we reach a point where it became too huge. Would it be possible to split it into smaller parts and distribute it on

Re: [openstack-dev] [oslo][db] oslo.db repository review request

2014-05-12 Thread Victor Stinner
to graduation of oslo.db. You can find the new oslo.db code at [3] There was a discussion recently about the oslo namespace. Sorry, but I don't remember the decision. Should the new module be called oslo.db or olsodb? (I would prefer oslodb, it avoids issues with the oslo package.) Victor

Re: [openstack-dev] [oslo][db] oslo.db repository review request

2014-05-14 Thread Victor Stinner
Le mardi 13 mai 2014, 07:31:34 Doug Hellmann a écrit : Since we think we have been able to solve all of the issues we were having with namespace packages before, ... I just tried to start my DevStack and again, I had issues with a builtin olso module: import oslo.config doesn't work, whereas

[openstack-dev] [oslo] Logging exceptions and Python 3

2014-05-15 Thread Victor Stinner
Hi, I'm trying to define some rules to port OpenStack code to Python 3. I just added a section in the Port Python 2 code to Python 3 about formatting exceptions and the logging module: https://wiki.openstack.org/wiki/Python3#logging_module_and_format_exceptions The problem is that I don't know

[openstack-dev] [oslo] strutils: enhance safe_decode() and safe_encode()

2014-05-15 Thread Victor Stinner
Hi, The functions safe_decode() and safe_encode() have been ported to Python 3, and changed more than once. IMO we can still improve these functions to make them more reliable and easier to use. (1) My first concern is that these functions try to guess user expectation about encodings. They

Re: [openstack-dev] [oslo] Logging exceptions and Python 3

2014-05-16 Thread Victor Stinner
Le vendredi 16 mai 2014, 06:03:53 Johannes Erdfelt a écrit : On Fri, May 16, 2014, Igor Kalnitsky ikalnit...@mirantis.com wrote: According to http://legacy.python.org/dev/peps/pep-0352/ the message attribute of BaseException is deprecated since Python 2.6 and was dropped in Python 3.0.

Re: [openstack-dev] [oslo] Mehdi Abaakouk added to oslo.messaging-core

2014-06-09 Thread Victor Stinner
Le vendredi 6 juin 2014, 15:57:09 Mark McLoughlin a écrit : Mehdi has been making great contributions and reviews on oslo.messaging for months now, so I've added him to oslo.messaging-core. Thank you for all your hard work Mehdi! Congrats Mehdi :-) Victor

[openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-23 Thread victor stinner
Hi, I'm happy to announce you that I just finished the last piece of the puzzle to add support for trollius coroutines in Oslo Messaging! See my two changes: * Add a new aiogreen executor: https://review.openstack.org/#/c/136653/ * Add an optional executor callback to dispatcher:

Re: [openstack-dev] Python 3 is dead, long live Python 3

2015-02-03 Thread Victor Stinner
Hi, It's good to move forward to Python 3.4 :-) [2] https://launchpad.net/bugs/1367907 This bug was introduced in Python 3.4.0 and fixed in Python 3.4.1. It's too bad that Ubunbu Trusty didn't upgraded yet Python 3.4 to 3.4.1 (released 6 months ago) or 3.4.2. Request to upgrade python 3.4 in

Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap

2015-01-27 Thread victor stinner
Hi, What is the Python bug? Do you have a reference to the bug report and the patch? Python 3.4.3 release schedule: 3.4.3rc1 will be tagged Saturday February 7 and released Sunday February 8. 3.4.3 final will follow two weeks later, tagged Saturday February 21 and released Sunday February 22.

[openstack-dev] [all] Replace eventlet with asyncio

2015-02-10 Thread Victor Stinner
volunteers to help fix in multiple projects. Primary assignee: Victor Stinner vstin...@redhat.com Work Items -- Work items or tasks -- break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we're mostly

[openstack-dev] [oslo] Progress of the port to Python 3

2015-01-06 Thread victor stinner
aioeventlet dependency https://review.openstack.org/138750 Victor Stinner ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] requirements-py{2, 3} and universal wheels

2015-03-18 Thread Victor Stinner
I haven't tested yet (and someone should) that it does all JUST WORK, but thats easy: put an environment marker in a requirements.txt file like so: argparse; python_version '3' I think the last time this came up the feature wasn't available in pip yet, and so using separate files was

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Victor Stinner
Hi, Davanum Srinivas wrote: +1 to fix Oslo's service module any ways, irrespective of this bug. By the way, the Service class is a blocker point for the implementation of asyncio and threads specs: https://review.openstack.org/#/c/153298/ https://review.openstack.org/#/c/156711/ We may

Re: [openstack-dev] [all] Replace eventlet with asyncio

2015-02-25 Thread Victor Stinner
Hi, I also just put up another proposal to consider: https://review.openstack.org/#/c/156711/ Sew over eventlet + patching with threads My asyncio spec is unclear about WSGI, I just wrote The spec doesn't change OpenStack components running WSGI servers like nova-api. The specific problem of

Re: [openstack-dev] [oslo] Test failures due to oslo_concurrency?

2015-01-13 Thread victor stinner
See also the following changes: oslo.concurrency: Don't decode stdout/stderr on Python 2 https://review.openstack.org/#/c/146984/ (my change) oslo.concurrency: Allow callers to disable encoding for execute() https://review.openstack.org/#/c/146932/ (Doug Hellmann) nova: Do not decode binary

Re: [openstack-dev] [stable] freeze exception

2015-04-03 Thread Victor Stinner
Hi, I had like to request a freeze exception for this nova fix: https://review.openstack.org/#/c/164713/ Related bug: https://bugs.launchpad.net/nova/+bug/1406486 Suspending an instance fails when using vnic_type=direct My change is a backport from master to juno of the fix already merged into

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-23 Thread Victor Stinner
Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3: https://github.com/linsomniac/python-memcached/pull/67 I don't understand. I saw a lot of

Re: [openstack-dev] [oslo][pbr] getting 0.11 out the door. Maybe even 1.0

2015-04-20 Thread Victor Stinner
I don't think its a bug in the applications. swift gets its version using pkg_resources, or falls back to pbr: https://github.com/openstack/swift/blob/master/swift/__init__.py I mean that other applications may do something similar? Victor

Re: [openstack-dev] [oslo][pbr] getting 0.11 out the door. Maybe even 1.0

2015-04-20 Thread Victor Stinner
I believe Redhat patch it out. I don't think they should need to, since we have explicit knobs for distros to use. pbr pulls pip which we don't want in RHEL. Example of patches in RDO: https://github.com/redhat-openstack/nova/commit/a19939c8f9a7b84b8a4d713fe3d26949e5664089

Re: [openstack-dev] [oslo][pbr] getting 0.11 out the door. Maybe even 1.0

2015-04-20 Thread Victor Stinner
If there's nothing majorly wrong mid-L, I'd like to release 1.0.0 just to get us into 'ok its stable' mentality. I read that many packages modify the source code of libraries and applications to avoid a dependency to pbr at runtime. What's the status of this issue? Is pbr still used/required

[openstack-dev] [nova] Port Nova to Python 3

2015-04-24 Thread Victor Stinner
Hi, Porting OpenStack applications during the Liberty Cycle was discussed last days in the thread [oslo] eventlet 0.17.3 is now fully Python 3 compatible. I wrote a spec to port Nova to Python 3: https://review.openstack.org/#/c/176868/ I mentioned the 2 other Python 3 specs for Neutron

Re: [openstack-dev] [nova] Port Nova to Python 3

2015-04-24 Thread Victor Stinner
Hi, If written to use python 3, I hope it will use all the new features of python 3.4 moving forward. The spec adds Python 3 support, but it keeps Python 2 support. It's too early to drop Python 2, Nova is deployed everywhere with Python 2. For example, argument annotations, coroutines,

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-23 Thread Victor Stinner
Hi, How invasive would the port to python3 be? I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d As announced, changes are boring, just obvious Python2/Python3 issues: - strip L

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-23 Thread Victor Stinner
I recommend to use mysqlclient instead of MySQL-python even on Python 2. https://pypi.python.org/pypi/mysqlclient https://github.com/PyMySQL/mysqlclient-python Is it packaged in popular distributions? RHEL? Fedora? SuSe? Ubuntu? Debian? Gentoo? If this library solves real bugs and

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-23 Thread Victor Stinner
Hi, These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs. Is there already a static analysis tool that helps find these things? (Would a pylint check for the above be useful? Some of them would be hard to find reliably, but a

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-22 Thread Victor Stinner
Hi, It's moving fast. I'm currently working on porting remaining libraries to prepare my spec for nova. oslo.db -- looks like it is almost there I don't know the status of oslo.db support of Python 3. I guess that it already works on Python 3, it's just a matter of running tests with MySQL

[openstack-dev] [oslo] release of Paste 2.0 with an experimental Python 3 support

2015-04-22 Thread Victor Stinner
Hi, Ian Bicking allowed me to publish new versions of the Paste project: https://bitbucket.org/ianb/paste This project blocked many OpenStack applications to be ported to Python 3: https://wiki.openstack.org/wiki/Python3#OpenStack_applications Good news: I just released Paste 2.0 which

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-22 Thread Victor Stinner
Hi, sqlachemy-migrate already works on Python3. I sent a patch to add missing Python 3 classifiers, so the caniusepython3 automated tool will stop reporting false alarm on this library. https://review.openstack.org/174738 Or is someone aware of issues with Python 3? Again, check the wiki page

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-24 Thread Victor Stinner
Hi, The part of keystoneclient that uses the memcached client was deprecated in Juno (as it was moved to the keystonemiddleware repo), Oh, I was not aware of the keystonemiddleware project. I see that Nova uses it for example. so I think we can remove it now. Do someone know if the

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-24 Thread Victor Stinner
Hi, I wrote my spec to Port Nova to Python 3: https://review.openstack.org/#/c/176868/ I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d I like how the sha1 starts with 'bad'

[openstack-dev] [all] Replace mysql-python with mysqlclient

2015-04-30 Thread Victor Stinner
Hi, I propose to replace mysql-python with mysqlclient in OpenStack applications to get Python 3 support, bug fixes and some new features (support MariaDB's libmysqlclient.so, support microsecond in TIME column). The MySQL database is popular, but the Python driver mysql-python doesn't look

Re: [openstack-dev] [nova] Port Nova to Python 3

2015-05-04 Thread Victor Stinner
Hi, I wrote a spec to port Nova to Python 3: https://review.openstack.org/#/c/176868/ I updated my spec to take in account all comments. Example of changes: - Explicitly say that Python 2 support is kept (ex: change the title to Adding Python 3.4 support to Nova) - Clarify what are the

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-04 Thread Victor Stinner
Hi, Mike Bayer wrote: It is not feasible to use MySQLclient in Python 2 because it uses the same module name as Python-MySQL, and would wreak havoc with distro packaging and many other things. IMO mysqlclient is just the new upstream for MySQL-Python, since MySQL-Python is no more

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-04 Thread Victor Stinner
I propose to replace mysql-python with mysqlclient in OpenStack applications to get Python 3 support, bug fixes and some new features (support MariaDB's libmysqlclient.so, support microsecond in TIME column). I just proposed a change to add mysqlclient dependency to global requirements:

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-04-30 Thread Victor Stinner
If we're going to move, shouldn't we be looking at something that supports our threading model? I would prefer to make baby steps, and first fix the Python 3 compatibility. Enhance concurrency/parallelism is a much more complex project than just replacing a single line in dependencies ;-)

Re: [openstack-dev] [oslo] Adding Mehdi Abaakouk (sileht) to oslo-core

2015-05-11 Thread Victor Stinner
Hi, I didn't know that Mehdi was only core reviewer on Oslo Messaging. +1 for Mehdi as Oslo core reviewer. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:

Re: [openstack-dev] [nova] Port Nova to Python 3

2015-05-11 Thread Victor Stinner
Hi, Oh, this makes me think: how would one fix something like this? The wiki page already contains some answer: https://wiki.openstack.org/wiki/Python3#Common_patterns Don't hesitate to complete the page if needed. See also my personal list of documents:

Re: [openstack-dev] [oslo] Adding Joshua Harlow to oslo-core

2015-05-05 Thread Victor Stinner
Hi, Great idea! Joshua did a lot for Oslo, I want him in the oslo-core team! Victor Hi fellows, I'd like to propose that we add Joshua Harlow to oslo-core. He is already maintaining some of the Oslo libraries (taskflow, tooz…) and he's helping on a lot of other ones for a while now. Let's

Re: [openstack-dev] [Ironic] Python 3.4 unit tests enabled on the gate for openstack/ironic

2015-05-13 Thread Victor Stinner
With this commit: https://review.openstack.org/181034 Python 3.4 unit tests are now being run for openstack/ironic. The unit tests are a voting job. Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4:

Re: [openstack-dev] [all] Replace mysql-python with mysqclient to get Python 3 support

2015-05-13 Thread Victor Stinner
Hi, Since the discussion on mysqlclient looks to be stuck, I chose the approach adopted in Ironic: use PyMySQL on Python 3. I proposed a patch to run Nova tests using PyMySQL on Python 3, but keep MySQL-python on Python 2. https://review.openstack.org/#/c/182709/ It only impacts how Nova run

[openstack-dev] [all] Replace mysql-python with mysqclient to get Python 3 support

2015-05-12 Thread Victor Stinner
Hi, TL;DR we must replace mysql-python with mysqclient to get Python 3 support, it's safe and *very* easy I restart the discussion because the previous one gone far from my point which is getting Python 3 support. IMO the fastest way to get Python 3 support is to replace mysql-python with

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-17 Thread Victor Stinner
For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. Why? I plan to work

Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-13 Thread Victor Stinner
Worth noting we've already switched to using PyMySQL in nodepool, storyboard and some of the subunit2sql tooling. It's been working out great so far. Great. Did you notice a performance regression? Mike wrote that PyMySQL is much slower than MySQL-Python. Victor

Re: [openstack-dev] [oslo] Add request_utils module in oslo-incubator

2015-04-14 Thread Victor Stinner
I'd support picking another existing oslo library instead of oslo-incubator itself. Can you please review to see where this would fit? In the review, I suggested oslo.log, but I'm not sure because of the notifier object. I don't know the type of the notifier object, is it something related

[openstack-dev] [all] Issues with pip 6.1

2015-04-07 Thread Victor Stinner
Hi, I tried to install a fresh DevStack but it doesn't work because of the new release of pip: pip 6.1 (probably released today, https://pip.pypa.io/en/stable/news.html still says 6.1.0 (unreleased)...). (1) update.py of openstack/requirements uses req.InstallRequirement.url attribute, but

Re: [openstack-dev] [all] Issues with pip 6.1

2015-04-07 Thread Victor Stinner
(2) it's not more possible to install argparse on Python 2.7 (...) Donald Stufft released pip 6.1.1, he reverted the change which prevented to install argparse. Thanks for the quick fix! (1) update.py of openstack/requirements uses req.InstallRequirement.url attribute, but this attribute

[openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-09 Thread Victor Stinner
Hi, During the last OpenStack Summit at Paris, we discussed how we can port OpenStack to Python 3, because eventlet was not compatible with Python 3. There are multiple approaches: port eventlet to Python 3, replace eventlet with asyncio, replace eventlet with threads, etc. We decided to not

Re: [openstack-dev] [all] gate pep8 jobs broken today

2015-05-19 Thread Victor Stinner
Hi, I found a related issue in glance. issue with pbr 1.0 release https://bugs.launchpad.net/glance/+bug/1456800 Modify entry point tests to not require deps https://review.openstack.org/#/c/184326/ Victor - Original Message - From: Robert Collins robe...@robertcollins.net To:

Re: [openstack-dev] [cinder] Port Cinder to Python 3

2015-06-09 Thread Victor Stinner
Hi, Le 26/05/2015 11:51, Duncan Thomas a écrit : Thanks Victor, that is pretty much exactly what I wanted to hear - there's a known, finite plan to get us to fully working with python3. I'll go change my reviews now. As expected, two weeks later, all my patches are in conflict :-) I rebased

Re: [openstack-dev] [oslo] oslo releasing is noisy...

2015-06-10 Thread Victor Stinner
Le 09/06/2015 18:31, Thierry Carrez a écrit : We are currently exploring the option to repurpose the openstack-announce ML to be the extensive record of release announcements. It's part of a larger plan to streamline library releases, about which Doug should start a discussion pretty soon now.

Re: [openstack-dev] [release][oslo] oslotest release 1.7.0 (liberty)

2015-06-10 Thread Victor Stinner
Good news: your fix is already merged into oslo.messaging ;-) oslo.messaging now uses directly mox3, on Python 2 and Python 3. Victor Le 10/06/2015 14:04, Victor Sergeyev a écrit : Hi All, this oslotest release break oslo.messaging gate - unittest fails with ImportError, on import mox from

[openstack-dev] [glance] Progress of the Python 3 port

2015-06-22 Thread Victor Stinner
Hi, We made great progress on porting Glance to Python 3. tox -e py34 now pass with a subset of tests and there is a non-voting py34 check job. I propose to make the py34 gate voting to avoid Python 3 regressions: https://review.openstack.org/194048 Make gate-glance-python34 voting When the

Re: [openstack-dev] Debian already using Python 3.5: please gate on that

2015-06-22 Thread Victor Stinner
Hi, Le 18/06/2015 15:44, Thomas Goirand a écrit : As per the subject line, we already have Python 3.5 in Debian (AFAICT, from Debian Experimental, in version beta 2). As a consequence, we're already running (unit) tests using Python 3.5. Some have failures: I could see issues in

[openstack-dev] [cinder][glance][nova] Python 3 coming!

2015-06-26 Thread Victor Stinner
Hi, The Python 3.4 gate (py34) of cinder, glance and nova just became voting. You now have to write Python code compatible with Python 2.7 and 3.4. If the py34 check job fails on your patch, you may try to rebase it to get the new tox.ini and updated requirements. You can find information

Re: [openstack-dev] Jenkins check failed on gate-glance_store-python34

2015-06-26 Thread Victor Stinner
Hi, Le 26/06/2015 09:17, 孙明达 a écrit : I am a new contributor for openstack. the following url is my code review for glance_store https://review.openstack.org/#/c/193403/ Jenkins check failed on gate-glance_store-python34 Referring to the console info, I have never changed the code

Re: [openstack-dev] [all][python3] use of six.iteritems()

2015-06-11 Thread Victor Stinner
Hi, Le 10/06/2015 02:15, Robert Collins a écrit : python2.7 -m timeit -s 'd=dict(enumerate(range(100)))' 'for i in d.items(): pass' 10 loops, best of 3: 76.6 msec per loop python2.7 -m timeit -s 'd=dict(enumerate(range(100)))' 'for i in d.iteritems(): pass' 100 loops, best of 3: 22.6

Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

2015-05-27 Thread Victor Stinner
Hi, By the way, who is the oslo liaison for nova? If there is nobody, I would like to take this position. Victor Le 25/05/2015 18:45, Ghe Rivero a écrit : My focus on the Ironic project has been decreasing in the last cycles, so it's about time to relinquish my position as a oslo-ironic

Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison

2015-05-28 Thread Victor Stinner
Joe is winding down the oslo liaison as well. https://wiki.openstack.org/wiki/CrossProjectLiaisons#Oslo Yup, thank you Victor! thanks, dims On Wed, May 27, 2015 at 5:12 AM, Victor Stinner vstin...@redhat.com mailto:vstin...@redhat.com wrote: Hi

Re: [openstack-dev] [nova] gate is wedged

2015-06-30 Thread Victor Stinner
Hi, Le 30/06/2015 05:49, Matt Riedemann a écrit : Alternatively, oslo.versionedobjects 0.5.1 is cut after https://review.openstack.org/#/c/196926/ is merged and then we just need haypo's test_db_api fix for the oslo.db 2.0.0 change: https://review.openstack.org/#/c/196719/ I updated my patch

Re: [openstack-dev] [nova] gate is wedged

2015-06-30 Thread Victor Stinner
Le 30/06/2015 10:32, Victor Stinner a écrit : I updated my patch Update test_db_api for oslo.db 2.0 (1) ... (1) https://review.openstack.org/#/c/196719/ I updated my patch again to also block oslo.versionedobjects 0.5.0 in requirements. So we can have two patches to fix the bug ;) The patch

Re: [openstack-dev] Let's get rid of tablib and cliff-tablib

2015-06-29 Thread Victor Stinner
Hi, Le 29/06/2015 11:03, Thomas Goirand a écrit : cliff-tablib is used for the unit tests of things like python-neutronclient. The annoying bit is that cliff-tablib depends on tablib, which itself is a huge mess. It has loads of 3rd party embedded packages and most of them aren't Python 3.x

[openstack-dev] [oslo] Ideas to detect Oslo regressions earlier

2015-06-30 Thread Victor Stinner
Hi, Unfortunately, it looks like each regressions introduced by releases of Oslo libraries are still common :-( We already have tools to detect regressions, but they are run manually. It would be nice to automate these tests and run them more frequently (at least one per week, or even

Re: [openstack-dev] [oslo] Adopt mox3

2015-06-11 Thread Victor Stinner
Hi, Le 10/06/2015 22:17, Davanum Srinivas a écrit : Oslo folks, everyone, mox3 needs to be maintained since some of our projects use it and we have it in our global requirements. Here's the proposal from Doug - https://review.openstack.org/#/c/190330/ Why not only creating a project on

Re: [openstack-dev] [nova][oslo] RPC Asynchronous Communication

2015-05-25 Thread Victor Stinner
Hi Sahid, There is a work-in-progress work to change how concurrency is handled in openstack processes: Replace eventlet with ??? https://review.openstack.org/164035 The latest updates can be found in etherpads from the OpenStack Vancouver Summit: -

[openstack-dev] [cinder] Port Cinder to Python 3

2015-05-26 Thread Victor Stinner
to add Python 3 support to all other OpenStack libraries and applications. Sorry if it makes the code more ugly. More information on porting code to Python 3: https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3 Victor Stinner aka haypo

[openstack-dev] [cinder] Port Cinder to Python 3

2015-05-26 Thread Victor Stinner
Hi, There is an ongoing effort to port OpenStack applications to Python 3 during the Liberty cycle. The following table gives the status of each application and links to patches: https://wiki.openstack.org/wiki/Python3#OpenStack_applications Specs were accepted for: * heat:

  1   2   >