[openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Hi, I would like to replace eventlet with asyncio in OpenStack for the asynchronous programming. The new asyncio module has a better design and is less magical. It is now part of python 3.4 arguably becoming the de-facto standard for asynchronous programming in Python world. The asyncio module is a new module of Python 3.4 written by Guido van Rossum as an abstraction of existing event loop (Twisted, Tornado, eventlet, etc.). In fact, it's more than an abstraction: it has its own event loop and can be used alone. For the background, read the PEP: http://www.python.org/dev/peps/pep-3156/ Asynchronous IO Support Rebooted: the asyncio Module For more information on asyncio, see its documentation: http://docs.python.org/dev/library/asyncio.html The asyncio module is also known as Tulip which is third-party project written for Python 3.3. Tulip was written first and then it was integrated in Python 3.4. http://code.google.com/p/tulip/ The main difference between eventlet and asyncio is that context switching between two concurrent tasks is explicit. When a task blocks (ex: wait for an event), it should use the yield from syntax which will switch to the next task: it's similar to greenlet.switch in eventlet. So it becomes obvious which parts of the code do switch and which don't. With eventlet, you have to read the source code of a function before calling it to check if it may call greenlet.switch() or not, and so debugging is more difficult. Or worse, the function may switch in a new version of a module, you won't notice the change. The asyncio module handles various kind of events: sockets, pipes, subprocesses, UNIX signals, etc. All these things are handled in a single event loop. You can uses an exector to run a blocking task in a pool of thread. There is a low-level API using transports and protocols, similar to Twisted transports and protocols. But there is also a high-level API using streams, which gives a syntax close to eventlet (except that you have to add yield from). See an example to send an HTTP request and print received HTTP headers, the yield from reader.readline() instruction blocks until it gets a full line: http://docs.python.org/dev/library/asyncio-stream.html#example The problem is that the asyncio module was written for Python 3.3, whereas OpenStack is not fully Python 3 compatible (yet). To easy the transition I have ported asyncio on Python 2, it's the new Trollis project which supports Python 2.6-3.4: https://bitbucket.org/enovance/trollius The Trollius API is the same than asyncio, the main difference is the syntax in coroutines: yield from task must be written yield task, and return value must be written raise Return(value). The first step to move from eventlet to asyncio is a new executor using Trollius in Olso Messaging: https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio https://review.openstack.org/#/c/70948 The transition from eventlet to asyncio can be done step by step. Thanks to the greenio project, asyncio can reuse the greenlet event loop (and so run in the main thread). So asyncio and eventlet become compatible. While asyncio can also run its own event loop in a separated thread. If eventlet is completely replaced with asyncio in a project, greenio can be dropped, and asyncio event loop can be run its own event loop in the main thread. When OpenStack will be compatible with Python 3.3, it will be possible to use the builtin asyncio module of Python 3.4 directly instead of Trollus. Since yield from is incompatible with Python 2, some parts of the code may need to have two versions (one for Python 2, one for Python 3) if we want to use the Python 3 flavor of asyncio... or Python 2 support might be simplify dropped. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Hi,
Thanks for taking this on, Victor.
Do you have a small eventlet program example that you can show the
transformation to asyncio?
Example getting the body of an HTML page and display it to the terminal. The
HTTP query is invalid, the Host header is missing, but supporting HTTP/1.0 and
the connected mode would make the example more complex.
Eventlet version:
---
import eventlet
from eventlet.green import socket
def getbody(url, limit=1024):
c = socket.socket()
ip = socket.gethostbyname(url)
c.connect((ip, 80))
c.sendall('HEAD /\r\n\r\n')
body = c.recv(limit)
return body
body = getbody('www.python.org')
print(body)
---
Trollius version:
---
import asyncio
import logging; logging.basicConfig()
from asyncio import Return
import socket
@asyncio.coroutine
def getbody(url):
reader, writer = yield asyncio.open_connection(url, 80)
writer.write('HEAD /\r\n\r\n')
# yield writer.drain()
body = yield reader.read()
raise Return(body)
loop = asyncio.get_event_loop()
body = loop.run_until_complete(getbody('www.python.org'))
print(body)
loop.close()
---
Remarks:
- no special sendall method: writer.write() always write all data, and it's
the same name for any kind of transport (socket, SSL socket, pipe, etc.)
- open_connection() accepts directly an hostname. It resolves the hostname
asynchronously (call socket.gethostbyname in a pool of threads)
- it's possible to call reader.read() with no limit: in this case, read()
stops when the server closes the connection. Using eventlet, you have to buffer
manually. There is also a convinient readline() buffer which reads until it
gets a newline character (\n)
- run_until_complete() is only used in short examples, usually run_forever()
is preferred, something will call loop.stop() to stop the loop
You can uncomment writer.drain() to wait until all data are sent to the
server. Otherwise, it will be done in background asynchronously.
See asyncio documentation for more examples:
http://docs.python.org/dev/library/asyncio.html
Victor
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Hi Joshua, Why not just create a eventlet micro version that underneath uses asyncio? Then code isn't riddled with yield from or Return() or similar. It seems possible (maybe?) to replace the eventlet hub with one that uses asyncio? This is then a nice change for all those who are using eventlet. I don't understand what do you expect from replacing the eventlet event loop with asyncio event loop. It's basically the same. The purpose of replacing eventlet with asyncio is to get a well defined control flow, no more surprising task switching at random points. You may read the report of Guido van Rossum's keynote at Pycon 2013, he explains probably better than me how asyncio is different from other asynchronous frameworks: http://lwn.net/Articles/544522/ For more information, see my notes about asyncio: http://haypo-notes.readthedocs.org/asyncio.html#talks-about-asyncio Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Hi, Chris Behrens wrote: Interesting thread. I have been working on a side project that is a gevent/eventlet replacement [1] that focuses on thread-safety and performance. This came about because of an outstanding bug we have with eventlet not being Thread safe. (We cannot safely enable thread pooling for DB calls so that they will not block.) There are DB drivers compatible with asyncio: PostgreSQL, MongoDB, Redis and memcached. There is also a driver for ZeroMQ which can be used in Oslo Messaging to have a more efficient (asynchronous) driver. There also many event loops for: gevent (geventreactor, gevent3), greenlet, libuv, GLib and Tornado. See the full list: http://code.google.com/p/tulip/wiki/ThirdParty Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Hi,
Thierry Carrez wrote:
The problem is that the asyncio module was written for Python 3.3, whereas
OpenStack is not fully Python 3 compatible (yet). To easy the transition I
have ported asyncio on Python 2, it's the new Trollis project which
supports Python 2.6-3.4:
https://bitbucket.org/enovance/trollius
How much code from asyncio did you reuse ? How deep was the porting
effort ? Is the port maintainable as asyncio gets more bugfixes over time ?
Technically, Trollius is a branch of the Tulip project. I host the repository
on Bitbucket, whereas Tulip is hosted at code.google.com. I use hg merge to
retrieve last changes from Tulip into Trollius.
Differences between Trollius and Tulip show how much work has been done between
Python 2.6 and 3.3 :-) Some examples:
- classes must inherit from object in Python 2.6 to be new-style classes
(it's not more needed in Python 3),
- {}.format() must be replaced with {0}.format(),
- IOError/OSError exceptions have been reworked and now have specialized
subclasses in Python 3.3 (I reimplemented them for Python 2.6),
- etc.
But most of the code is still the same between Tulip and Trollius. In my
opinion, the major difference is that Tulip uses yield from wheras Trollius
uses yield, which imply subtle differences in the module iteself. You may not
notice them if you use Trollius, but the implementation is a little bit
different because of that (differences are limited to the asyncio/tasks.py
file).
I'm working actively on Tulip (asyncio). We are fixing last bugs before the
release of Python 3.4, scheduled for March 16, 2014. So I track changes in
Tulip and I will port them into Trollius.
The Trollius API is the same than asyncio, the main difference is the
syntax in coroutines: yield from task must be written yield task, and
return value must be written raise Return(value).
Could we use a helper library (like six) to have the same syntax in Py2
and Py3 ? Something like from six.asyncio import yield_from,
return_task and use those functions for py2/py3 compatible syntax ?
You can use Trollius with Python 3 (I tested it on Python 3.2, 3.3 and 3.4), so
use yield syntax on Python 2 and Python 3.
Guido van Rossum proposed to use yield From(future) in Trollius, so it would
be easier to port Trollius code (yield) on Tulip (yield from). Since OpenStack
is not going to drop Python 2 support, I don't think that it's really useful
(for OpenStack).
Victor
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Hi, Joshua Harlow: Any mysql DB drivers (I think the majority of openstack deployments use mysql?). I don't know. Here are some asynchronous clients for MySQL: https://github.com/PyMySQL/PyMySQL/ https://launchpad.net/myconnpy https://github.com/hybridlogic/txMySQL http://chartio.com/blog/2011/06/making-mysql-queries-asynchronous-in-tornado http://www.arpalert.org/mysac.html http://code.google.com/p/perl-mysql-async/ IMO to have an efficient driver for asyncio, it should give access to the socket / file descriptor, so asyncio can watch it can execute a callback when some data can be read on the socket. A pure Python connector should fit such requirements. Or the API should use a callback when the result is available. How about sqlalchemy (what would possibly need to change there for it to work)? I found some projects using SQLAchemy asynchronously, but only with PostgreSQL. The pain that I see is that to connect all these libraries into asyncio they have to invert how they work (sqlalchemy would have to become asyncio compatible (?), which probably means a big rewrite). There is no problem to call slow blocking functions in asyncio. But if you want to have efficient code, it's better to run the blocking code asynchronously. For example, use loop.run_in_executor() with a thread pool. This is where it would be great to have a 'eventlet' like-thing built ontop of asyncio (letting existing libraries work without rewrites). Eventually I guess in-time (if tulip succeeds) then this 'eventlet' like-thing could be removed. It's a little bit weird to design an abstraction on top of asyncio, since asyncio has be designed an an abstraction of existing event loops. But I wrote an asyncio executor for Oslo Messaging which has already such abstraction. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Sean Dague wrote: First, very cool! Thanks. This is very promising work. It might be really interesting to figure out if there was a smaller project inside of OpenStack that could be test ported over to this (even as a stackforge project), and something we could run in the gate. Oslo Messaging is a small project, but it's more a library. For a full daemon, my colleague Mehdi Abaakouk has a proof-on-concept for Ceilometer replacing eventlet with asyncio. Mehdi told me that he doesn't like to debug eventlet race conditions :-) Our experience is the OpenStack CI system catches bugs in libraries and underlying components that no one else catches, and definitely getting something running workloads hard on this might be helpful in maturing Trollius. Basically coevolve it with a piece of OpenStack to know that it can actually work on OpenStack and be a viable path forward. Replacing eventlet with asyncio is a huge change. I don't want to force users to use it right now, nor to do the change in one huge commit. The change will be done step by step, and when possible, optional. For example, in Olso Messaging, you can choose the executor: eventlet or blocking (and I want to add asyncio). Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Le mardi 4 février 2014, 12:53:24 Kevin Conway a écrit : One of the great benefits of using a green thread abstraction, like eventlet or gevent, is that it lets you write normal Python code and slap your concurrency management over the top. It even lets us program in familiar models by pretending like we are using threads. Using Tulip/Trollius, thanks to yield-from/yield, the code a function is sequential. So it's very close to eventlet programming if you ignore yield/yield from in the code. Basically, just replace data = obj.read(10) with data = yield obj.read(10). Switching our async IO management from eventlet to asyncio would not be a trivial task. Tell me when I'm wrong, but it would require that we completely change our programming model from typical, function-call based programming to use generator-iterators for everything. No, that's wrong. The only functions that will need to be made asynchronous are all functions which are performance critical. Examples: HTTP query to a server or request to a database. All other code will be left unchanged. I don't see why external libraries should be modified. Only the few libraries sending HTTP queries and requests to the database should handle asyncio. Dummy example: the iso8601 module used to parse time doesn't need to be aware of asyncio. *** I don't know OpenStack well enough to say how much code should be modified, but I guess that only few functions need to be modified to be asynchronous. Dummy example: --- def task(): print(hello) time.sleep(1) print(world) --- With eventlet and monkey-patching, this task is suspended and other tasks can be executed before world is displayed. With eventlet without monkey-patching orwith asyncio: this function will block for 1 second. To make this function asynchronous and allow other tasks to run during the sleep, you have to write: --- @asyncio.coroutine def task(): print(hello) yield from asyncio.sleep(1) print(world) --- But you will also have to modify how the function is called. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Le jeudi 6 février 2014, 22:32:26 Zane Bitter a écrit : [BTW am I right in thinking that in trollius you _have_ to drive your co-routines from a top-level Task? That's not the case in Heat (although we do anyway), or IIUC in asyncio - I was expecting e.g. the exception handling for Return to happen in the coroutine decorator rather than in Task. Just curious.] Sorry, but I don't understand what you mean by to drive your co-routines from a top-level Task. Here is a simple example chaining two coroutines: http://docs.python.org/dev/library/asyncio-task.html#example-chain-coroutines The main reason asyncio exists (as opposed to, say, twisted) is that it's supposed to be the standard place to get an event loop from, so that everybody ends up using the same event loop. Right now, though, there are still lots of competing event loops (including eventlet) and it's difficult to see how introducing another one helps in the short term. asyncio can use eventlet (greenlet in fact) event loop though greenio. asyncio is more than an event loop, it's a nice API to use it. For example, I prefer coroutines (yield from) to callbacks! Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Asynchrounous programming: replace eventlet with asyncio
Le vendredi 7 février 2014, 09:37:27 Vishvananda Ishaya a écrit : To be clear, since many people weren’t around in ye olde days, nova started using tornado. We exchanged tornado for twisted, and finally moved to eventlet. People have suggested gevent and threads in the past, and now asyncio. There are advantages to all of these other solutions, but a change at this point is going to be a huge pain, even the abstracting one you mention above. Oh, I knew that OpenStack used Twisted, but not Tornado. What were the reasons to switch from one framework to another? Is there a blueprint or summary somewhere? Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [solum] async / threading for python 2 and 3
Le vendredi 21 février 2014, 09:27:49 Angus Salkeld a écrit : Honestly, I have no answer to your question right now (How useful is trollius ...). (...) I asked your question on Tulip mailing list to see how a single code base could support Tulip (yield from) and Trollius (yield). At least check if it's technically possible. Short answer: it's possible to write code working on Trollius (Python 2 and 3) / Tulip (Python 3.3+) / CPython 3.4 (asyncio) if you use callbacks. The core of the asyncio module (event loop and scheduler) uses callbacks. If you only uses callbacks, you can also support Twisted and Tornado frameworks. For example, the AutobahnPython project adopted this design and thanks to that, it supports Trollius, Tulip and CPython asyncio, but also Twisted: https://github.com/tavendo/AutobahnPython So you have to find a WSGI server using callbacks instead of yield from. It should not be hard since asyncio is young, callbacks was the only option before greenlet/eventlet, and Twisted and Tornado (which use callbacks) are still widely used. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Python 3.3 patches (using six)
Hi, I'm working for eNovance and we are working on porting OpenStack to Python 3. Status of the port: http://techs.enovance.com/6722/status-of-the-openstack-port-to-python-3-2 https://wiki.openstack.org/wiki/Python3 I understand that it becomes late for Python 3 changes before Icehouse release, but I don't agree on the second part of your mail. David wrote: (...) I'm not sure it makes sense to do this work piecemeal until we are near ready to introduce a py3 gate job. I'm not sure that I understood correctly. You want first to see all Python 3 tests pass, and then accept changes to fix Python 3 issues? Adding a py33 gate is nice, but it is almost useless before it becomes voting if nobody reads failures. And I don't expect that anyone will care of the py33 gate before it becomes voting. It's not possible to fix all Python 3 issues at once. It requires many small changes which are carefully viewed and discussed. It is not possible to see all issues at once neither. For example, you have first to fix obvious syntax errors to see less trivial Python 3 issues. Changes are done incrementally, as other changes in OpenStack. Yes, it's possible to reintroduce Python 3 incompatible code, but I expect much fewer regressions compared to the number of fixed issues. Cyril Roelandt is improving the hacking tool to detect the most obvious cases of Python 3 incompatible code: https://review.openstack.org/#/c/80171/ We are working on clients and Olso Incubator first, but we are also preparing the port of servers. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
Hi, My patches for Oslo Messaging are naive and inefficient. They are just a first step to prepare OpenStack for Trollius. I chose to run asyncio event loop in its own dedicated thread and dispatch messages in a pool of threads. It would be nice to run asyncio event loop in the main thread, as the last blocker function of the main() function of applications. You can already pass your asyncio event loop with my patch, the thread is only created if you don't pass the loop. I wrote that my patch is inefficient because it calls Olso Messaging listener with a short timeout (ex: 500 ms) in a busy-loop. The timeout is needed to be able to exit the threads. Later, each driver should be modified to give access to the low-level file descriptor, so asyncio can react to ready to read and ready to write events. None of my Trollius patches has been merged yet. I have no experience of Trollius cooperating (or not) with eventlet yet. There are different options: - run asyncio tasks in an eventlet event loop - run eventlet tasks in an asyncio event loop - run asyncio and eventlet in two separated event loops (eventlet API is not written as an event loop), typically in two threads The best would be to use asyncio with non-blocking file descriptors, without eventlet, and register these file descriptors in asyncio. But it cannot be done right now, it requires too much changes. Sorry, I have no concrete code to show you right now. I stopped working on Trollius in OpenStack because I was asked to wait after Icehouse release. Victor In addition to the installation requirements, we also need to deal with the code-level changes. For example, when using the eventlet executor eventlet needs to have been imported very early by the application so it can monkeypatch the I/O libraries. When not using the eventlet executor, that monkeypatching should not be done because it will interfere with the regular I/O. So now we have an operation that needs to happen during code initialization that is dictated by a configuration option (which executor is being used) only available after all of the code initialization has happened. My first impression is that when we have an executor that works with asyncio/trollius we will want to drop eventlet entirely, but that's just a first impression. There may be similar trade-offs with the other libraries. Victor, what do you think? Doug On Mon, Mar 17, 2014 at 9:52 PM, Davanum Srinivas dava...@gmail.com wrote: Adrian, We are too close to icehouse release candidates to bump up global requirements with new rev of oslo.messaging. So even if we all agree and cut a new rev of oslo.messaging it's too late for icehouse as release candidates are rolling next week. I'd definitely support a way to get python33 support via trollius or thread-exec that Joshua pointed out very soon. It may be a good idea to keep solum's py33 non-voting till we nail down all other other dependencies, so +1 for that as well. -- dims On Mon, Mar 17, 2014 at 7:29 PM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? Thanks, Adrian [1] https://review.openstack.org/70948 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] WSGI and Python paste
Hi, According to the following table, paste is blocking many OpenStack servers to be ported to Python 3: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects The author of paste, Ian Bicking, gave me the commit permission to paste. I integrated patches from Debian and my colleague Cyril Roelandt, and I added even more patches. All these changes are just for the Python 3 syntax (import, except as, print, etc.). It looks like paste doesn't know anything about Python 3 and WSGI 1.0.1 (PEP ): http://legacy.python.org/dev/peps/pep-/ A function handling a web page must return bytes (b'data' in Python 3), whereas nativate string can be used in Python 2. It looks like paste is old (last release was 4 years ago, version 1.7.5.1 in 2010). Even the author of paste suggest to use something else like WebOb: Paste has been under development for a while, and has lots of code in it. Too much code! The code is largely decoupled except for some core functions shared by many parts of the code. Those core functions are largely replaced in WebOb, and replaced with better implementations. http://pythonpaste.org/future.html#introduction What is the plan for OpenStack? Should we use something else? Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Sprint at Pycon: Port OpenStack to Python 3
Hi, I will organize a sprint to Port OpenStack to Python 3 during 4 days at Montreal (Canada) during Pycon Montreal 2014, between April, 14 (Monday) and April, 17 (Thursday). The goal of the sprint is to port OpenStack components and OpenStack dependencies to Python 3, send patches to port as much code as possible. If you don't know OpenStack, you may focus more on OpenStack dependencies (MySQL-python, python-ldap, websockify, ...). This is also a good opportunity to try to replace eventlet with trollius (asyncio)! Wiki page of the sprint: https://wiki.openstack.org/wiki/Python3/SprintPycon2014 I'm writing to openstack-dev to: * know if you are interested to join the sprint * know which OpenStack components are most interested to receive and *review* Python 3 patches during the sprint * know which OpenStack components are interested to replace eventlet with trollius for the Juno release, or at least have an optional support of trollius * more generally, get your feedback :-) I know that the sprint doesn't fit well with the Icehouse release (scheduled at the last day of the sprint!). I hope that patches can be reviewed during the sprint, so developers can update them if needed during the sprint. I guess that it's fine if patches are only merged after the sprint. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
Hi, Le lundi 17 mars 2014, 09:19:03 John Eckersberg a écrit : There are a couple of known bugs that can prevent rabbitmq-server from starting on F20. First one (same bug, two BZs) is related to SELinux and port probing: https://bugzilla.redhat.com/show_bug.cgi?id=1032595#c8 https://bugzilla.redhat.com/show_bug.cgi?id=998682 I just got this issue and loose 1 hour because of it. It's not easy to identify the bug (hint: rotation of the audit log). I'm ashamed. I had exactly the same issue 6 months ago, but I forgot it and how to fix it :-( Until the issue is fixed in RabbitMQ or in the SELinux policy of RabbitMQ, I suggest this change in DevStack to warn users of the bug: https://review.openstack.org/#/c/84482/ Note: I'm running DevStack on Fedora 20. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3
Hi, Le mardi 1 avril 2014, 08:12:20 Doug Hellmann a écrit : I won't be attending PyCon this year, but count me in as a remote reviewer. Cool :-) Will you be setting up an IRC channel we should join? Hum, let's say #openstack-pycon on the Freenode server. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3
Hi, Le mardi 1 avril 2014, 09:11:52 John Dennis a écrit : What are the plans for python-ldap? Only a small part of python-ldap is pure python, are you also planning on tackling the CPython code? Oh, python-ldap was just an example, I don't have concrete plan for each dependency. We are porting dependencies since some weeks, and many have already a pending patch or pull request: https://wiki.openstack.org/wiki/Python3#Dependencies I know the Python C API and I know well all the Unicode issues, so I'm not afraid of having to hack python-ldap if it's written in C ;-) For your information, I am a Python core developer and I'm fixing Unicode issues in Python since 4 years or more :-) I also wrote a free ebook Programming with Unicode: http://unicodebook.readthedocs.org/ The biggest change in Py3 is unicode/str. The biggest pain point in the 2.x version of python-ldap is unicode -- utf-8 at the API. Currently with python-ldap we have to encode most every parameter to utf-8 before calling python-ldap and then decode the result back from utf-8 to unicode. According to the RFC 4511, LDAP speaks UTF-8 since the version 3. If the encoding is always UTF-8, it's much easier :-) I always thought this should have been done inside the python-ldap binding and it was a design failure it didn't correctly handle Python's unicode objects. FWIW the binding relied in CPython's automatic encoding conversion which applied the default encoding of ASCII which causes encoding encoding exceptions, the CPython binding just never used the correct argument processing in Py_ParseTuple() and PyParseTupleWithKeywords() which allows you to specify the desired encoding (the C API for LDAP specifies UTF-8 as does the RFC's). python-ldap can be modified to handle the unicode type in Python 2 and use UTF-8. The Py3 porting work for python-ldap is probably going to have to address the unicode changes in Py3. If the Py3 port of python-ldap brings sanity to the unicode -- utf-8 conversion then that makes a significant API change between the Py2 and Py3 versions of python-ldap making calling the python-ldap API significantly different between Py2 and Py3. Does that mean you're also planning on backporting the Py3 changes in python-ldap to Py2 to keep the API more or less consistent? When I port an application to Python 3, I'm now trying to keep the same code base for both Python versions and have the same API. I don't know python-ldap, I will have to take a look at its source code to see its current status and what should be done. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3
Le mardi 1 avril 2014, 09:44:11 John Dennis a écrit : The goal of the sprint is to port OpenStack components and OpenStack dependencies to Python 3, This is a great goal, thank you! But I'm concerned it might be premature. The portage is already in progress. There are many components (8 clients) where the py33 (Python 3.3) gate is voting. We try to keep this page up to date: https://wiki.openstack.org/wiki/Python3 There are already a lot of dependencies which are already Python 3 compatible, and the portage of OpenStack server components already started. My concern is this. The singled biggest change in Py2 - Py3 is string handling, especially with regards to str vs. unicode. We have a significant number of bugs in the current code base with regards to encoding exceptions, I just got done fixing a number of them, I know there are others. In which OpenStack component? While I was fixing them I searched the OpenStack coding guidelines to find out coding practices were supposed to be enforcing with regards to non-ASCII strings and discovered there is isn't much, it seems incomplete. Some of it seems based more on speculation than actual knowledge of defined Python behavior. I'm not sure, but given we do not have clear guidelines for unicode in Py2, never mind guidelines that will allow running under both Py2 and Py3 I'm willing to guess we have little in the gate testing that enforces any string handling guidelines. It's an ongoing effort. We are slowly porting code but also adding new tests for non-ASCII data. For example, one of my recent patch for swiftclient adds new tests for non-ASCII URLs, whereas the existing tests only test ASCII (which is irrevelant for this specific test): https://review.openstack.org/#/c/84102/ To be honest, we invest more time on fixing Python 3 issues than on adding new tests to check for non-regression. The problem is that currently, you cannot even import the Python module, so it's hard to run tests and harder to add new tests. I hope that it will become easier to run tests on Python 2 and Python 3, and to add more tests for non-ASCII data. I'm just in the process of finishing up a document to address these concerns. Unfortunately I'm going to be off-line for several weeks and I didn't want to start a discussion I couldn't participate in (plus there are some Py3 issues in the document I need to clean up) so I was going to wait to post it. There are some guidelines to port Python 2 code on Python 3 on the wiki page. You may complete it? https://wiki.openstack.org/wiki/Python3 My concern is we need to get our Py2 house in order *before* tackling Py3 porting. Doing Py3 porting before we have clear guidelines on unicode, str, bytes, encoding, etc. along with gate tests that enforce these guidelines is putting the cart before the horse. Whatever patches come out of a Py3 porting sprint might have to be completely redone. It's not easy to detect Unicode issues using Python 2 since most setup are in english, only no test using non-ASCII data right now, and Python 2 uses implicit conversion between bytes and Unicode strings. It's much easier to detect Unicode issues using Python 3. I don't want to drop Python 2 support, just *add* Python 3 support. The code will work on 2.6-3.3. FWIW projects that deal with web services, wire protocols, external datastores, etc. who have already started porting to Py3 have encountered significant pain points with Py3, some of which is just being resolved and which have caused on-going changes in Py3. We deal with a lot of these same issues in OpenStack. Before we just start hacking away I think it would behoove us to first have a very clear and explicit document on how we're going to address these issues *before* we start changing code. There are many web servers and clients already running on Python 3. For example, Django supports Python 3 since its version 1.5 (released in February 2013). I ported Paste and PasteScript on Python 3, but there are also web modules already Python 3 compatible (ex: WebOb). Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Sprint at Pycon: Port OpenStack to Python 3
Hi, Le mardi 1 avril 2014, 10:48:21 John Dennis a écrit : Oh, almost forgot. One of the significant issues in Py3 string handling occurs when dealing with the underlying OS, specifically Posix, the interaction with Posix objects such as pathnames, hostnames, environment values, etc. Virtually any place where in C you would pass a pointer to char in the Posix API where the intention is you're passing a character string. Unfortunately Posix does not enforce the concept of a character or a character string, the pointer to char ends up being a pointer to octets (e.g. binary data) which means you can end up with strings that can't be encoded. I know well these things because I worked directly in Python to have the best Unicode support for filenames on UNIX and Windows. The summary is Python 3 just works with Unicode filenames. You don't have to do anything. For example, you don't have to worry of the encoding *of the filename* for such code : for filename in os.listdir(conf/*.conf): with open(filename, rb, encoding=utf-8) as fp: ... os.listdir() and open() use the same encoding: the filesystem encoding (sys.getfilesystemencoding()) with surrogateescape error handler. If you prefer bytes filenames, there are also supported on UNIX, but deprecated on Windows. Py3 has attempted to deal with this by introducing something called surrogate escapes which attempts to preserve non-encodable binary data in what is supposed to be a character string so as not to corrupt data as it transitions between Py3 and a host OS. OpenStack deals a lot with Posix API's, thus this is another area where we need to be careful and have clear guidelines. We're going to have to deal with the whole problem of encoding/decoding in the presence of surrogates. My policy is to always store filenames as Unicode. It's easy to follow this policy since Python 3 returns Unicode filenames (ex: os.listdir(directory)). You should not have to worry of invalid filenames / surrogate characters since this case should be very rare. --- Surrogates are only used if a filename contains a non-ASCII character and the locale encoding is unable to decode it. This case should be very rare. Usually, file content can contain non-ASCII data, it's more rare for file names. I mean for a Linux server running OpenStack, but it's common for user documents on Windows for example. You only have to worry of surrogates if you want to display a filename or write filenames in a file (ex: configuration file). Another major change in Python 3 is that the UTF-8 encoder became strict: surrogate characters cannot be encoded anymore (except if you use the surrogatepass error handler). So if your locale encoding is UTF-8, print(filename) or open(test, w).write(filename) will fail if the filename contains a surrogate character. If you want to display filenames containing surrogate characters, use repr(filename) or an error handler: replace, backslashreplace or surrogateescape, depending on your use case. If you want to write filenames containing surrogate characters into a file, you might use the surrogateescape error handler, but it's probably a bad idea because we may get error when *reading* again this file. It's maybe better to raise an error if a filename is invalid (contains surrogate characters). If you system uses the ASCII locale encoding, another option is also to fix your setup configuration to use a locale using the UTF-8 encoding. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] devstack: Unable to restart rabbitmq-server
Le mardi 1 avril 2014, 12:17:39 John Eckersberg a écrit : An update has been submitted for Fedora 20 which corrects this issue: https://admin.fedoraproject.org/updates/rabbitmq-server-3.1.5-4.fc20 Please give it a try and if it works for you, be sure to leave karma on the update so it gets pushed to stable faster. Oh great! I tested it and it works with SELinux configured in Enforcing. I tried to submit a +1 using bodhi but it requests a password, and I don't know which password it is. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Issues when running unit tests in OpenStack
Hi, I have some issues when running unit tests in OpenStack. I would like to help, but I don't know where I should start and how I can fix these bugs. My use case is to run unit tests and rerun a single test if one or more tests failed. Well, it should be the most basic use case, no? (1) First problem: if a Python module cannot be loaded, my terminal is flooded with a binary stream which looks like: ... tCase.test_deprecated_without_replacement\xd7\xe1\x06\xa1\xb3)\x01@l...@atests.unit.test_versionutils.DeprecatedTestCa ... IMO it's a huge bug in the testr tool: testr run command should not write binary data into stdout. It makes development very hard. (2) When a test fails, it's hard to find the command to rerun a single failing test. Using the tool trial, I can just copy/paste the FQDN name of the failing test and run trial FQDN. Example: trial tests.unit.test_timeutils.TestIso8601Time.test_west_normalize Using the tool nosetests, you have to add a column between the module and the method. Example: nosetests tests.unit.test_timeutils:TestIso8601Time.test_west_normalize Using tox, in most OpenStack projects, adding the name of the failing test to the tox command is usually ignored. I guess that it's an issue with tox.ini of the project? tox rerun the whole test suite which is usually very slow (it takes some minutes even on fast computer). Example: tox -e py27 tests.unit.test_timeutils.TestIso8601Time.test_west_normalize I try sometimes to activate the virtualenv and then type: testr run tests.unit.test_timeutils.TestIso8601Time.test_west_normalize It usually fails for different reasons. Example with python-swiftclient. I run unit tests using tox -e py33. Some tests are failing. I enter the virtual environment and type the following command to rerun a failing test: testr run tests.test_swiftclient.TestPutObject.test_unicode_ok The test is not run again, no test is run. It's surprising because the same command works with Python 2. It's probably a bug in testr? (3) testscenarios doesn't work with nosetests. It's annoying because for the reasons listed above, I prefer to run tests using nosetests. Why do we use testscenarios and not something else? Do we plan to support nosetests (and other Python test runners) for testscenarios? Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Issues when running unit tests in OpenStack
Oh, I already got feedback from colleagues :-) Thanks. (1) First problem: if a Python module cannot be loaded, my terminal is flooded with a binary stream which looks like: ... tCase.test_deprecated_without_replacement\xd7\xe1\x06\xa1\xb3)\x01@l...@atests.unit.test_versionutils.DeprecatedTestCa ... This issue was reported: https://bugs.launchpad.net/testrepository/+bug/1271133 It looks to depend on 4 changes in 4 projects: * subunit: https://code.launchpad.net/~alexei-kornienko/subunit/bug-1271133 * testrepository: https://code.launchpad.net/~alexei-kornienko/testrepository/bug-1271133 * testtools: https://github.com/testing-cabal/testtools/pull/77 * Python unittest (for testtools): http://bugs.python.org/issue19746 (2) When a test fails, it's hard to find the command to rerun a single failing test. ... I try sometimes to activate the virtualenv and then type: testr run tests.unit.test_timeutils.TestIso8601Time.test_west_normalize In the virtual environment, the following command works and is fast: python -m testtools.run tests.test_swiftclient.TestPutObject.test_unicode_ok Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Issues when running unit tests in OpenStack
Sylvain Bauza wrote: (2) When a test fails, it's hard to find the command to rerun a single failing test. (...) See the wiki page I gave to you. Some helpful tricks are there. That said, I never had the issue you mentioned related to only checking one unittest by providing the path. When I'm isolating one test, tox -epy27 pattern is enough for me. For the parameter ignore, here is an example with Oslo Incubator (which uses testr, at least for Python 2): $ tox -e py27 ... FAIL: tests.unit.test_timeutils.TestIso8601Time.test_west_normalize ... Ran 8675 (+8673) tests in 16.137s (+16.083s) ... $ tox -e py27 tests.unit.test_timeutils.TestIso8601Time.test_west_normalize ... FAIL: tests.unit.test_timeutils.TestIso8601Time.test_west_normalize ... Ran 8675 (+8455) tests in 17.250s (-39.332s) You can run testtools without testr. Nosetests has been marked as non-supported, IIRC. Oslo Incubator runs unit tests using nosetests. I tried to use testr, but it fails on Python 3 because the openstack.common.eventlet_backdoor module cannot be loaded (eventlet is not available on Python 3). How can I ignore openstack.common.eventlet_backdoor using testr? I understood that testr first loads all modules and then filter them using the regex. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] use of the oslo namespace package
Hi, Le mardi 8 avril 2014, 10:54:24 Julien Danjou a écrit : On Mon, Apr 07 2014, Doug Hellmann wrote: We can avoid adding to the problem by putting each new library in its own package. We still want the Oslo name attached for libraries that are really only meant to be used by OpenStack projects, and so we need a naming convention. I'm not entirely happy with the crammed together approach for oslotest and oslosphinx. At one point Dims and I talked about using a prefix oslo_ instead of just oslo, so we would have oslo_db, oslo_i18n, etc. That's also a bit ugly, though. Opinions? Honestly, I think it'd be better to not have oslo at all and use independent – if possible explicit – names for everything I agree. oslo name remembers me the zope fiasco. Except of zope.interfaces and the ZODB, I don't think that any Zope module was widely used outside Zope and it was a big fail. Because of that, Zope 3 restarted almost from scratch with small independent modules. oslo and openstack.common look more and more like Zope bloated modules. For example, Oslo Incubator has 44 dependencies. Who outside OpenStack would like to use a module which has 44 dependencies? Especially if you need a single module like timeutils. nova.openstack.common.timeutils name doesn't look correct: the Zen of Python says Flat is better than nested: xxx.timeutils would be better. Same remark for oslo.config.cfg = xxx.cfg. Choosing a name is hard. Dropping oslo requires to find a completly new name. For example, oslo.config cannot be renamed to config, this name is already used on PyPI. Same issue for messaging (and message is also reserved). oslo.rootwrap can be simply renamed to rootwrap. Other suggestions: * olso.config = cmdconfig * olso.messaging = msqqueue Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] Split Oslo Incubator?
(Follow-up of the [olso] use of the oslo namespace package thread) Hi, The openstack.common module also known as Oslo Incubator or OpenStack Common Libraries has 44 dependencies. IMO we reach a point where it became too huge. Would it be possible to split it into smaller parts and distribute it on PyPI with a stable API? I don't know Olso Incubator enough to suggest the best granularity. A hint can be the number of dependencies. Sharing code is a good idea, but now we have SQLAchmey, WSGI, cryptographic, RPC, etc. in the same module. Who needs all these features at once? Olso Incubator must be usable outside OpenStack. Currently, Oslo Incubator is installed and updated manually using a update.sh script which copy .py files and replace openstack.common with nova.openstack.common (where nova is the name of the project where Oslo Incubator is installed). I guess that update.sh was written to solve the two following points, tell me if I'm wrong: - unstable API: the code changes too often, whereas users don't want to update their code regulary. Nova has maybe an old version of Olso Incubator because of that. - only copy a few files to avoid a lot of dependencies and copy useless files Smaller modules should solve these issues. They should be used as module: installed system-wide, not copied in each project. So fixing a bug would only require a single change, without having to synchronize each project. Yesterday, I proposed to add a new time_monotonic() function to the timeutils module. We asked me to enhance existing modules (like Monotime). We should now maybe move code from Oslo Incubator to upstream projects. For example, timeutils extends the iso8601 module. We should maybe contribute to this project and replace usage of timeutils with directy call to iso8601? Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Asyncio and oslo.messaging
Hi, I promise a status of my work on Trollius and greenio to Mark, but it's not easy to summarize it because there are still a few pending patches to implement the final greenio executor. There are different parts: asyncio, Trollius, greenio, Olso Messaging. The design of the asyncio is the PEP 3156 (*) which was accepted and implemented in Python 3.4, released 4 months ago. After the released of Python 3.4, many bugs were fixed in asyncio. The API is stable, it didn't change (and it cannot change because backward compatibility matters in Python, even if the module is still tagged as provisional in Python 3.4). http://legacy.python.org/dev/peps/pep-3156/ Since January, I released regulary new versions of Trollius. Trollius API is the same than the asyncio API, except of the syntax of coroutines: http://trollius.readthedocs.org/#differences-between-trollius-and-tulip The next Trollius release will probably be the version 1.0 because I consider that the API is now stable. Last incompatible changes were made to make Trollius look closer to asyncio, and to ease the transition from Trollius to asyncio. I also renamed the module from asyncio to trollius to support Python 3.4 (which already has an asyncio module in the standard library) and to make it more explicit than Trollius coroutines are different than asyncio coroutines. The greenio project was written for asyncio and it is available on PyPI. greenio only support a few features of asyncio, in short: it only supports executing coroutines. But we only need this feature in Oslo Messaging. I sent a pull request to port greenio to Trollius: https://github.com/1st1/greenio/pull/5/ The pull request requires a new task factory: I sent a patch to asyncio for that. For Oslo Messaging, my change to poll with a timeout has been merged. (I just sent a fix because my change doesn't work with RabbitMQ.) I will work on the greenio executor when other pending patches will be merged. We talked with Mark about this greenio executor. It will be based on the eventlet executor, with a few lines to support Trollius coroutines. We also have to modify the notifier to support to pass an optional execute function which executes the endpoint function which may be a coroutine. According to Mark, this change is short and acceptable in Olso Messaging: thanks to the execute function, it will be possible to restrict code using greenio in the greenio executor (no need to put greenio nor trollius everywhere in Oslo Messaging). I listed a lot of projects and pending patches, but I expect that all pieces of the puzzle with be done before the end of the month. We are very close to having a working greenio executor in Oslo Messaging ;-) Victor - Mail original - De: Mark McLoughlin mar...@redhat.com À: openstack-dev@lists.openstack.org Envoyé: Jeudi 3 Juillet 2014 17:27:58 Objet: [openstack-dev] [oslo] Asyncio and oslo.messaging Hey This is an attempt to summarize a really useful discussion that Victor, Flavio and I have been having today. At the bottom are some background links - basically what I have open in my browser right now thinking through all of this. We're attempting to take baby-steps towards moving completely from eventlet to asyncio/trollius. The thinking is for Ceilometer to be the first victim. Ceilometer's code is run in response to various I/O events like REST API requests, RPC calls, notifications received, etc. We eventually want the asyncio event loop to be what schedules Ceilometer's code in response to these events. Right now, it is eventlet doing that. Now, because we're using eventlet, the code that is run in response to these events looks like synchronous code that makes a bunch of synchronous calls. For example, the code might do some_sync_op() and that will cause a context switch to a different greenthread (within the same native thread) where we might handle another I/O event (like a REST API request) while we're waiting for some_sync_op() to return: def foo(self): result = some_sync_op() # this may yield to another greenlet return do_stuff(result) Eventlet's infamous monkey patching is what make this magic happen. When we switch to asyncio's event loop, all of this code needs to be ported to asyncio's explicitly asynchronous approach. We might do: @asyncio.coroutine def foo(self): result = yield from some_async_op(...) return do_stuff(result) or: @asyncio.coroutine def foo(self): fut = Future() some_async_op(callback=fut.set_result) ... result = yield from fut return do_stuff(result) Porting from eventlet's implicit async approach to asyncio's explicit async API will be seriously time consuming and we need to be able to do it piece-by-piece. The question then becomes what do we need to do in order to port a single oslo.messaging RPC endpoint method in Ceilometer to
Re: [openstack-dev] [oslo] Asyncio and oslo.messaging
Hi, Le lundi 7 juillet 2014, 12:48:59 Nikola Đipanov a écrit : When I read all of this stuff and got my head around it (took some time :) ), a glaring drawback of such an approach, and as I mentioned on the spec proposing it [1] is that we would not really doing asyncio, we would just be pretending we are by using a subset of it's APIs, and having all of the really important stuff for overall design of the code (code that needs to do IO in the callbacks for example) and ultimately - performance, completely unavailable to us when porting. The global plan is to: 1. use asyncio API 2. detect code relying on implicit scheduling and patch it to use explicit scheduling (use the coroutine syntax with yield) 3. just change the event loop from greenio to a classic select event loop (select, poll, epoll, kqueue, etc.) of Trollius I see asyncio as an API: it doesn't really matter which event loop is used, but I want to get rid of eventlet :-) So in Mark's example above: @asyncio.coroutine def foo(self): result = yield from some_async_op(...) return do_stuff(result) A developer would not need to do anything that asyncio requires like make sure that some_async_op() registers a callback with the eventloop (...) It's not possible to break the world right now, some people will complain :-) The idea is to have a smooth transition. We will write tools to detect implicit scheduling and fix code. I don't know the best option for that right now (monkey-patch eventlet, greenio or trollius?). So I hacked up together a small POC of a different approach. In short - we actually use a real asyncio selector eventloop in a separate thread, and dispatch stuff to it when we figure out that our callback is in fact a coroutine. See my previous attempty: the asyncio executor runs the asyncio event loop in a dedicated thread: https://review.openstack.org/#/c/70948/ I'm not sure that it's possible to use it in OpenStack right now because the whole Python standard library is monkey patched, including the threading module. The issue is also to switch the control flow between the event loop thread and the main thread. There is no explicit event loop in the main thread. The most obvious solution for that is to schedule tasks using eventlet... That's exactly the purpose of greenio: glue between asyncio and greenlet. And using greenio, there is no need of running a new event loop in a thread, which makes the code simpler. (..) we would probably not be 'greening the world' but rather importing patched non-ported modules when we need to dispatch to them. This may sound like a big deal, and it is, but it is critical to actually running ported code in a real asyncio evenloop. It will probably require a lot of work to get rid of eventlet. The greenio approach is more realistic because projects can be patched one by one, one file by one file. The goal is also to run projects unmodified with the greenio executor. Another interesting problem is (as I have briefly mentioned in [1]) - what happens when we need to synchronize between eventlet-run and asyncio-run callbacks while we are in the process of porting. Such issue is solved by greenio. As I wrote, it's not a good idea to have two event loops in the same process. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Asyncio and oslo.messaging
Le lundi 7 juillet 2014, 11:26:27 Gordon Sim a écrit : When we switch to asyncio's event loop, all of this code needs to be ported to asyncio's explicitly asynchronous approach. We might do: @asyncio.coroutine def foo(self): result = yield from some_async_op(...) return do_stuff(result) or: @asyncio.coroutine def foo(self): fut = Future() some_async_op(callback=fut.set_result) ... result = yield from fut return do_stuff(result) Porting from eventlet's implicit async approach to asyncio's explicit async API will be seriously time consuming and we need to be able to do it piece-by-piece. Am I right in saying that this implies a change to the effective API for oslo.messaging[1]? I.e. every invocation on the library, e.g. a call or a cast, will need to be changed to be explicitly asynchronous? [1] Not necessarily a change to the signature of functions, but a change to the manner in which they are invoked. The first step is to patch endpoints to add @trollius.coroutine to the methods, and add yield From(...) on asynchronous tasks. Later we may modify Oslo Messaging to be able to call an RPC method asynchronously, a method which would return a Trollius coroutine or task directly. The problem is that Oslo Messaging currently hides implementation details like eventlet. Returning a Trollius object means that Oslo Messaging will use explicitly Trollius. I'm not sure that OpenStack is ready for that today. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Asyncio and oslo.messaging
Le lundi 7 juillet 2014, 19:14:45 Angus Salkeld a écrit : 4) retraining of OpenStack developers/reviews to understand the new event loop. (eventlet has warts, but a lot of devs know about them). Wait, what? Sorry if it was not clear, but the *whole* point of replacing eventlet with asyncio is to solve the most critical eventlet issue: eventlet may switch to another greenthread *anywhere* which causes dangerous and very complex bugs. I asked different OpenStack developers: almost nobody in OpenStack is able to understand these issues nor fix them. Most OpenStack are just not aware of the issue. A colleague told me that he's alone to fix eventlet bugs, and nobody else cares because he's fixing them... Read also the What's wrong with eventlet ? section of my article: http://techs.enovance.com/6562/asyncio-openstack-python3 eventlet does not support Python 3 yet. They made some progress, but it is not working yet (at least in Oslo Incubator). eventlet is now the dependency blocking most OpenStack projects to port them to Python 3: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Once we are in asyncio heaven, would we look back and say it would have been more valuable to focus on X, where X could have been say ease-of-upgrades or general-stability? I would like to work on asyncio, I don't force anyone to work on it. It's not like they are only 2 developers working on OpenStack :-) OpenStack already evolves fast enough (maybe too fast according to some people!). It looks like other developers are interested to help me. Probably because they want to get rid of eventlet for the reason I just explained. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Asyncio and oslo.messaging
Le jeudi 10 juillet 2014, 14:48:04 Yuriy Taraday a écrit : I'm not suggesting that taskflow is useless and asyncio is better (apple vs oranges). I'm saying that using coroutines (asyncio) can improve ways we can use taskflow and provide clearer method of developing these flows. This was mostly response to the this is impossible with coroutines. I say it is possible and it can even be better. It would be nice to modify taskflow to support trollius coroutines. Coroutines supports asynchronous operations and has a better syntax than callbacks. For Mark's spec, add a new greenio executor to Oslo Messaging: I don't see the direct link to taskflow. taskflow can use Oslo Messaging to call RPC, but I don't see how to use taskflow internally to read a socket (driver), wait for the completion of the callback and then send back the result to the socket (driver). I see trollius as a low-level tool to handle simple asynchronous operations, whereas taskflow is more high level to chain correctly more complex operations. trollius and taskflow must not be exclusive options, they should cooperate, as we plan to support trollius coroutines in Oslo Messaging. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Asyncio and oslo.messaging
Hi, Le lundi 7 juillet 2014, 19:18:38 Mark McLoughlin a écrit : I'd expect us to add e.g. @asyncio.coroutine def call_async(self, ctxt, method, **kwargs): ... to RPCClient. Perhaps we'd need to add an AsyncRPCClient in a separate module and only add the method there - I don't have a good sense of it yet. I don't want to make trollius a mandatory dependency of Oslo Messaging, at least not right now. An option is to only declare the method if trollius is installed. try: import trollius except ImportError: trollius = None and then if trollius is not None: @trollius.coroutine def cal_async(): Or maybe a different module (maybe using a subclass) is better. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] run tests using testr on Python 3
Hi, Olso Incubator runs tests using testr on Python 2, but it uses nosetests on Python 3 to only run a subset of the test suite (modules and tests ported to Python 3). In my latest patch for Oslo Incubator (gettext), Ben Nemec wrote: I think we could get around the nose issue by using a testr regex to filter the tests we run for py33 (like I did in py27 for running everything but rpc in parallel), but that's outside the scope of this change so +2. I tried to run Olso Incubator tests with testr on Python 3, but testr fails to load openstack.common.eventlet_backdoor, because eventlet module is not installed (eventlet is not Python 3 compatible yet). If I understood correctly, testr first loads all modules and then filter the tests to run using the regex passed on the command line. If I'm correct, I don't see right now how to run Olso Incubator tests with testr on Python 3. But I don't know well the Testr tool, so I missed probably an option. I would like to use testr because many Olso Incubator tests use testscenarios (which doesn't work with nosetests). By the way, would it be possible to fix nosetests to use testscenarios? Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Split Oslo Incubator?
Hi, Le mardi 8 avril 2014, 10:05:31 Doug Hellmann a écrit : The openstack.common module also known as Oslo Incubator or OpenStack Common Libraries has 44 dependencies. IMO we reach a point where it became too huge. Would it be possible to split it into smaller parts and distribute it on PyPI with a stable API? I don't know Olso Incubator enough to suggest the best granularity. A hint can be the number of dependencies. Yes, as others have pointed out we will be doing this in Juno. See https://wiki.openstack.org/wiki/Oslo/JunoGraduationPlans I talk with Julien Danjou on Hangout. He explained me the origins of the code and that everyone knows that Oslo Incubator can be improved, and even how it can be improved, but this project always lacks workforce. Good news! I'm interested to help to enhance Olso Incubator! That may make sense for your new function. I think there are some other things in timeutils that don't make sense to upstream. The isotime() and parse_isotime() functions are relatively simple wrappers around existing functions that give us consistent timestamps across projects, for example. Those are useful to us as OpenStack developers, but I'm not sure they're useful to anyone else as written. I now understood that each submodule, or even each function, of Olso Incubator should be audited to check if it is still used, if it should be removed, if it should move upstream, or if a module can be distributed separatly on the Python Cheeseshop (PyPI). Sorry, I was not aware of the Oslo Juno Library Graduation Plan, I will read it! https://wiki.openstack.org/wiki/Oslo/JunoGraduationPlans Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][db] oslo.db repository review request
Hi, Le vendredi 18 avril 2014, 17:28:04 Victor Sergeyev a écrit : During Icehouse release cycle our team has been working on splitting of openstack common db code into a separate library blueprint [1]. At the moment the issues, mentioned in this bp and [2] are solved and we are moving forward to graduation of oslo.db. You can find the new oslo.db code at [3] There was a discussion recently about the oslo namespace. Sorry, but I don't remember the decision. Should the new module be called oslo.db or olsodb? (I would prefer oslodb, it avoids issues with the oslo package.) Victor (Stinner aka haypo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][db] oslo.db repository review request
Le mardi 13 mai 2014, 07:31:34 Doug Hellmann a écrit :
Since we think we have been able to solve all of the issues we were
having with namespace packages before, ...
I just tried to start my DevStack and again, I had issues with a builtin
olso module: import oslo.config doesn't work, whereas olso.config was
installed (system wide) by pip.
pip list|grep olso told me that oslo.config, oslo.messaging, oslo.rootwrap
and oslo.vmware are installed.
My workaround is to uninstall all olso modules:
sudo pip uninstall oslo.config oslo.messaging oslo.rootwrap oslo.vmware
./stack.sh reinstalls them and now it works.
--
Current state:
haypo@devstackdev$ pip list|grep oslo
oslo.config (1.3.0a0.40.gb347519)
oslo.messaging (1.3.0.8.gc0c8557)
oslo.rootwrap (1.2.0)
oslo.vmware (0.3.1.g49097c0)
haypo@devstackdev$ python
Python 2.7.5 (default, Feb 19 2014, 13:47:28)
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2
Type help, copyright, credits or license for more information.
import oslo
oslo
module 'oslo' (built-in)
import oslo.config
import oslo.messaging
import oslo.rootwrap
import oslo.vmware
I never understood how these .pth files work.
haypo@devstackdev$ cd /usr/lib/python2.7/site-packages
haypo@devstackdev$ ls oslo*.pth -1
oslo.config-1.3.0a0.40.gb347519-py2.7-nspkg.pth
oslo.messaging-1.3.0.8.gc0c8557-py2.7-nspkg.pth
oslo.rootwrap-1.2.0-py2.7-nspkg.pth
oslo.vmware-0.3.1.g49097c0-py2.7-nspkg.pth
haypo@devstackdev$ md5sum oslo*.pth
002fd4bf040a30d396d4df8e1ed378a8 oslo.config-1.3.0a0.40.gb347519-py2.7-
nspkg.pth
002fd4bf040a30d396d4df8e1ed378a8 oslo.messaging-1.3.0.8.gc0c8557-py2.7-
nspkg.pth
002fd4bf040a30d396d4df8e1ed378a8 oslo.rootwrap-1.2.0-py2.7-nspkg.pth
002fd4bf040a30d396d4df8e1ed378a8 oslo.vmware-0.3.1.g49097c0-py2.7-nspkg.pth
haypo@devstackdev$ cat oslo.config-1.3.0a0.40.gb347519-py2.7-nspkg.pth
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('oslo',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie
and sys.modules.setdefault('oslo',types.ModuleType('oslo')); mp = (m or [])
and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
Victor
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] Logging exceptions and Python 3
Hi, I'm trying to define some rules to port OpenStack code to Python 3. I just added a section in the Port Python 2 code to Python 3 about formatting exceptions and the logging module: https://wiki.openstack.org/wiki/Python3#logging_module_and_format_exceptions The problem is that I don't know what is the best syntax to log exceptions. Some projects convert the exception to Unicode, others use str(). I also saw six.u(str(exc)) which is wrong IMO (it can raise unicode error if the message contains a non-ASCII character). IMO the safest option is to use str(exc). For example, use LOG.debug(str(exc)). Is there a reason to log the exception as Unicode on Python 2? Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] strutils: enhance safe_decode() and safe_encode()
Hi, The functions safe_decode() and safe_encode() have been ported to Python 3, and changed more than once. IMO we can still improve these functions to make them more reliable and easier to use. (1) My first concern is that these functions try to guess user expectation about encodings. They use sys.stdin.encoding or sys.getdefaultencoding() as the default encoding to decode, but this encoding depends on the locale encoding (stdin encoding), on stdin (is stdin a TTY? is stdin mocked?), and on the Python major version. IMO the default encoding should be UTF-8 because most OpenStack components expect this encoding. Or maybe users want to display data to the terminal, and so the locale encoding should be used? In this case, locale.getpreferredencoding() would be more reliable than sys.stdin.encoding. (2) My second concern is that safe_encode(bytes, incoming, encoding) transcodes the bytes string from incoming to encoding if these two encodings are different. When I port code to Python 3, I'm looking for a function to replace this common pattern: if isinstance(data, six.text_type): data = data.encode(encoding) I don't want to modify data encoding if it is already a bytes string. So I would prefer to have: def safe_encode(data, encoding='utf-8'): if isinstance(data, six.text_type): data = data.encode(encoding) return data Changing safe_encode() like this will break applications relying on the transcode feature (incoming = encoding). If such usage exists, I suggest to add a new function (ex: transcode ?) with an API fitting this use case. For example, the incoming encoding would be mandatory. Is there really applications using the incoming parameter? So, what do you think about that? Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Logging exceptions and Python 3
Le vendredi 16 mai 2014, 06:03:53 Johannes Erdfelt a écrit : On Fri, May 16, 2014, Igor Kalnitsky ikalnit...@mirantis.com wrote: According to http://legacy.python.org/dev/peps/pep-0352/ the message attribute of BaseException is deprecated since Python 2.6 and was dropped in Python 3.0. Some projects have custom exception hierarchy, with strictly defined attributes (e.g. message, or something else). In a previous mail, I mean exactly that case, not the case with a built-in exceptions. That's a fragile assumption to make. unicode(exc) (or six.text_type(exc)) works for all exceptions, built-in or custom. I don't see the reason why it's being avoided. See my documentation: https://wiki.openstack.org/wiki/Python3#logging_module_and_format_exceptions six.text_type(exc): always use Unicode. It may raise unicode error depending on the exception, be careful. Example of such error in python 2: unicode(Exception(nonascii:\xe9)). unicode(exc) works with such exception classes: --- class MyException1(Exception): pass exc = MyException1() exc.message = u\u20ac unicode(exc) #ok class MyException2(Exception): def __unicode__(self): return u\20ac exc = MyException2() unicode(exc) #ok --- If we want to format an exception as Unicode, we need a function trying unicode(), or use str() and then guess the encoding. It means adding a new safe function to Olso to format an exception. Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Mehdi Abaakouk added to oslo.messaging-core
Le vendredi 6 juin 2014, 15:57:09 Mark McLoughlin a écrit : Mehdi has been making great contributions and reviews on oslo.messaging for months now, so I've added him to oslo.messaging-core. Thank you for all your hard work Mehdi! Congrats Mehdi :-) Victor ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging
Hi, I'm happy to announce you that I just finished the last piece of the puzzle to add support for trollius coroutines in Oslo Messaging! See my two changes: * Add a new aiogreen executor: https://review.openstack.org/#/c/136653/ * Add an optional executor callback to dispatcher: https://review.openstack.org/#/c/136652/ Related projects: * asyncio is an event loop which is now part of Python 3.4: http://docs.python.org/dev/library/asyncio.html * trollius is the port of the new asyncio module to Python 2: http://trollius.readthedocs.org/ * aiogreen implements the asyncio API on top of eventlet: http://aiogreen.readthedocs.org/ For the long story and the full history of my work on asyncio in OpenStack since one year, read: http://aiogreen.readthedocs.org/openstack.html The last piece of the puzzle is the new aiogreen project that I released a few days ago. aiogreen is well integrated and fully compatible with eventlet, it can be used in OpenStack without having to modify code. It is almost fully based on trollius, it just has a small glue to reuse eventlet event loop (get read/write notifications of file descriptors). In the past, I tried to use the greenio project, which also implements the asyncio API, but it didn't fit well with eventlet. That's why I wrote a new project. Supporting trollius coroutines in Oslo Messaging is just the first part of the global project. Here is my full plan to replace eventlet with asyncio. First part (in progress): add support for trollius coroutines - Prepare OpenStack (Oslo Messaging) to support trollius coroutines using ``yield``: explicit asynchronous programming. Eventlet is still supported, used by default, and applications and libraries don't need to be modified at this point. Already done: * Write the trollius project: port asyncio to Python 2 * Stabilize trollius API * Add trollius dependency to OpenStack * Write the aiogreen project to provide the asyncio API on top of eventlet To do: * Stabilize aiogreen API * Add aiogreen dependency to OpenStack * Write an aiogreen executor for Oslo Messaging: rewrite greenio executor to replace greenio with aiogreen Second part (to do): rewrite code as trollius coroutines Switch from implicit asynchronous programming (eventlet using greenthreads) to explicit asynchronous programming (trollius coroutines using ``yield``). Need to modify OpenStack Common Libraries and applications. Modifications can be done step by step, the switch will take more than 6 months. The first application candidate is Ceilometer. The Ceilometer project is young, developers are aware of eventlet issues and like Python 3, and Ceilometer don't rely so much on asynchronous programming: most time is spent into waiting the database anyway. The goal is to port Ceilometer to explicit asynchronous programming during the cycle of OpenStack Kilo. Some applications may continue to use implicit asynchronous programming. For example, nova is probably the most complex part beacuse it is and old project with a lot of legacy code, it has many drivers and the code base is large. To do: * Ceilometer: add trollius dependency and set the trollius event loop policy to aiogreen * Ceilometer: change Oslo Messaging executor from eventlet to aiogreen * Redesign the service class of Oslo Incubator to support aiogreen and/or trollius. Currently, the class is designed for eventlet. The service class is instanciated before forking, which requires hacks on eventlet to update file descriptors. * In Ceilometer and its OpenStack depedencencies: add new functions which are written with explicit asynchronous programming in mind (ex: trollius coroutines written with ``yield``). * Rewrite Ceilometer endpoints (RPC methods) as trollius coroutines. Questions: * What about WSGI? aiohttp is not compatible with trollius yet. * The quantity of code which need to be ported to asynchronous programming is unknown right now. * We should be prepared to see deadlocks. OpenStack was designed for eventlet which implicitly switch on blocking operations. Critical sections may not be protected with locks, or not the right kind of lock. * For performances, blocking operations can be executed in threads. OpenStack code is probably not thread-safe, which means new kinds of race conditions. But the code executed in threads will be explicitly scheduled to be executed in a thread (with ``loop.run_in_executor()``), so regressions can be easily identified. * This part will take a lot of time. We may need to split it into subparts to have milestones, which is more attractive for developers. Last part (to do): drop eventlet Replace aiogreen event loop with trollius event loop, drop aiogreen and drop eventlet at the end. This change will be done on applications one by one. This is no need to
Re: [openstack-dev] Python 3 is dead, long live Python 3
Hi, It's good to move forward to Python 3.4 :-) [2] https://launchpad.net/bugs/1367907 This bug was introduced in Python 3.4.0 and fixed in Python 3.4.1. It's too bad that Ubunbu Trusty didn't upgraded yet Python 3.4 to 3.4.1 (released 6 months ago) or 3.4.2. Request to upgrade python 3.4 in Ubuntu Trusty: https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1348954 (upgrade already scheduled at the end of february, after the release of python 3.4.3) Debian Testing (Jessie) and Unstable (Sid) provide Python 3.4.2. Debian Stable (Wheezy) only provides Python 3.2.3 (which doesn't accept uunicode syntax :-/ and doesn't support yield-from). Fedora 21 provides Python 3.4.1. (Fedora 20 only provides Python 3.3.2). Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
Hi, What is the Python bug? Do you have a reference to the bug report and the patch? Python 3.4.3 release schedule: 3.4.3rc1 will be tagged Saturday February 7 and released Sunday February 8. 3.4.3 final will follow two weeks later, tagged Saturday February 21 and released Sunday February 22. https://mail.python.org/pipermail/python-dev/2015-January/137773.html Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Replace eventlet with asyncio
/bench_generator.py`_, consuming a generator takes 951 ns whereas calling a function takes 228 ns: consuming a generator is 4.2x slower than calling a function. Basically, the microbenchmark measures the time to raise an exception (``StopIteration``) and then to catch it: 723 nanoseconds. A database query typically takes 50 ms or less. Other deployer impact - NA Developer impact To write efficient code, developers have to learn how to write asyncio code, but only on functions which must be asynchronous. Only projects which chose to use asyncio will have to be modified. Other projects are free to continue to use eventlet. Implementation == Assignee(s) --- Assignee is for moving these guidelines through the review process to something that we all agree on. The expectation is that these become review criteria that we can reference and are implemented by a large number of people. Once approved, will also drive collecting volunteers to help fix in multiple projects. Primary assignee: Victor Stinner vstin...@redhat.com Work Items -- Work items or tasks -- break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we're mostly trying to understand the time-line for implementation. Recently merged changes: * `Add aioeventlet dependency https://review.openstack.org/#/c/138750/`_ * `Add a new aioeventlet executor https://review.openstack.org/#/c/136653/`_: already approved Dependencies The implementation requires a new dependency: the ``aioeventlet`` module. It is already added to global requirements. The ``trollius`` module was already added to global requirements. Testing === NA Documentation Impact NA Comparison of eventlet and asyncio code === Call a function --- eventlet:: evenlet.spawn(func, arg) asyncio:: loop.call_soon(func, arg) Schedule a function in 10 seconds - eventlet:: eventlet.spawn_after(10, func, arg) asyncio:: loop.call_later(10, func, arg) Asynchronous task - eventlet:: def async_multiply(arg): # interrupt the execution of the current greenthread eventlet.sleep(1.0) return arg * 2 def func(): gt = eventlet.spawn(async_multiply, 5) # block the current greenthread result = gt.wait() print(5 * 2 = %s % result) asyncio:: @asyncio.coroutine def async_multiply(arg): # interrupt the execution of the current task yield from asyncio.sleep(1.0) return arg * 2 @asyncio.coroutine def func(): # block the current coroutine result = yield from async_multiply(5) print(5 * 2 = %s % result) asyncio === Trollius: asyncio port to Python 2 -- asyncio requires Python 3.3 and newer. asyncio was ported to Python 2 in a new project called `trollius http://trollius.readthedocs.org/`_. Changes made in asyncio are merged in trollius, trollius is a branch of the mercurial repository of tulip (asyncio upstream). The major difference between Trollius and Tulip is the syntax of coroutines: == == Tulip Trollius == == ``yield from ...`` ``yield From(...)`` ``yield from []`` ``yield From(None)`` ``return`` ``raise Return()`` ``return x````raise Return(x)`` ``return x, y`` ``raise Return(x, y)`` == == It is possible to write code working with trollius and asyncio in the same code base if coroutines are not used, but only callbacks and futures. Some libraries already support asyncio and trollius like AutobahnPython (Websockets and WAMP), Pulsar and Tornado. Another option is to provide functions returning ``Future`` objects, so the caller can decide to use callback using ``fut.add_done_callback(callback)`` or to use coroutines (``yield From(fut)`` for Trollius, or ``yield from fut`` for Tulip). This option is used by the `aiodns https://github.com/saghul/aiodns`_ project for example. On Python 3.3 and newer, Trollius supports also asyncio coroutines. The trollius module is compatible with asyncio, the opposite is false. Trollius works on Python 2.6-3.5. aioeventlet: asyncio API on top of eventlet --- In OpenStack, eventlet cannot be replaced with asyncio in all projects in a single commit. The OpenStack development is made with small and limited changes. To have a smooth transition, the `aioeventlet project http://aioeventlet.readthedocs.org/`_ was written to support the asyncio API on top of eventlet. It makes possible to add support for asyncio coroutines to an existing OpenStack component without having to replace immediatly its ``main
[openstack-dev] [oslo] Progress of the port to Python 3
Hi, I'm still working on porting OpenStack to Python 3. I'm fixing Python 3 issues in eventlet and I'm trying to replace eventlet with trollius in OpenStack. Two different and complementary ways to port OpenStack to Python 3. I fixed some eventlet issues related to Python 3 and monkey-patching. My changes are part of eventlet 0.16 which was released the 2014-12-30. I tried this version with Oslo Messaging: with another fix in eventlet (threading.Condition) and a change in the zmq driver, only 3 tests are now failing (which is a great success for me). See my Oslo Messaging change for more information: https://review.openstack.org/#/c/145241/ It looks like eventlet 0.16 works much better with Python 3 and monkey-patching. You should try it on your own project! Tell me if you need help to fix issues. -- About asyncio, I renamed my aiogreen project to aioeventlet (to make it more explicit that it is specific to eventlet). With the release 0.4, the API is now considered as stable. http://aioeventlet.readthedocs.org/ Mehdi Abaakouk voted +2 on my Oslo Messaging patches to support trollius coroutines using aioeventlet project, but it looks that he's alone to review Oslo Messaging patches :-/ Anyway to approve my changes? * Add an optional executor callback to dispatcher https://review.openstack.org/136652 * Add a new aioeventlet executor https://review.openstack.org/136653 By the way, my review to add aioeventlet dependency also waits for a review: * Add aioeventlet dependency https://review.openstack.org/138750 Victor Stinner ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] requirements-py{2, 3} and universal wheels
I haven't tested yet (and someone should) that it does all JUST WORK, but thats easy: put an environment marker in a requirements.txt file like so: argparse; python_version '3' I think the last time this came up the feature wasn't available in pip yet, and so using separate files was the work-around. Are environment markers fully supported by pip/setuptools/whatever now? Yeah, I developed this feature for OpenStack. My change was merged into pip 6.0: https://github.com/pypa/pip/pull/1472 I forgot to finish the work in OpenStack. I don't know where the pip requirement is specified: we need at least pip 6.0. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues
Hi, Davanum Srinivas wrote: +1 to fix Oslo's service module any ways, irrespective of this bug. By the way, the Service class is a blocker point for the implementation of asyncio and threads specs: https://review.openstack.org/#/c/153298/ https://review.openstack.org/#/c/156711/ We may allow to execute a function before fork() to explicitly share some things with all child processes. But most things (instanciate the application, open DB connections, etc.) should be done after the fork. Well, it looks like everyone agree. We just need someone to implement the idea :-) We may write a new class instead of modifying the existing class, to not break applications. Doug Hellamnn even proposed once to have an abstraction of the concurrency model (eventlet, threads, asyncio). I don't know if it's worth it. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Replace eventlet with asyncio
Hi, I also just put up another proposal to consider: https://review.openstack.org/#/c/156711/ Sew over eventlet + patching with threads My asyncio spec is unclear about WSGI, I just wrote The spec doesn't change OpenStack components running WSGI servers like nova-api. The specific problem of using asyncio with WSGI will need a separated spec. Joshua's threads spec proposes: I would prefer to let applications such as apache or others handle the request as they see fit and just make sure that our applications provide wsgi entrypoints that are stateless and can be horizontally scaled as needed (aka remove all eventlet and thread ... semantics and usage from these entrypoints entirely). Keystone wants to do the same: https://review.openstack.org/#/c/157495/ Deprecate Eventlet Deployment in favor of wsgi containers This deprecates Eventlet support in documentation and on invocation of keystone-all. I agree: we don't need concurrency in the code handling a single HTTP request: use blocking functions calls. You should rely on highly efficient HTTP servers like Apache, nginx, werkzeug, etc. There is a lot of choice, just pick your favorite server ;-) Each HTTP request is handled in a thread. You can use N processes and each process running M threads. It's a common architecture design which is efficient. For database accesses, just use regular blocking calls (no need to modify SQLAchemy). According to Mike Bayer's benchmark (*), it's even the fastest method if your code is database intensive. You may share a pool of database connections between the threads, but a connection should only be used by a single thread. (*) http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ I don't think that we need a spec if everybody already agree on the design :-) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Test failures due to oslo_concurrency?
See also the following changes: oslo.concurrency: Don't decode stdout/stderr on Python 2 https://review.openstack.org/#/c/146984/ (my change) oslo.concurrency: Allow callers to disable encoding for execute() https://review.openstack.org/#/c/146932/ (Doug Hellmann) nova: Do not decode binary data from crypt commands https://review.openstack.org/#/c/146937/ (Doug Hellmann) nova: use -out for openssl encrypted blobs https://review.openstack.org/#/c/146929/ (Sean Dague) Victor - Mail original - De: Kevin L. Mitchell kevin.mitch...@rackspace.com À: openstack-dev openstack-dev@lists.openstack.org Envoyé: Mardi 13 Janvier 2015 22:20:35 Objet: [openstack-dev] [oslo] Test failures due to oslo_concurrency? Noticed that https://review.openstack.org/#/c/145626/ failed tests. That's kind of curious, as all it does is alter HACKING.rst. Investigation of the test failures[1] showed some UTF-8 decoding errors that appeared to be coming from oslo_concurrency/processutils.py. Any ideas? [1] http://logs.openstack.org/26/145626/2/check/gate-nova-python27/b350bbc/console.html -- Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] freeze exception
Hi, I had like to request a freeze exception for this nova fix: https://review.openstack.org/#/c/164713/ Related bug: https://bugs.launchpad.net/nova/+bug/1406486 Suspending an instance fails when using vnic_type=direct My change is a backport from master to juno of the fix already merged into kilo. The backport didn't conflict, the change is the same: https://review.openstack.org/#/c/148096/ I compared manually using two tabs in Firefox :-) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Also, on the Python 3 topic, there's still a big issue with memcached (aka: python-memcache). Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3: https://github.com/linsomniac/python-memcached/pull/67 I don't understand. I saw a lot of Port to Python 3 fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed. Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client. It's blocking me from adding Python3 support to keystoneclient, and as a consequence, to almost all of OpenStack. python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, tox -e py34 pass. The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run! Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][pbr] getting 0.11 out the door. Maybe even 1.0
I don't think its a bug in the applications. swift gets its version using pkg_resources, or falls back to pbr: https://github.com/openstack/swift/blob/master/swift/__init__.py I mean that other applications may do something similar? Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][pbr] getting 0.11 out the door. Maybe even 1.0
I believe Redhat patch it out. I don't think they should need to, since we have explicit knobs for distros to use. pbr pulls pip which we don't want in RHEL. Example of patches in RDO: https://github.com/redhat-openstack/nova/commit/a19939c8f9a7b84b8a4d713fe3d26949e5664089 https://github.com/redhat-openstack/python-keystoneclient/commit/e02d529a87aef8aaca0616c8ee81de224bf1f52a https://github.com/redhat-openstack/neutron/commit/85302b75362df30270383e3a373e60e81b1b2384 (well, it's always the same change) Can't we enhance pbr to build (source/wheel) distributions of applications which don't depend on pbr? Basically implement these patches in pbr? I read somewhere that pkg_resources may also be used to get the version. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][pbr] getting 0.11 out the door. Maybe even 1.0
If there's nothing majorly wrong mid-L, I'd like to release 1.0.0 just to get us into 'ok its stable' mentality. I read that many packages modify the source code of libraries and applications to avoid a dependency to pbr at runtime. What's the status of this issue? Is pbr still used/required to get the version of a package a runtime? I'm not sure that it's an issue in pbr itself. Maybe applications should be fixed instead. (Sorry if it was already discussed and I missed the conclusion.) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] Port Nova to Python 3
Hi, Porting OpenStack applications during the Liberty Cycle was discussed last days in the thread [oslo] eventlet 0.17.3 is now fully Python 3 compatible. I wrote a spec to port Nova to Python 3: https://review.openstack.org/#/c/176868/ I mentioned the 2 other Python 3 specs for Neutron and Heat. You can reply to this email, or comment the review, if you want to discuss Python 3 in Nova, or if you have any question related to Python 3. See also the Python 3 wiki page: https://wiki.openstack.org/wiki/Python3 Thanks, Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Port Nova to Python 3
Hi, If written to use python 3, I hope it will use all the new features of python 3.4 moving forward. The spec adds Python 3 support, but it keeps Python 2 support. It's too early to drop Python 2, Nova is deployed everywhere with Python 2. For example, argument annotations, coroutines, asyncio, etc. Argument annotations are not used in practice :-( There is a PEP under review which targets the future Python 3.5 version: https://www.python.org/dev/peps/pep-0484/ I'm working actively on asyncio. I wrote a spec to replace eventlet with asyncio: https://review.openstack.org/#/c/153298/ superseded by: https://review.openstack.org/#/c/164035/ For OpenStack, I ported asyncio to Python 2: it's the Trollius project: https://trollius.readthedocs.org/ I would also prefer to be able to use new shiny Python 3 features, but it's not possible yet. We have to move step by step. There is no choice like Python 2 only or Python 3 only with new Python 3 features. Changes must be done incrementally in OpenStack. We all know that. At my last workplace, we tried to make our project python2 and 3 compatible (ie, you could run it under 2.7 or 3.3+) but this was the worst of all worlds. Does it mean that you are against the whole spec? I don't know any Python project in the wild which was really ported to Python 3: drop Python 2 support at the same time. Supporting only Python 3 only slowly becomes a good choice for *new* projects. All projects are ported by adding Python 3 support in addition to Python 2 support. The six module is a great module to limit changes on the source code. See my early draft patch to port nova to Python 3: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d Joe Goron wrote I like how the sha1 starts with 'bad'. Overall that is a pretty small patch. ;-) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, How invasive would the port to python3 be? I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d As announced, changes are boring, just obvious Python2/Python3 issues: - strip L from long integer literals: 123L = 123 - replace dict.iteritems() with six.iteritems(dict) - replace list.sort(cmp_func) with list.sort(key=key_func) - replace raise exc_info[0], exc_info[1], exc_info[2] with six.reraise(*exc_info) - moved functions / modules * get http.client, urllib.request and other stdlib modules from six.moves since they moved between Python 2 and Python 3 * get itertools.izip/zip_longest from six.moves * replace unichr() with six.unichr() - replace filter(func, data) with [item for item in data if func(item)] - replace unicode() with six.text_type - replace (int, long) with six.integer_types - replace file() with open() - replace StringIO() with six.StringIO() These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs. Reminder: Python 2 compatibility is kept! There is not reason right now to drop Python 2 support, it would be a pain to upgrade! Would it be easier to have a python3 migration sprint and rip the band aid off instead of dragging it out and having partial python3 support for a whole cycle? Do you mean a physical meeting? Or focus on Python 3 during a short period (review/merge all Python 3 patches)? A single week may not be enough to fix all Python 3 issues. I prefer to submit changes smoothly during the Liberty cycle. In general what is the least painful way to add python3 support for a very large python project? Send patches and make incremental changes, as any other change made in OpenStack. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
I recommend to use mysqlclient instead of MySQL-python even on Python 2. https://pypi.python.org/pypi/mysqlclient https://github.com/PyMySQL/mysqlclient-python Is it packaged in popular distributions? RHEL? Fedora? SuSe? Ubuntu? Debian? Gentoo? If this library solves real bugs and provides Python 3 compatibility, I think it's worth to replace MySQL-python with this new library. Linux distro can package it, they can probably copy the packaging of MySQL-Python since it's a fork. Note: mysqlclient-python conflicts with MySQL-Python, both libraries use the same MySQLdb Python module. The good news is that mysqlclient-python is a drop-in library for MySQL-Python: no need to change any line of code, only modify requirements. @Naoki: Did you try to contact MySQL-Python authors to take over their project instead of forking? Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs. Is there already a static analysis tool that helps find these things? (Would a pylint check for the above be useful? Some of them would be hard to find reliably, but a bunch of the above would be trivial) I read that hacking has some checks. It's quite easy to run 2to3 to see modified lines. Personally, I prefer to just run tests and fix issues one by one. It's faster to fix failing tests without having the modify the whole project at one. Sometimes, I'm also using regular expressions (in vim or grep). For example, I used [0-9]+L to find 123L. You can also use regex to modify code inplace. 2to3 tool drops Python 2 compatibility, so it's not great. But it helps to find code that needs to be modified. I now prefer 2to6 which uses six and so keeps Python 2 compatibility. I modified it a little bit to reduce changes: https://github.com/haypo/2to6/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, It's moving fast. I'm currently working on porting remaining libraries to prepare my spec for nova. oslo.db -- looks like it is almost there I don't know the status of oslo.db support of Python 3. I guess that it already works on Python 3, it's just a matter of running tests with MySQL (MySQL-Python blocks again here). oslo.messaging -- same Two changes of my Python 3 changes were already merged last week. Three remaining Python 3 changes are almost there, they are mostly blocked by the freeze on requirements, but changes are already approved. The blocker is the bump of eventlet to 0.17.3: https://review.openstack.org/#/c/172132/ paste -- almost there I fixed that last night (!) with the release of Paste 2.0. It's the first Paste release since 2010. Paste 2.0 has an experimental support of Python 3. It should be enough for nova, according to my quick tests in my local nova repository where I fixed most obvious Python 3 issues. Ian Bicking allowed me to publish new versions of Paste. sqlalchemy-migrate -- almost there It already supports Python 3, so it doesn't block. suds -- with the suds fork shouldn't be too hard You should just switch to the fork. I contacted the author of suds-jurko: he plans to rename its component from suds-jurko to suds. So his fork would be simple drop-in which would not require any change in OpenStack except of requirements. websockify -- unknown It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I didn't check yet if it supports Python 3, but since the project is active (last commit 12 hours ago), it's shouldn't be hard to fix them quickly. libvirt-python -- unknown Oh, I missed this dependency. It's my next target :-) mysql-python -- alternitive looks viable. Yes, it's possible to replace it with PyMySQL. Does anyone know the status of this switch? Based on there being two unknowns, and a lot of dependencies that are just almost there, and a few we may want to migrate off of, I was assuming addressing those issues would make it hard for us to make nova python3 compatible for Liberty. My plan for Liberty is not to support fully Python 3 in nova, but only start the port (well, continue to be exact). The idea is to fix syntax errors and obvious issues like dict.iteritems() = six.iteritems(dict), then more complex changes. Maybe it's possible to finish the port in a cycle, but it really depends on the time taken to merge patches. I started to port nova in my local repository. Some unit tests already pass. nova already uses six in many places, so it's not like we really start from scratch, the port is already an ingoing process. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] release of Paste 2.0 with an experimental Python 3 support
Hi, Ian Bicking allowed me to publish new versions of the Paste project: https://bitbucket.org/ianb/paste This project blocked many OpenStack applications to be ported to Python 3: https://wiki.openstack.org/wiki/Python3#OpenStack_applications Good news: I just released Paste 2.0 which has an experimental Python 3 support. According to my quick tests, it should be enough for nova for example. Paste version is not capped in global-requirements.txt: Paste, so you should get Paste 2.0 in gates in next hours if it's not already the case. If you see any regression, please bug me so I can fix them quickly! All Paste unit tests now pass on Python 2.6 and 2.7 (I had to fix some simple issues before the release), and only a few remaining unit tests are still failing on Python 3. The previous Paste release (1.7.5.1) was published 5 years ago! ;-) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, sqlachemy-migrate already works on Python3. I sent a patch to add missing Python 3 classifiers, so the caniusepython3 automated tool will stop reporting false alarm on this library. https://review.openstack.org/174738 Or is someone aware of issues with Python 3? Again, check the wiki page for up to date information: https://wiki.openstack.org/wiki/Python3#Dependencies There seems to have been some work on that already: https://github.com/stackforge/sqlalchemy-migrate/commit/a03b141a95 Yeah, Cyril made great work! I ported a lot of OpenStack libraries. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, The part of keystoneclient that uses the memcached client was deprecated in Juno (as it was moved to the keystonemiddleware repo), Oh, I was not aware of the keystonemiddleware project. I see that Nova uses it for example. so I think we can remove it now. Do someone know if the middleware of keystoneclient is still used? You might want to patch it out of your keystoneclient package if you know everything's using the auth_token middleware from keystonemiddleware. What should we do with the following bug? memcache tests are skipped on python 3 https://bugs.launchpad.net/python-keystoneclient/+bug/1447731 I also wrote a patch for keystoneclient: Enable test_auth_token_middleware() on Python 2 https://review.openstack.org/#/c/176778/ If the keystoneclient middleware is removed, we can simply close the bug and abandon the change. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, I wrote my spec to Port Nova to Python 3: https://review.openstack.org/#/c/176868/ I squashed all my commits into a single commit of my draft port and I pushed it at: https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d I like how the sha1 starts with 'bad' Ah ah, I didn't notice that. I would prefer python prefix, but it's not possible. Overall that is a pretty small patch. Cool. My general concern, is having to manually review new code for python3 compliance. Do you prefer a single very large patch (as the one I posted above) or multiple small patches fixing similar issues? If this will take more then a week or two to make a big project python3 compat (during a virtual sprint), it would be good to have some tooling in place to make sure we don't add a lot more burden on reviewers to make sure new patches are python3 compatible by hand. I tried to write a detailed plan in my spec. Until tox -e py34 pass, I would prefer to not touch nova gates nor any add Python 3 checks. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Replace mysql-python with mysqlclient
Hi, I propose to replace mysql-python with mysqlclient in OpenStack applications to get Python 3 support, bug fixes and some new features (support MariaDB's libmysqlclient.so, support microsecond in TIME column). The MySQL database is popular, but the Python driver mysql-python doesn't look to be maintained anymore. The latest commit was done in january 2014, before the release of MySQL-python 1.2.5: https://github.com/farcepest/MySQLdb1/commits/master One major issue is that mysql-python doesn't support Python 3. It blocks porting most OpenStack applications to Python 3. There are now 32 open issues and 25 pending pull requests. I also sent an email to Andy Dustman (aka farcepest) last week, but I didn't get any reply yet. There is an open discussion to replace mysql-python with PyMySQL, but PyMySQL has worse performance: https://wiki.openstack.org/wiki/PyMySQL_evaluation Naoki INADA, the PyMySQL maintainer, forked mysql-python as the new project mysqlclient. Quote of his email (part of long thread [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible): I'm maintainer of PyMySQL and mysqlclient. mysqlclient is fork of MySQL-python. It uses libmysqlclient.so. It fixes some bugs, build issues and it support Python 3. For example: * Support MariaDB's libmysqlclient.so * Support microsecond in TIME column I recommend to use mysqlclient instead of MySQL-python even on Python 2. https://pypi.python.org/pypi/mysqlclient https://github.com/PyMySQL/mysqlclient-python Since mysqlclient is fork, it should have no impact on performances. On PyPI, mysql-python and mysqlclient have a different name, but the Python module has the same name (MySQLdb). OpenStack code doesn't need to be modified, only dependencies. mysqlclient is also tested in the PyMySQL evaluation. mysqlclient shares mysql-python drawbacks. It is implemented in C and it is not eventlet friendly (cannot be monkey-patched). A workaround is to run it in a thread or a thread pool. I want to replace mysql-python with mysqlclient to get Python 3 compatibility in short term. We can reconsider PyMySQL later for other advantages like the ability to monkey-patch it. At the same time, there are also a deeper discussion to change how OpenStack handles concurrency (replace eventlet with threads, asyncio or something else): https://review.openstack.org/#/c/164035/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Port Nova to Python 3
Hi, I wrote a spec to port Nova to Python 3: https://review.openstack.org/#/c/176868/ I updated my spec to take in account all comments. Example of changes: - Explicitly say that Python 2 support is kept (ex: change the title to Adding Python 3.4 support to Nova) - Clarify what are the Nova tests - Shorter spec I prefer to exclude Tempest tests, and restrict the scope of the spec to Nova unit and functional tests. Most Python 3 issues should be catched by Nova unit and functional tests anyway. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
Hi, Mike Bayer wrote: It is not feasible to use MySQLclient in Python 2 because it uses the same module name as Python-MySQL, and would wreak havoc with distro packaging and many other things. IMO mysqlclient is just the new upstream for MySQL-Python, since MySQL-Python is no more maintained. Why Linux distributions would not package mysqlclient if it provides Python 3 support, contains bugfixes and more features? It's quite common to have two packages in conflicts beceause they provide the same function, same library, same program, etc. I would even suggest packagers to use mysqlclient as the new source without modifying their package. It is also imprudent to switch production openstack applications to a driver that is new and untested (even though it is a port), nor is it necessary. Why do you consider that mysqlclient is not tested or less tested than mysql-python? Which kind of regression do you expect in mysqlclient? As mysql-python, mysqlclient Github project is connected to Travis: https://travis-ci.org/PyMySQL/mysqlclient-python (tests pass) I trust more a project which is actively developed. There should be no reason Openstack applications are hardcoded to one database driver. The approach should be simply that in Python 3, the mysqlclient library is installed instead of mysql-python. Technically, it's now possible to have different dependencies on Python 2 and Python 3. But in practice, there are some annoying corner cases. It's more convinient to have same dependencies on Python 2 and Python 3. Using mysqlclient on Python 2 and Python 3 would avoid to have bugs specific to Python 2 (bugs already fixed in mysqlclient) and new features only available on Python 3. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
I propose to replace mysql-python with mysqlclient in OpenStack applications to get Python 3 support, bug fixes and some new features (support MariaDB's libmysqlclient.so, support microsecond in TIME column). I just proposed a change to add mysqlclient dependency to global requirements: https://review.openstack.org/#/c/179745/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
If we're going to move, shouldn't we be looking at something that supports our threading model? I would prefer to make baby steps, and first fix the Python 3 compatibility. Enhance concurrency/parallelism is a much more complex project than just replacing a single line in dependencies ;-) See my email, I mentioned a workaround for mysqlclient and a spec discussing a more general solution for concurrency. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Adding Mehdi Abaakouk (sileht) to oslo-core
Hi, I didn't know that Mehdi was only core reviewer on Oslo Messaging. +1 for Mehdi as Oslo core reviewer. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Port Nova to Python 3
Hi, Oh, this makes me think: how would one fix something like this? The wiki page already contains some answer: https://wiki.openstack.org/wiki/Python3#Common_patterns Don't hesitate to complete the page if needed. See also my personal list of documents: https://haypo-notes.readthedocs.org/python.html#port-python-2-code-to-python-3 def unicode_convert(self, item): try: return unicode(item, utf-8) E NameError: name 'unicode' is not defined Use six.text_type. def make(self, idp, sp, args): md5 = hashlib.md5() for arg in args: md5.update(arg.encode(utf-8)) md5.update(sp) E TypeError: Unicode-objects must be encoded before hashing It depends on the code. Sometimes, you should encode sp in the caller, sometimes you should encode just before calling make(). If you don't know, use: if isinstance(sp, six.text_type): sp = sp.encode(utf-8) md5.update(sp) and one last one: def harvest_element_tree(self, tree): # Fill in the instance members from the contents of the # XML tree. for child in tree: self._convert_element_tree_to_member(child) for attribute, value in tree.attrib.iteritems(): self._convert_element_attribute_to_member(attribute, value) E AttributeError: 'dict' object has no attribute 'iteritems' Use six.iteritems(). BTW, I did this: -from Cookie import SimpleCookie +try: +from Cookie import SimpleCookie +except: +from http.cookies import SimpleCookie Is there anything smarter to do with six? What's the rule with six.moves? Should I always just use the new location? I did the same. If it's a very common pattern in your code, you can register you own move in six.moves: https://pythonhosted.org/six/#advanced-customizing-renames We used that fox mox/mox3 in tests of Oslo projects. Also, is this a correct fix for the basestring issue in Py3? +try: +basestring +except NameError: +basestring = (str,bytes) I prefer six.string_types. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Adding Joshua Harlow to oslo-core
Hi, Great idea! Joshua did a lot for Oslo, I want him in the oslo-core team! Victor Hi fellows, I'd like to propose that we add Joshua Harlow to oslo-core. He is already maintaining some of the Oslo libraries (taskflow, tooz…) and he's helping on a lot of other ones for a while now. Let's bring him in for real! __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] Python 3.4 unit tests enabled on the gate for openstack/ironic
With this commit: https://review.openstack.org/181034 Python 3.4 unit tests are now being run for openstack/ironic. The unit tests are a voting job. Thanks to Victor Sergeyev for all of his work to update the Ironic code to make it pass the unit tests using Python 3.4: https://review.openstack.org/#/c/156192/ Great job! It's cool that the job is already voting to avoid Python 3 regressions. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Replace mysql-python with mysqclient to get Python 3 support
Hi, Since the discussion on mysqlclient looks to be stuck, I chose the approach adopted in Ironic: use PyMySQL on Python 3. I proposed a patch to run Nova tests using PyMySQL on Python 3, but keep MySQL-python on Python 2. https://review.openstack.org/#/c/182709/ It only impacts how Nova run tests. Deployers are still free to use mysql-python, mysqlclient, PyMySQL or whatever. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Replace mysql-python with mysqclient to get Python 3 support
Hi, TL;DR we must replace mysql-python with mysqclient to get Python 3 support, it's safe and *very* easy I restart the discussion because the previous one gone far from my point which is getting Python 3 support. IMO the fastest way to get Python 3 support is to replace mysql-python with mysqclient. Update since the previous discussion began: * Debian and Red Hat told me that they will probably base their package on mysqlclient * I ran OpenStack tests on mysqlclient: all tests pass * Nova, Neutron and Heat specs for Python 3 are almost accepted (or already accepted for Heat) -- Good news, the portage of OpenStack to Python 3 is going well: Nova and Neutron specs to add Python 3 are almost accepted, the Heat spec was already merged. https://review.openstack.org/176868 https://review.openstack.org/172962 https://review.openstack.org/175340 The two hot dependencies are suds and mysql-python: none support Python 3. My patch to replace suds with suds-jurko has been merged into oslo.vmware. I'm not aware of any blocking point for cinder and nova patches (to replace suds with suds-jurko or drop suds dependency): https://review.openstack.org/#/c/180130/ https://review.openstack.org/#/c/181554/ suds project is no more maintained since 2011. suds-jurko is a fork which adds Python 3 compatibility (and problably other cool things like bugfixes). Now, MySQL: OpenStack currently uses MySQL-python as the MySQL driver. MySQL-python is no more maintained since january 2014 (last commit, last release), there are 32 open issues and 25 pending pull requests. I heard that the maintainer basically gave up. Different people (including me) reported that they sent emails to the maintainer last 6 months, but he didn't reply. Check yourself the status of the project: https://github.com/farcepest/MySQLdb1/ Good news (again, I only have good news today!), as suds-jurko, mysqlclient is a fork of mysql-python, it is actively maintainted and supports Python 3: https://github.com/PyMySQL/mysqlclient-python Debian and Red Hat maintainers told me that they will probably base their package on mysqlclient without changing the name of the package (python-mysqldb and MySQL-python). I propose to do the same in OpenStack to get Python 3 support. Patches for requirements and nova: https://review.openstack.org/#/c/179745/ https://review.openstack.org/#/c/180128/ Note: I sent an email to the mysql-python maintainer to ask him to allow mysqlclient maintainer to takeover mysql-python, it would be the most natural and simplest solution, but he doesn't reply :-( -- Mike Bayer wants better performances, he proposes to run DB queries in thread pools or use PyMySQL to be able to monkey-patch the driver. IMO it's a different topic, it may impact the architecture of applications. Such change is more complex than simply replacing *one line* in requirements.txt or test-requirements.txt, and we need Python 3 support right now ;-) Mike Bayer also fears bugs or compatibility issues. I ran OpenStack tests with mysqlclient: well, all tests pass: https://review.openstack.org/#/c/180128/ But I agree, as any other project, mysqlclient can have bugs. IMO it's safer to rely on a library actively developed to quickly get fixes and new releases. Think about security vulnerabilities: what happens if a vulnerability is found in MySQL-python? Remind that the project didn't get an commit since january 2014... Sorry, I don't understand why it's much more difficult to replace MySQL-Python with mysqlclient, than replacing suds with suds-jurko, the case is exactly the same. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
For the full list, see the wiki page: https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects Thanks for updating the wiki page that is a very useful list. From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen. Why? I plan to work on porting nova to Python 3. I proposed a nova session on Python 3 at the next OpenStack Summit at Vancouver. I plan to write a spec too. I'm not aware of any real blocker for nova. What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now. I just checked sqlachemy-migrate. The README and the documentation are completly outdated, but the project is very active: latest commit one month ago and latest release (0.9.6) one month ago. There are py33 and py34 environments and tests pass on Python 3.3 and Python 3.4! I didn't check yet, but I guess that sqlachemy-migrate 0.9.6 already works on Python 3. Python 3 classifiers are just missing in setup.cfg. I sent patches to update the doc, to add Python 3 classifiers and to upgrade requirements. The project moved to stackforge, reviews are at review.openstack.org: https://review.openstack.org/#/q/status:open+project:stackforge/sqlalchemy-migrate,n,z The wiki page said that scripttest and ibm-db-sa were not Python 3 compatible. It's no more true: scripttest is compatible Python 3, and there is ibm-db-sa-py3 which is Python 3 compatible. I updated the wiki page for sqlachemy-migrate. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Worth noting we've already switched to using PyMySQL in nodepool, storyboard and some of the subunit2sql tooling. It's been working out great so far. Great. Did you notice a performance regression? Mike wrote that PyMySQL is much slower than MySQL-Python. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Add request_utils module in oslo-incubator
I'd support picking another existing oslo library instead of oslo-incubator itself. Can you please review to see where this would fit? In the review, I suggested oslo.log, but I'm not sure because of the notifier object. I don't know the type of the notifier object, is it something related to logging? Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Issues with pip 6.1
Hi,
I tried to install a fresh DevStack but it doesn't work because of the new
release of pip: pip 6.1 (probably released today,
https://pip.pypa.io/en/stable/news.html still says 6.1.0 (unreleased)...).
(1) update.py of openstack/requirements uses req.InstallRequirement.url
attribute, but this attribute is gone (replaced with
req.InstallRequirement.link.url)
= see https://bugs.launchpad.net/tempest/+bug/1440984
(2) it's not more possible to install argparse on Python 2.7, pip install
argparse simply write Skipping requirement: argparse because argparse is a
stdlib package. It makes sense, but different OpenStack components *require*
argparse. I mean, pkg_resources raises an ImportError if argparse is not
installed with pip. Example:
2015-04-07 10:08:34.084 | + /usr/bin/keystone-manage db_sync
2015-04-07 10:08:34.239 | Traceback (most recent call last):
2015-04-07 10:08:34.239 | File /usr/bin/keystone-manage, line 4, in module
2015-04-07 10:08:34.239 |
__import__('pkg_resources').require('keystone==2015.1.dev143')
2015-04-07 10:08:34.239 | File
/usr/lib/python2.7/site-packages/pkg_resources/__init__.py, line 3057, in
module
2015-04-07 10:08:34.239 | working_set = WorkingSet._build_master()
2015-04-07 10:08:34.240 | File
/usr/lib/python2.7/site-packages/pkg_resources/__init__.py, line 639, in
_build_master
2015-04-07 10:08:34.240 | ws.require(__requires__)
2015-04-07 10:08:34.240 | File
/usr/lib/python2.7/site-packages/pkg_resources/__init__.py, line 940, in
require
2015-04-07 10:08:34.240 | needed =
self.resolve(parse_requirements(requirements))
2015-04-07 10:08:34.240 | File
/usr/lib/python2.7/site-packages/pkg_resources/__init__.py, line 827, in
resolve
2015-04-07 10:08:34.240 | raise DistributionNotFound(req, requirers)
2015-04-07 10:08:34.241 | pkg_resources.DistributionNotFound: The 'argparse'
distribution was not found and is required by oslo.config,
python-keystoneclient, pysaml2
2015-04-07 10:08:34.246 | + exit_trap
Workaround: install pip 6.0.8, run pip install argparse
= I just opened https://bugs.launchpad.net/keystone/+bug/1441083
Victor
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Issues with pip 6.1
(2) it's not more possible to install argparse on Python 2.7 (...) Donald Stufft released pip 6.1.1, he reverted the change which prevented to install argparse. Thanks for the quick fix! (1) update.py of openstack/requirements uses req.InstallRequirement.url attribute, but this attribute is gone (replaced with req.InstallRequirement.link.url) = see https://bugs.launchpad.net/tempest/+bug/1440984 pip 6.1.1 reintroduced the url attribute but there is a bug in the property :-/ According to pip developers, the problem is that OpenStack should not use Python pip API because the Python API is private and not stable. Only the CLI has a stable API. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible
Hi, During the last OpenStack Summit at Paris, we discussed how we can port OpenStack to Python 3, because eventlet was not compatible with Python 3. There are multiple approaches: port eventlet to Python 3, replace eventlet with asyncio, replace eventlet with threads, etc. We decided to not take a decision and instead investigate all options. I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped). I just sent a patch for requirements and Oslo Messaging to bump to eventlet 0.17.3, but it will have to wait until everyone has master as Liberty. https://review.openstack.org/#/c/172132/ https://review.openstack.org/#/c/172135/ It becomes possible to port more projects depending on eventlet to Python 3! Liberty cycle will be a good opportunity to port more OpenStack components to Python 3. Most OpenStack clients and Common Libraries are *already* Python 3 compatible, see the wiki page: https://wiki.openstack.org/wiki/Python3 -- To replace eventlet, I wrote a spec to replace it with asyncio: https://review.openstack.org/#/c/153298/ Joshua Harlow wrote a spec to replace eventlet with threads: https://review.openstack.org/#/c/156711/ But then he wrote a single spec Replace eventlet + monkey-patching with ?? which covers threads and asyncio: https://review.openstack.org/#/c/164035/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] gate pep8 jobs broken today
Hi, I found a related issue in glance. issue with pbr 1.0 release https://bugs.launchpad.net/glance/+bug/1456800 Modify entry point tests to not require deps https://review.openstack.org/#/c/184326/ Victor - Original Message - From: Robert Collins robe...@robertcollins.net To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Sent: Monday, May 18, 2015 5:54:43 PM Subject: [openstack-dev] [all] gate pep8 jobs broken today Hi, we had a gate outage today for a few hours. http://pad.lv/1456376 The issue was an interaction between the existence of pbr 1.0, project local requirements, hacking 0.10.1 and flake8 2.4.1. When flake8 2.4.1 loads plugins (which hacking is) it uses pkg_resources and calls load(), which checks requirements. pbr in the pep8 jobs is installed by the project requirements.txt files, which per global-requirements mostly now say =0.11, 2.0, so pbr 1.0.0 was immediately installed once it was released. hacking is installed from release, so hacking 0.10.1 was installed, which has the constraint for pbr of 1.0 that we had prior to bumping the releases in global-requirements. And so boom. We've now released hacking 0.10.2, which contains only the updated pbr constraint, and we don't expect any additional fallout from it. Thanks Clark, Doug, Ian, Sean, and Joe for helping unwind, analyze and fix this. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Port Cinder to Python 3
Hi, Le 26/05/2015 11:51, Duncan Thomas a écrit : Thanks Victor, that is pretty much exactly what I wanted to hear - there's a known, finite plan to get us to fully working with python3. I'll go change my reviews now. As expected, two weeks later, all my patches are in conflict :-) I rebased my Python 3 patches for Cinder: https://review.openstack.org/#/q/status:open+project:openstack/cinder+branch:master+topic:py3,n,z In the meantime, I enhanced my sixer tool, so the generated patches are larger but fixes more code. I splitted the six.moves patch into 3 patches: six.moves, StringIO and urllib. Can someone please review them to not have to rebase them every 2 weeks? ;-) Thanks, Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] oslo releasing is noisy...
Le 09/06/2015 18:31, Thierry Carrez a écrit : We are currently exploring the option to repurpose the openstack-announce ML to be the extensive record of release announcements. It's part of a larger plan to streamline library releases, about which Doug should start a discussion pretty soon now. In the Python community, there are Special Interest Groups (SIG) which have their mailing lists: distutils-sig, import-sig, mobile-sig, etc. https://www.python.org/community/sigs/ Maybe we can move some traffic to new mailing lists, like a new openstack-oslo@ list, instead of using tags in the subject? Anyone would be able to choose to subscribe or not to openstack-oslo. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [release][oslo] oslotest release 1.7.0 (liberty)
Good news: your fix is already merged into oslo.messaging ;-) oslo.messaging now uses directly mox3, on Python 2 and Python 3. Victor Le 10/06/2015 14:04, Victor Sergeyev a écrit : Hi All, this oslotest release break oslo.messaging gate - unittest fails with ImportError, on import mox from six.moves - [0]. It caused by commit [1], which removed adding mox to six moves. There is a fix for oslo.messaging - [2] - please, help to merge it. [0] - http://paste.openstack.org/show/281091/ [1] - https://github.com/openstack/oslotest/commit/9e0c8ad2c251274128499a7fcfb591c488d27d2b [2] - https://review.openstack.org/190136 Thanks, Victor On Tue, Jun 9, 2015 at 6:14 PM, d...@doughellmann.com mailto:d...@doughellmann.com wrote: We are content to announce the release of: oslotest 1.7.0: Oslo test framework This release is part of the liberty release series. With source available at: http://git.openstack.org/cgit/openstack/oslotest For more details, please see the git log history below and: http://launchpad.net/oslotest/+milestone/1.7.0 Please report issues through launchpad: http://bugs.launchpad.net/oslotest Changes in oslotest 1.6.0..1.7.0 5bd9b31 Updated from global requirements ff9b38e Fix argument handling in oslo_run_cross_tests 960e444 Add class to deal with clouds.yaml support 1c0863f Remove unneeded runtime pbr dep 5f2e635 Updated from global requirements f8a5a3c Advertise support for Python3.4 / Remove support for Python 3.3 a063f25 Do not sync run_cross_tests.sh 0782ab7 Remove unused discover dependency 9e0c8ad Remove six.moves call Diffstat (except docs and test files) - openstack-common.conf | 7 -- oslotest/__init__.py | 1 - oslotest/functional.py | 59 ++ oslotest/moxstubout.py | 2 +- requirements.txt | 3 +-- setup.cfg | 6 + tox.ini| 3 +-- 8 files changed, 83 insertions(+), 30 deletions(-) Requirements updates diff --git a/requirements.txt b/requirements.txt index 674130c..1486ede 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,2 +4,0 @@ -pbr=0.6,!=0.7,1.0 -discover @@ -14,0 +13 @@ mox3=0.7.0 +os-client-config=1.2.0 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [glance] Progress of the Python 3 port
Hi, We made great progress on porting Glance to Python 3. tox -e py34 now pass with a subset of tests and there is a non-voting py34 check job. I propose to make the py34 gate voting to avoid Python 3 regressions: https://review.openstack.org/194048 Make gate-glance-python34 voting When the py34 will become voting, it will no more possible to introduce code incompatible with Python 3, at least in the code tested by tox -e py34. Currently, py34 uses the development branch of glance_store because there is no release including my Python 3 fixes. For glance_store, it would be nice if someone can review my last patches: https://review.openstack.org/#/q/status:open+project:openstack/glance_store+branch:master+topic:py3,n,z With these 3 patches to port drivers, it becomes to possible to run all tests in tox -e py34, not only a subset of tests. So glance_store will be fully compatible with Python 3. My current patch tox -e py34: use a subset of working tests may be updated to run all tests, not only a subset, when other py3 patches will be merged. Glance currently only uses PyMySQL on Python 3. For Python 2, there is a pending patch. It was blocked by oslo.db which didn't use PyMySQL. It changed: oslo.db now also uses PyMySQL driver by default for MySQL (but there is no release yet including this change). Glance patch: https://review.openstack.org/#/c/184373/ Switch from MySQL-python to PyMySQL Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Debian already using Python 3.5: please gate on that
Hi, Le 18/06/2015 15:44, Thomas Goirand a écrit : As per the subject line, we already have Python 3.5 in Debian (AFAICT, from Debian Experimental, in version beta 2). As a consequence, we're already running (unit) tests using Python 3.5. Some have failures: I could see issues in ceilometerclient, keystoneclient, glanceclient and more (yes, I am planning to report these issues, and we already started doing so). Can you please share the list of issues? I may take a look. Python 3.5 is not supposed to break backward compatibility. Sorry, I don't remember which one, but I remember that a project is running tests using the default (development) branch of CPython, and it already helped us (CPython) to fix issues before the final release. I don't think that OpenStack has resources (CPU and manpower) to run OpenStack tests on top of CPython default, nor on a beta version. But as I wrote, I'm interested to take a look at issues with Python 3.5 ;-) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder][glance][nova] Python 3 coming!
Hi, The Python 3.4 gate (py34) of cinder, glance and nova just became voting. You now have to write Python code compatible with Python 2.7 and 3.4. If the py34 check job fails on your patch, you may try to rebase it to get the new tox.ini and updated requirements. You can find information on Python 3 in OpenStack in the following wiki page: https://wiki.openstack.org/wiki/Python3 If you have issues with Python 3 or the py34 gates, reply to this email or contact me privately. The Cinder blueprint and Nova spec for Python 3 explain the overall plan to add Python 3 support to these projects. I have the same plan for Glance. https://blueprints.launchpad.net/cinder/+spec/cinder-python3 http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/adding-python34-support-to-nova.html Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Jenkins check failed on gate-glance_store-python34
Hi, Le 26/06/2015 09:17, 孙明达 a écrit : I am a new contributor for openstack. the following url is my code review for glance_store https://review.openstack.org/#/c/193403/ Jenkins check failed on gate-glance_store-python34 Referring to the console info, I have never changed the code mentioned. Please rebase your patch to get the new tox.ini. py34 should pass with the new tox.ini. The py34 gate of glance just became voting! Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][python3] use of six.iteritems()
Hi,
Le 10/06/2015 02:15, Robert Collins a écrit :
python2.7 -m timeit -s 'd=dict(enumerate(range(100)))' 'for i in
d.items(): pass'
10 loops, best of 3: 76.6 msec per loop
python2.7 -m timeit -s 'd=dict(enumerate(range(100)))' 'for i in
d.iteritems(): pass'
100 loops, best of 3: 22.6 msec per loop
.items() is 3x as slow as .iteritems(). Hum, I don't have the same
results. Try attached benchmark. I'm using my own wrapper on top of
timeit, because timeit is bad at calibrating the benchmark :-/ timeit
gives unreliable results.
Results on with CPU model: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz:
[ 10 keys ]
713 ns: iteritems
922 ns (+29%): items
[ 10^3 keys ]
42.1 us: iteritems
59.4 us (+41%): items
[ 10^6 keys (1 million) ]
89.3 ms: iteritems
442 ms (+395%): items
In my benchmark, .items() is 5x as slow as .iteritems(). The code to
iterate on 1 million items takes almost an half second. IMO adding 300
ms to each request is not negligible on an application. If this delay is
added multiple times (multiple loops iterating on 1 million items), we
may reach up to 1 second on an user request :-/
Anyway, when I write patches to port a project to Python 3, I don't want
to touch *anything* to Python 2. The API, the performances, the
behaviour, etc. must not change.
I don't want to be responsible of a slow down, and I don't feel able to
estimate if replacing dict.iteritems() with dict.items() has a cost on a
real application.
As Ihar wrote: it must be done in a separated patch, by developers
knowning well the project.
Currently, most developers writing Python 3 patches are not heavily
involved in each ported project.
There is also dict.itervalues(), not only dict.iteritems().
for key in dict.iterkeys() can simply be written for key in dict:.
There is also xrange() vs range(), the debate is similar:
https://review.openstack.org/#/c/185418/
For Python 3, I suggest to use from six.moves import range to get the
Python 3 behaviour on Python 2: range() always create an iterator, it
doesn't create a temporary list. IMO it makes the code more readable
because for i in xrange(n): becomes for i in range(n):. six is not
written outside imports and range() is better than xrange() for
developers starting to learn Python.
Victor
Micro-benchmark for the Python operation key in dict. Run it with:
./python.orig benchmark.py script bench_str.py --file=orig
./python.patched benchmark.py script bench_str.py --file=patched
./python.patched benchmark.py compare_to orig patched
Download benchmark.py from:
https://bitbucket.org/haypo/misc/raw/tip/python/benchmark.py
import gc
def consume_items(dico):
for key, value in dico.items():
pass
def consume_iteritems(dico):
for key, value in dico.iteritems():
pass
def run_benchmark(bench):
for nkeys in (10, 10**3, 10**6):
bench.start_group('%s keys' % nkeys)
dico = {str(index): index for index in range(nkeys)}
bench.compare_functions(
('iteritems', consume_iteritems, dico),
('items', consume_items, dico),
)
dico = None
gc.collect()
gc.collect()
if __name__ == __main__:
import benchmark
benchmark.main()
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison
Hi, By the way, who is the oslo liaison for nova? If there is nobody, I would like to take this position. Victor Le 25/05/2015 18:45, Ghe Rivero a écrit : My focus on the Ironic project has been decreasing in the last cycles, so it's about time to relinquish my position as a oslo-ironic liaison so new contributors can take over it and help ironic to be the vibrant project it is. So long, and thanks for all the fish, Ghe Rivero -- Pinky: Gee, Brain, what do you want to do tonight? The Brain: The same thing we do every night, Pinky—try to take over the world! .''`. Pienso, Luego Incordio : :' : `. `' `- www.debian.org http://www.debian.org www.openstack.com http://www.openstack.com GPG Key: 26F020F7 GPG fingerprint: 4986 39DA D152 050B 4699 9A71 66DB 5A36 26F0 20F7 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic][oslo] Stepping down from oslo-ironic liaison
Oh, on the wiki page I read that The liaison should be a core reviewer for the project, but does not need to be the PTL.. I'm not a core reviewer for nova. Is it an issue? On the wiki page, I see that John Villalovos (happycamp) is the Nova liaison for Oslo, not Joe Goron. I don't understand. Victor Le 27/05/2015 20:44, Joe Gordon a écrit : On Wed, May 27, 2015 at 3:20 AM, Davanum Srinivas dava...@gmail.com mailto:dava...@gmail.com wrote: Victor, Nice, yes, Joe was the liaison with Nova so far. Yes, please go ahead and add your name in the wiki for Nova as i believe Joe is winding down the oslo liaison as well. https://wiki.openstack.org/wiki/CrossProjectLiaisons#Oslo Yup, thank you Victor! thanks, dims On Wed, May 27, 2015 at 5:12 AM, Victor Stinner vstin...@redhat.com mailto:vstin...@redhat.com wrote: Hi, By the way, who is the oslo liaison for nova? If there is nobody, I would like to take this position. Victor Le 25/05/2015 18:45, Ghe Rivero a écrit : My focus on the Ironic project has been decreasing in the last cycles, so it's about time to relinquish my position as a oslo-ironic liaison so new contributors can take over it and help ironic to be the vibrant project it is. So long, and thanks for all the fish, Ghe Rivero -- Pinky: Gee, Brain, what do you want to do tonight? The Brain: The same thing we do every night, Pinky—try to take over the world! .''`. Pienso, Luego Incordio : :' : `. `' `- www.debian.org http://www.debian.org http://www.debian.org www.openstack.com http://www.openstack.com http://www.openstack.com GPG Key: 26F020F7 GPG fingerprint: 4986 39DA D152 050B 4699 9A71 66DB 5A36 26F0 20F7 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] gate is wedged
Hi, Le 30/06/2015 05:49, Matt Riedemann a écrit : Alternatively, oslo.versionedobjects 0.5.1 is cut after https://review.openstack.org/#/c/196926/ is merged and then we just need haypo's test_db_api fix for the oslo.db 2.0.0 change: https://review.openstack.org/#/c/196719/ I updated my patch Update test_db_api for oslo.db 2.0 (1) to not depend on my Fix Python 3 issues in nova.db.sqlalchemy patch. I should now be easier to fix gates. (1) https://review.openstack.org/#/c/196719/ (2) https://review.openstack.org/#/c/195191/ I suggest to block my Python 3 patch (2) until gates are fixed. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] gate is wedged
Le 30/06/2015 10:32, Victor Stinner a écrit : I updated my patch Update test_db_api for oslo.db 2.0 (1) ... (1) https://review.openstack.org/#/c/196719/ I updated my patch again to also block oslo.versionedobjects 0.5.0 in requirements. So we can have two patches to fix the bug ;) The patch (1) only fixes the bug, whereas the Python 3 patch fixes many other things (Python 3 support, remove test-requirements-py3.txt, etc.). At the same time, we have issues on the OpenStack infra :-/ 10:48 -!- ChanServ changed the topic of #openstack-infra to: OpenStack CI is down due to hard drive failures Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Let's get rid of tablib and cliff-tablib
Hi, Le 29/06/2015 11:03, Thomas Goirand a écrit : cliff-tablib is used for the unit tests of things like python-neutronclient. The annoying bit is that cliff-tablib depends on tablib, which itself is a huge mess. It has loads of 3rd party embedded packages and most of them aren't Python 3.x compatible. tablib includes copies of various dependencies in its tablib/packages/ directory. Some of them are for Python 2, others are for Python 3. It would be better to use dependencies (requirements in setup.py), not copies. Do you try to contact tablib authors to ask them to remove completly tablib/packages/? setup.py uses a different list of packages on Python 2 and Python 3. I tried python3 setup.py install: the bytecode compilation of markup.py fails with an obvious SyntaxError, the code is for Python 2. But there is also markup3.py which is compiled successfully. Even if the compilation of the markup.py fails, python setup.py install succeed with the exit code 0. What is your problem? setup.py should be fixed to skip markup.py on Python 3, and skip markup3.py on Python 2. A workaround is to remove manually the file depending on the Python major version. Note: pip install tablib works on Python 3 (pip uses the binary wheel package). I've seen that for python-openstackclient, recently, cliff-tablib was added. Let's do the reverse, and remove cliff-tablib whenever possible. If we really want to keep using cliff-tablib, then someone has to do the work to port tablib to Python3 (good luck with that...). cliff-tablib is used in tests. If you remove the cliff-tablib dependency, tests will obviously fail. What do you propose? Modify tests to reimplement cliff-tablib? Remove tests? Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] Ideas to detect Oslo regressions earlier
Hi, Unfortunately, it looks like each regressions introduced by releases of Oslo libraries are still common :-( We already have tools to detect regressions, but they are run manually. It would be nice to automate these tests and run them more frequently (at least one per week, or even daily?). There are tools to run unit tests of projects like neutronclient or nova on the development version of oslo.* libraries. They take up to 12 hours to run all tests. Example of tools: - tox -e venv -- oslo_run_pre_release_tests in each Oslo project - tools/oslo_run_cross_tests in oslotest To prepare the latest bunch of releases, dims wrote two patches to run nova unit tests and tempest: * https://review.openstack.org/#/c/186413/ * https://review.openstack.org/#/c/186418/ Unfortunately, the stable version of some oslo.* projects were used instead of the development version, and 2 regressions were missed :-/ It would nice to automate a job running cinder, nova and neutron unit tests and tempest on the development versions (git master branch) of all oslo.* projects. We can start with something simple: run tools/oslo_run_cross_tests and send the result by email every day to some people interested by the result (ex: Oslo liaisons, or just me if nobody cares). It would be nice to have a dedicated resource in the OpenStack infra for such job. I proposed to run all tests using Gerrit for each patch send to an Oslo project, but it would use too much resources of the OpenStack infra. Another idea would be to write a check job dedicated to Oslo releases and use Gerrit to prepare a release (at least to tag a version in Git). Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Adopt mox3
Hi, Le 10/06/2015 22:17, Davanum Srinivas a écrit : Oslo folks, everyone, mox3 needs to be maintained since some of our projects use it and we have it in our global requirements. Here's the proposal from Doug - https://review.openstack.org/#/c/190330/ Why not only creating a project on Github? Do we need all OpenStack tools for mox3? I don't expect much enhancements in the library. For me, OpenStack looks more restrictive than a classic Github project, it's more heavy to contribute (sign a CLA, use review.openstack.org, etc.). I prefer mock over mox, the API is very different. mock is now part of Python stdlib (unittest.mock since Python 3.3). In the past, I ported some mox tests to mock, but it takes a lot of time :-( https://docs.python.org/dev/library/unittest.mock.html Did anyone try to contact original mox authors? smiddlek is the last active contributor. I see changes in 2012 in the Subversion repository of mox, whereas the latest mox release was in 2010 (mox 0.5.3). It would be nice to merge back enhancements of mox3 (Python 3 support) into mox. It's annoying to have a specific project to get Python 3 support. mox is hosted on code.google.com which is closing! Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][oslo] RPC Asynchronous Communication
Hi Sahid, There is a work-in-progress work to change how concurrency is handled in openstack processes: Replace eventlet with ??? https://review.openstack.org/164035 The latest updates can be found in etherpads from the OpenStack Vancouver Summit: - https://etherpad.openstack.org/p/liberty-cross-project-managing-concurrency - https://etherpad.openstack.org/p/YVR-oslo-asyncio No choice was made at Vancouver. In short, we should continue to investigate the different options (asyncio/trollius, threads, etc.) to get more information to be able to take a decision. Victor - Original Message - From: Sahid Orentino Ferdjaoui sahid.ferdja...@redhat.com To: openstack-dev@lists.openstack.org Sent: Thursday, May 7, 2015 11:34:51 AM Subject: [openstack-dev] [nova][oslo] RPC Asynchronous Communication Hi, The primary point of this expected discussion around asynchronous communication is to optimize performance by reducing latency. For instance the design used in Nova and probably other projects let able to operate ascynchronous operations from two way. 1. When communicate between inter-services 2. When communicate to the database 1 and 2 are close since they use the same API but I prefer to keep a difference here since the high level layer is not the same. From Oslo Messaging point of view we currently have two methods to invoke an RPC: Cast and Call: The first one is not bloking and will invoke a RPC without to wait any response while the second will block the process and wait for the response. The aim is to add new method which will return without to block the process an object let's call it Future which will provide some basic methods to wait and get a response at any time. The benefice from Nova will comes on a higher level: 1. When communicate between services it will be not necessary to block the process and use this free time to execute some other computations. future = rpcapi.invoke_long_process() ... do something else here ... result = future.get_response() 2. We can use the benefice of all of the work previously done with the Conductor and so by updating the framework Objects and Indirection Api we should take advantage of async operations to the database. MyObject = MyClassObject.get_async() ... do something else here ... MyObject.wait() MyObject.foo = bar MyObject.save_async() ... do something else here ... MyObject.wait() All of this is to illustrate and have to be discussed. I guess the first job needs to come from Oslo Messaging so the question is to know the feeling here and then from Nova since it will be the primary consumer of this feature. https://blueprints.launchpad.net/nova/+spec/asynchronous-communication Thanks, s. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder] Port Cinder to Python 3
(Oops, I sent a draft by mistake, sorry about that.) Hi, There is an ongoing effort to port OpenStack applications to Python 3 during the Liberty cycle. The following table gives the status of each application and links to patches: https://wiki.openstack.org/wiki/Python3#OpenStack_applications Specs were accepted for: * heat: https://review.openstack.org/#/c/175340/ * keystone: https://review.openstack.org/#/c/177380/ * neutron: https://review.openstack.org/#/c/172962/ * nova: https://review.openstack.org/176868 For cinder, I ported the last blocking dependency to Python 3, rtslib-fb: https://github.com/agrover/rtslib-fb/pull/62 (There is not release including the fix yet.) MySQL-python should be replaced with PyMySQL to run Python 3 tests, as done in Ironic, Nova and other applications. I started to write patches to port Cinder code to Python 3: https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:py3,n,z Duncan Thomas rejected one of my patch: As commented on https://review.openstack.org/#/c/185411/ I'd really like to be convinced that there's an end in sight for this python3 work, or I'm going to start rejecting it. This change is making the code noticeably harder to read, and has no checks to stop it recurring in future, and is lacking substantial justification as to it's benefits. In the other issue, he wrote: Is there a master list of remaining python3 issues anywhere? At the moment we are introducing more and more little changes into the code, including some really ugly six stuff, without any idea if we are even close to actually being able to work with python3. To be honest, I think that I'd like to see a working python3 tree before we merge any more - we can then pull in the fixes in a clean manner, knowing the end-goal is possible. Thoughts appreciated, otherwise I'm tempted to start hitting -2 on python3 stuff. Ok, I checked the status of Cinder port to Python 3. I merged my pending patches into a local branch and I fixed manually dependencies for tox -e py34 (replace MySQL-python with PyMySQL, install the development version of oslo.vmware). With 4 minor changes, I'm able to run one cinder unit test (cinder/tests/unit/test_quota.py). Cool! The next step will be to run a subset of tests in tox -e py34 to make it pass, and then add a checking job. When the job becomes stable, make it voting to avoid further Python 3 regressions. Then more code should be ported to Python 3 to slowly port the whole Cinder code base. Well, that's exactly the plan approved for Nova, and other OpenStack applications have the same (or a similar) plan for Python 3. About the usage of six: we chose six to add Python 3 support to all other OpenStack libraries and applications. Sorry if it makes the code more ugly. More information on porting code to Python 3: https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3 Victor Stinner aka haypo __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder] Port Cinder to Python 3
Hi, There is an ongoing effort to port OpenStack applications to Python 3 during the Liberty cycle. The following table gives the status of each application and links to patches: https://wiki.openstack.org/wiki/Python3#OpenStack_applications Specs were accepted for: * heat: https://review.openstack.org/#/c/175340/ * keystone: https://review.openstack.org/#/c/177380/ * neutron: https://review.openstack.org/#/c/172962/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
