Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
On Tue, Mar 18, 2014 at 1:37 AM, Angus Salkeld angus.salk...@rackspace.comwrote: On 18/03/14 07:39 +0530, Noorul Islam Kamal Malmiyoda wrote: On Tue, Mar 18, 2014 at 4:59 AM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? I think we should not make python33 gate non-voting as we will miss out issues related others. We can toggle the oslo.messaging related tests to not run in python33. Even if we disable our tests, we can't even add oslo.messaging to our requirements.txt as it can't even be installed. Actually, Julien recently added support to pbr for separate requirements files for python 3 (requirements-py3.txt and test-requirements-py3.txt). If the python 3 file exists, the default file is not used at all, so it is possible to list completely different requirements for python 2 and 3. Doug The only practical solution I can see is to make py33 non-voting until oslo.messaging can handle py33. -Angus Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
In addition to the installation requirements, we also need to deal with the code-level changes. For example, when using the eventlet executor eventlet needs to have been imported very early by the application so it can monkeypatch the I/O libraries. When not using the eventlet executor, that monkeypatching should not be done because it will interfere with the regular I/O. So now we have an operation that needs to happen during code initialization that is dictated by a configuration option (which executor is being used) only available after all of the code initialization has happened. My first impression is that when we have an executor that works with asyncio/trollius we will want to drop eventlet entirely, but that's just a first impression. There may be similar trade-offs with the other libraries. Victor, what do you think? Doug On Mon, Mar 17, 2014 at 9:52 PM, Davanum Srinivas dava...@gmail.com wrote: Adrian, We are too close to icehouse release candidates to bump up global requirements with new rev of oslo.messaging. So even if we all agree and cut a new rev of oslo.messaging it's too late for icehouse as release candidates are rolling next week. I'd definitely support a way to get python33 support via trollius or thread-exec that Joshua pointed out very soon. It may be a good idea to keep solum's py33 non-voting till we nail down all other other dependencies, so +1 for that as well. -- dims On Mon, Mar 17, 2014 at 7:29 PM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? Thanks, Adrian [1] https://review.openstack.org/70948 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
Hi, My patches for Oslo Messaging are naive and inefficient. They are just a first step to prepare OpenStack for Trollius. I chose to run asyncio event loop in its own dedicated thread and dispatch messages in a pool of threads. It would be nice to run asyncio event loop in the main thread, as the last blocker function of the main() function of applications. You can already pass your asyncio event loop with my patch, the thread is only created if you don't pass the loop. I wrote that my patch is inefficient because it calls Olso Messaging listener with a short timeout (ex: 500 ms) in a busy-loop. The timeout is needed to be able to exit the threads. Later, each driver should be modified to give access to the low-level file descriptor, so asyncio can react to ready to read and ready to write events. None of my Trollius patches has been merged yet. I have no experience of Trollius cooperating (or not) with eventlet yet. There are different options: - run asyncio tasks in an eventlet event loop - run eventlet tasks in an asyncio event loop - run asyncio and eventlet in two separated event loops (eventlet API is not written as an event loop), typically in two threads The best would be to use asyncio with non-blocking file descriptors, without eventlet, and register these file descriptors in asyncio. But it cannot be done right now, it requires too much changes. Sorry, I have no concrete code to show you right now. I stopped working on Trollius in OpenStack because I was asked to wait after Icehouse release. Victor In addition to the installation requirements, we also need to deal with the code-level changes. For example, when using the eventlet executor eventlet needs to have been imported very early by the application so it can monkeypatch the I/O libraries. When not using the eventlet executor, that monkeypatching should not be done because it will interfere with the regular I/O. So now we have an operation that needs to happen during code initialization that is dictated by a configuration option (which executor is being used) only available after all of the code initialization has happened. My first impression is that when we have an executor that works with asyncio/trollius we will want to drop eventlet entirely, but that's just a first impression. There may be similar trade-offs with the other libraries. Victor, what do you think? Doug On Mon, Mar 17, 2014 at 9:52 PM, Davanum Srinivas dava...@gmail.com wrote: Adrian, We are too close to icehouse release candidates to bump up global requirements with new rev of oslo.messaging. So even if we all agree and cut a new rev of oslo.messaging it's too late for icehouse as release candidates are rolling next week. I'd definitely support a way to get python33 support via trollius or thread-exec that Joshua pointed out very soon. It may be a good idea to keep solum's py33 non-voting till we nail down all other other dependencies, so +1 for that as well. -- dims On Mon, Mar 17, 2014 at 7:29 PM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? Thanks, Adrian [1] https://review.openstack.org/70948 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
Awesome, great to see this, will try it out :-) Is that in the recently released pbr (0.7.0?) From: Doug Hellmann doug.hellm...@dreamhost.commailto:doug.hellm...@dreamhost.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 at 4:51 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging? On Tue, Mar 18, 2014 at 1:37 AM, Angus Salkeld angus.salk...@rackspace.commailto:angus.salk...@rackspace.com wrote: On 18/03/14 07:39 +0530, Noorul Islam Kamal Malmiyoda wrote: On Tue, Mar 18, 2014 at 4:59 AM, Adrian Otto adrian.o...@rackspace.commailto:adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? I think we should not make python33 gate non-voting as we will miss out issues related others. We can toggle the oslo.messaging related tests to not run in python33. Even if we disable our tests, we can't even add oslo.messaging to our requirements.txt as it can't even be installed. Actually, Julien recently added support to pbr for separate requirements files for python 3 (requirements-py3.txt and test-requirements-py3.txt). If the python 3 file exists, the default file is not used at all, so it is possible to list completely different requirements for python 2 and 3. Doug The only practical solution I can see is to make py33 non-voting until oslo.messaging can handle py33. -Angus Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
On Tue, Mar 18, 2014 at 2:21 PM, Joshua Harlow harlo...@yahoo-inc.comwrote: Awesome, great to see this, will try it out :-) Is that in the recently released pbr (0.7.0?) That feature was added in pbr 0.6. Doug From: Doug Hellmann doug.hellm...@dreamhost.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 at 4:51 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging? On Tue, Mar 18, 2014 at 1:37 AM, Angus Salkeld angus.salk...@rackspace.com wrote: On 18/03/14 07:39 +0530, Noorul Islam Kamal Malmiyoda wrote: On Tue, Mar 18, 2014 at 4:59 AM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? I think we should not make python33 gate non-voting as we will miss out issues related others. We can toggle the oslo.messaging related tests to not run in python33. Even if we disable our tests, we can't even add oslo.messaging to our requirements.txt as it can't even be installed. Actually, Julien recently added support to pbr for separate requirements files for python 3 (requirements-py3.txt and test-requirements-py3.txt). If the python 3 file exists, the default file is not used at all, so it is possible to list completely different requirements for python 2 and 3. Doug The only practical solution I can see is to make py33 non-voting until oslo.messaging can handle py33. -Angus Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? Thanks, Adrian [1] https://review.openstack.org/70948 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
https://review.openstack.org/#/c/70914/ could also be another option, besides trollius. Although are all the dependencies that currently exist (besides eventlet) also py33 compatible?? From https://wiki.openstack.org/wiki/Python3 it seems like qpid-python might be a problem to? -Josh From: Adrian Otto adrian.o...@rackspace.commailto:adrian.o...@rackspace.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, March 17, 2014 at 3:29 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging? Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? Thanks, Adrian [1] https://review.openstack.org/70948 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
Adrian, We are too close to icehouse release candidates to bump up global requirements with new rev of oslo.messaging. So even if we all agree and cut a new rev of oslo.messaging it's too late for icehouse as release candidates are rolling next week. I'd definitely support a way to get python33 support via trollius or thread-exec that Joshua pointed out very soon. It may be a good idea to keep solum's py33 non-voting till we nail down all other other dependencies, so +1 for that as well. -- dims On Mon, Mar 17, 2014 at 7:29 PM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? Thanks, Adrian [1] https://review.openstack.org/70948 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
On Tue, Mar 18, 2014 at 4:59 AM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? I think we should not make python33 gate non-voting as we will miss out issues related others. We can toggle the oslo.messaging related tests to not run in python33. Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum][Oslo] Next Release of oslo.messaging?
On 18/03/14 07:39 +0530, Noorul Islam Kamal Malmiyoda wrote: On Tue, Mar 18, 2014 at 4:59 AM, Adrian Otto adrian.o...@rackspace.com wrote: Doug Hellmann and Victror Stinner (+ oslo cores), Solum currently depends on a py33 gate. We want to use oslo.messaging, but are worried that in the current state, we will be stuck without py33 support. We hope that by adding the Trollius code[1], and getting a new release number, that we can add the oslo.messaging library to our requirements and proceed with our async messaging plan. I am seeking guidance from you on when the above might happen. If it's a short time, we may just wait for it. If it's a long time, we may need to relax our py33 gate to non-voting in order to prevent breakage of our Stackforge CI while we work with the oslo.messaging code. We are also considering doing an ugly workaround of creating a bunch of worker processes on the same messaging topic until we can clear this concern. Thoughts? I think we should not make python33 gate non-voting as we will miss out issues related others. We can toggle the oslo.messaging related tests to not run in python33. Even if we disable our tests, we can't even add oslo.messaging to our requirements.txt as it can't even be installed. The only practical solution I can see is to make py33 non-voting until oslo.messaging can handle py33. -Angus Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
