Re: [openstack-dev] [oslo] Can we stop global requirements update?
On 05/20/2017 12:04 PM, Julien Danjou wrote: On Fri, May 19 2017, Mike Bayer wrote: IMO that's a bug for them. Of course it's a bug. IIRC Mehdi tried to fix it without much success. I'm inspired to see that Keystone, Nova etc. are able to move between and eventlet backend and a mod_wsgi backend.IMO eventlet is really not needed for those services that present a REST interface. Although for a message queue with lots of long-running connections that receive events, that's a place where I *would* want to use a polling / non-blocking model. But I'd use it explicitly, not with monkeypatching. +1 I'd ask why not oslo.cotyledon but it seems there's a faction here that is overall moving out of the Openstack umbrella in any case. Not oslo because it can be used by other projects than just OpenStack. And it's a condition of success. As Mehdi said, Oslo has been deserted in the recent cycles, so putting a lib there as very little chance of seeing its community and maintenance help grow. Whereas trying to reach the whole Python ecosystem is more likely to get traction. As a maintainer of SQLAlchemy I'm surprised you even suggest that. Or do you plan on doing oslo.sqlalchemy? ;) I do oslo.db (which also is not "abandoned" in any way). the point of oslo is that it is an openstack-centric mediation layer between some common service/library and openstack. it looks like there already is essentially such a layer for cotyledon. I'd just name it "oslo.cotyledon" :) or oslo. something. We have a moose. It's cool. Basically I think openstack should be getting off eventlet in a big way so I guess my sentiment here is that the Gnocchi / Cotyledon /etc. faction is just splitting off rather than serving as any kind of direction for the rest of Openstack to start looking. But that's only an impression, maybe projects will use Cotyledon anyway. If every project goes off and uses something completely different though, then I think we're losing. The point of oslo was to prevent that. I understand your concern and opinion. I think you, me and Mehdi don't have the experience as contributors in OpenStack. I invite you to try moving any major OpenStack project to something like oslo.service2 or Cotyledon or to achieve any technical debt resolution in OpenStack to have a view on hard it is to tackle. Then you'll see where we stand. :) Sure, that's an area where I think the whole direction of openstack would benefit from more centralized planning, but i have been here just enough to observe that this kind of thing has been discussed before and it is of course very tricky to implement. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Fri, May 19 2017, Mike Bayer wrote: > IMO that's a bug for them. Of course it's a bug. IIRC Mehdi tried to fix it without much success. > I'm inspired to see that Keystone, Nova etc. are > able to move between and eventlet backend and a mod_wsgi backend.IMO > eventlet is really not needed for those services that present a REST > interface. > Although for a message queue with lots of long-running connections that > receive > events, that's a place where I *would* want to use a polling / non-blocking > model. But I'd use it explicitly, not with monkeypatching. +1 > I'd ask why not oslo.cotyledon but it seems there's a faction here that is > overall moving out of the Openstack umbrella in any case. Not oslo because it can be used by other projects than just OpenStack. And it's a condition of success. As Mehdi said, Oslo has been deserted in the recent cycles, so putting a lib there as very little chance of seeing its community and maintenance help grow. Whereas trying to reach the whole Python ecosystem is more likely to get traction. As a maintainer of SQLAlchemy I'm surprised you even suggest that. Or do you plan on doing oslo.sqlalchemy? ;) > Basically I think openstack should be getting off eventlet in a big way so I > guess my sentiment here is that the Gnocchi / Cotyledon /etc. faction is just > splitting off rather than serving as any kind of direction for the rest of > Openstack to start looking. But that's only an impression, maybe projects > will > use Cotyledon anyway. If every project goes off and uses something > completely > different though, then I think we're losing. The point of oslo was to > prevent > that. I understand your concern and opinion. I think you, me and Mehdi don't have the experience as contributors in OpenStack. I invite you to try moving any major OpenStack project to something like oslo.service2 or Cotyledon or to achieve any technical debt resolution in OpenStack to have a view on hard it is to tackle. Then you'll see where we stand. :) Especially when your job is not doing that, but e.g. working on Telemetry. :) -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On 05/19/2017 04:23 AM, Mehdi Abaakouk wrote: And some applications rely on implicit internal contract/behavior/assumption. IMO that's a bug for them.I'm inspired to see that Keystone, Nova etc. are able to move between and eventlet backend and a mod_wsgi backend.IMO eventlet is really not needed for those services that present a REST interface. Although for a message queue with lots of long-running connections that receive events, that's a place where I *would* want to use a polling / non-blocking model. But I'd use it explicitly, not with monkeypatching. Since a new API is needed, why not writing a new lib. Anyways when you get rid of eventlet you have so many thing to change to ensure your performance will not drop. While I don't know the specifics for your project(s), I don't buy that in general because IMO eventlet is not giving us any performance boost in the majority of cases. most of our IO is blocking on the database and all the applications have DB connections throttled at about 50 per process at the max, and that's only recently, it used to be just 15. Changing from oslo.service to cotyledon is really easy on the side. I'd ask why not oslo.cotyledon but it seems there's a faction here that is overall moving out of the Openstack umbrella in any case. Docs state: "oslo.service being impossible to fix and bringing an heavy dependency on eventlet, " is there a discussion thread on that? Not really, I just put some comments on reviews and discus this on IRC. Since nobody except Telemetry have expressed/try to get rid of eventlet. Many (most?) of the web services can run under mod_wsgi with threads, Keystone seems to be standard on this now and I get the impression Nova is going in that direction too.(anyone correct me if I'm wrong on any of that, I looked to ask around on IRC but it's too late in the day). For the story we first get rid of eventlet in Telemetry, fixes couple of performance issue due to using threading/process instead greenlet/greenthread. Then we fall into some weird issue due to oslo.service internal implementation. Process not exiting properly, signals not received, deadlock when signal are received, unkillable process, tooz/oslo.messaging heartbeat not scheduled correctly, worker not restarted when they are dead. All of what we expect from oslo.service was not working correctly anymore because we remove the line 'eventlet.monkeypatch()'. So, I've used gevent more than eventlet in my own upstream non-blocking work, and while this opinion is like spilling water in the ocean, I think applications should never use monkeypatching. They should call into the eventlet/gevent greenlet API directly if that's what they want to do. Of course this means that database logic has to move out of greenlets entirely since none of the DBAPIs use non-blocking IO. That's fine. Database-oriented business logic should not be in greenlets.I've written about this as well.If one is familiar enough with greenlets and threads you can write an application that makes explicit use of both. However, that's application level stuff. Web service apps like Nova conductor / Neutron / Keystone should not be aware of any of that. They should be coded to assume nothing about context switching. IMO the threading model is "safer" to code towards since you have to handle locking and concurrency contingencies explicitly without hardwiring that to your assumptions about when context switching is to take place and when it's not. For example, when oslo.service receive a signal, it can arrive on any thread, this thread is paused, the callback is run in this thread context, but if the callback try to discus to your code in this thread, the process lockup, because your code is paused. Python offers tool to avoid that (signal.set_wakeup_fd), but oslo.service don't use it. I have tried to run callbacks only on the main thread with set_wakeup_fd, to avoid this kind of issue but I fail. The whole oslo.service code is clearly not designed to be threadsafe/signalsafe. Well, it works for eventlet because you have only one real thread. And this is just one example on complicated thing I have tried to fix, before starting cotyledon. I've no doubt oslo.service has major eventlet problems baked in, I've looked at it a little bit but didn't go too far with it. That still doesn't mean there shouldn't be an "oslo.service2" that can effectively produce a concurrency-agnostic platform.It of course would have the goal in mind of moving projects off eventlet since as I mentioned, eventlet monkeypatching should not be used which means our services should do most of their "implicitly concurrent" work within threads. Basically I think openstack should be getting off eventlet in a big way so I guess my sentiment here is that the Gnocchi / Cotyledon /etc. faction is just splitting off rather than serving as any kind of d
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Mehdi Abaakouk wrote: Not really, I just put some comments on reviews and discus this on IRC. Since nobody except Telemetry have expressed/try to get rid of eventlet. Octavia is using cotyledon and they have gotten rid of eventlet. Didn't seem like it was that hard either to do it (of course the experience in how easy it was is likely not transferable to other projects...) -Josh __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Excerpts from Mehdi Abaakouk's message of 2017-05-19 10:23:09 +0200: > On Thu, May 18, 2017 at 03:16:20PM -0400, Mike Bayer wrote: > > > > > >On 05/18/2017 02:37 PM, Julien Danjou wrote: > >>On Thu, May 18 2017, Mike Bayer wrote: > >> > >>>I'm not understanding this? do you mean this? > >> > >>In the long run, yes. Unfortunately, we're not happy with the way Oslo > >>libraries are managed and too OpenStack centric. I've tried for the last > >>couple of years to move things on, but it's barely possible to deprecate > >>anything and contribute, so I feel it's safer to start fresh and better > >>alternative. Cotyledon by Mehdi is a good example of what can be > >>achieved. > > > > > >here's cotyledon: > > > >https://cotyledon.readthedocs.io/en/latest/ > > > > > >replaces oslo.service with a multiprocessing approach that doesn't use > >eventlet. great! any openstack service that rides on oslo.service > >would like to be able to transparently switch from eventlet to > >multiprocessing the same way they can more or less switch to mod_wsgi > >at the moment. IMO this should be part of oslo.service itself. > > I have quickly presented cotyledon some summit ago, we said we will wait > to see if other projects want to get rid of eventlet before adopting > such new lib (or merge it with oslo.service). > > But for now, the lib is still under telemetry umbrella. > > Keeping the current API and supporting both are (I think) impossible. > The current API is too eventlet centric. And some applications rely > on implicit internal contract/behavior/assumption. > > Dealing about concurrent/thread/signal safety in multithreading app or > eventlet app is already hard enough. So having the lib that deals with > both is even harder. We already have oslo.messaging that deals with > 3 threads models, this is just an unending story of race conditions. > > Since a new API is needed, why not writing a new lib. Anyways when you > get rid of eventlet you have so many thing to change to ensure your > performance will not drop. Changing from oslo.service to cotyledon is > really easy on the side. > > >Docs state: "oslo.service being impossible to fix and bringing an > >heavy dependency on eventlet, " is there a discussion thread on that? > > Not really, I just put some comments on reviews and discus this on IRC. > Since nobody except Telemetry have expressed/try to get rid of eventlet. > > For the story we first get rid of eventlet in Telemetry, fixes couple of > performance issue due to using threading/process instead > greenlet/greenthread. > > Then we fall into some weird issue due to oslo.service internal > implementation. Process not exiting properly, signals not received, > deadlock when signal are received, unkillable process, > tooz/oslo.messaging heartbeat not scheduled correctly, worker not > restarted when they are dead. All of what we expect from oslo.service > was not working correctly anymore because we remove the line > 'eventlet.monkeypatch()'. > > For example, when oslo.service receive a signal, it can arrive on any > thread, this thread is paused, the callback is run in this thread > context, but if the callback try to discus to your code in this thread, > the process lockup, because your code is paused. Python > offers tool to avoid that (signal.set_wakeup_fd), but oslo.service don't > use it. I have tried to run callbacks only on the main thread with > set_wakeup_fd, to avoid this kind of issue but I fail. The whole > oslo.service code is clearly not designed to be threadsafe/signalsafe. > Well, it works for eventlet because you have only one real thread. > > And this is just one example on complicated thing I have tried to fix, > before starting cotyledon. > > >I'm finding it hard to believe that only a few years ago, everyone saw > >the wisdom of not re-implementing everything in their own projects and > >using a common layer like oslo, and already that whole situation is > >becoming forgotten - not just for consistency, but also when a bug is > >found, if fixed in oslo it gets fixed for everyone. > > Because the internal of cotyledon and oslo.service are so different. > Having the code in oslo or not doesn't help for maintenance anymore. > Cotyledon is a lib, code and bugs :) can already be shared between > projects that doesn't want eventlet. Yes, I remember discussing this some time ago and I agree that starting a new library was the right approach. The changes needed to make oslo.service work without eventlet are too big, and rather than have 2 separate implementations in the same library a second library makes sense. > >An increase in the scope of oslo is essential to dealing with the > >issue of "complexity" in openstack. > > Increasing the scope of oslo works only if libs have maintainers. But > most of them lack of people today. Most of oslo libs are in maintenance > mode. But that another subject. > > > The state of openstack as dozens > >of individual software projects e
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Fri, May 19 2017, Mehdi Abaakouk wrote: > Not really, I just put some comments on reviews and discus this on IRC. > Since nobody except Telemetry have expressed/try to get rid of eventlet. TBH Keystone get rid of it too. But they only provide WSGI servers. They don't build any daemon so they don't need and user either Cotyledon nor oslo.service. :) > Because the internal of cotyledon and oslo.service are so different. > Having the code in oslo or not doesn't help for maintenance anymore. > Cotyledon is a lib, code and bugs :) can already be shared between > projects that doesn't want eventlet. Cotyledon is explicitly better just by being out of Oslo, because it's usable by the whole Python ecosystem. :) -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Thu, May 18, 2017 at 03:16:20PM -0400, Mike Bayer wrote: On 05/18/2017 02:37 PM, Julien Danjou wrote: On Thu, May 18 2017, Mike Bayer wrote: I'm not understanding this? do you mean this? In the long run, yes. Unfortunately, we're not happy with the way Oslo libraries are managed and too OpenStack centric. I've tried for the last couple of years to move things on, but it's barely possible to deprecate anything and contribute, so I feel it's safer to start fresh and better alternative. Cotyledon by Mehdi is a good example of what can be achieved. here's cotyledon: https://cotyledon.readthedocs.io/en/latest/ replaces oslo.service with a multiprocessing approach that doesn't use eventlet. great! any openstack service that rides on oslo.service would like to be able to transparently switch from eventlet to multiprocessing the same way they can more or less switch to mod_wsgi at the moment. IMO this should be part of oslo.service itself. I have quickly presented cotyledon some summit ago, we said we will wait to see if other projects want to get rid of eventlet before adopting such new lib (or merge it with oslo.service). But for now, the lib is still under telemetry umbrella. Keeping the current API and supporting both are (I think) impossible. The current API is too eventlet centric. And some applications rely on implicit internal contract/behavior/assumption. Dealing about concurrent/thread/signal safety in multithreading app or eventlet app is already hard enough. So having the lib that deals with both is even harder. We already have oslo.messaging that deals with 3 threads models, this is just an unending story of race conditions. Since a new API is needed, why not writing a new lib. Anyways when you get rid of eventlet you have so many thing to change to ensure your performance will not drop. Changing from oslo.service to cotyledon is really easy on the side. Docs state: "oslo.service being impossible to fix and bringing an heavy dependency on eventlet, " is there a discussion thread on that? Not really, I just put some comments on reviews and discus this on IRC. Since nobody except Telemetry have expressed/try to get rid of eventlet. For the story we first get rid of eventlet in Telemetry, fixes couple of performance issue due to using threading/process instead greenlet/greenthread. Then we fall into some weird issue due to oslo.service internal implementation. Process not exiting properly, signals not received, deadlock when signal are received, unkillable process, tooz/oslo.messaging heartbeat not scheduled correctly, worker not restarted when they are dead. All of what we expect from oslo.service was not working correctly anymore because we remove the line 'eventlet.monkeypatch()'. For example, when oslo.service receive a signal, it can arrive on any thread, this thread is paused, the callback is run in this thread context, but if the callback try to discus to your code in this thread, the process lockup, because your code is paused. Python offers tool to avoid that (signal.set_wakeup_fd), but oslo.service don't use it. I have tried to run callbacks only on the main thread with set_wakeup_fd, to avoid this kind of issue but I fail. The whole oslo.service code is clearly not designed to be threadsafe/signalsafe. Well, it works for eventlet because you have only one real thread. And this is just one example on complicated thing I have tried to fix, before starting cotyledon. I'm finding it hard to believe that only a few years ago, everyone saw the wisdom of not re-implementing everything in their own projects and using a common layer like oslo, and already that whole situation is becoming forgotten - not just for consistency, but also when a bug is found, if fixed in oslo it gets fixed for everyone. Because the internal of cotyledon and oslo.service are so different. Having the code in oslo or not doesn't help for maintenance anymore. Cotyledon is a lib, code and bugs :) can already be shared between projects that doesn't want eventlet. An increase in the scope of oslo is essential to dealing with the issue of "complexity" in openstack. Increasing the scope of oslo works only if libs have maintainers. But most of them lack of people today. Most of oslo libs are in maintenance mode. But that another subject. The state of openstack as dozens of individual software projects each with their own idiosyncratic quirks, CLIs, process and deployment models, and everything else that is visible to operators is ground zero for perceived operator complexity. Cotyledon have been written to be Openstack agnostic. But I have also write an optional module within the library to glue oslo.config and cotyledon. Mainly to mimic the oslo.config options/reload of oslo.service and make operators experience unchanged for Openstack people. -- Mehdi Abaakouk mail: sil...@sileht.net irc: sileht signature.asc Description: PGP signature __
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Thu, May 18 2017, Mike Bayer wrote: > replaces oslo.service with a multiprocessing approach that doesn't use > eventlet. great! any openstack service that rides on oslo.service would like > to be able to transparently switch from eventlet to multiprocessing the same > way they can more or less switch to mod_wsgi at the moment.IMO this should > be part of oslo.service itself. Docs state: "oslo.service being impossible > to > fix and bringing an heavy dependency on eventlet, " is there a discussion > thread on that? Yes, and many reviews around that. I'll let Mehdi comments if he feels like it. :) > I'm finding it hard to believe that only a few years ago, everyone saw the > wisdom of not re-implementing everything in their own projects and using a > common layer like oslo, and already that whole situation is becoming forgotten > - not just for consistency, but also when a bug is found, if fixed in oslo it > gets fixed for everyone. I guess it depends what you mean by everyone. FTR, one of the two first projects in OpenStack, Swift, never used anything from Oslo for anything and always refused to depends on any of its library. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On 05/18/2017 02:37 PM, Julien Danjou wrote: On Thu, May 18 2017, Mike Bayer wrote: I'm not understanding this? do you mean this? In the long run, yes. Unfortunately, we're not happy with the way Oslo libraries are managed and too OpenStack centric. I've tried for the last couple of years to move things on, but it's barely possible to deprecate anything and contribute, so I feel it's safer to start fresh and better alternative. Cotyledon by Mehdi is a good example of what can be achieved. here's cotyledon: https://cotyledon.readthedocs.io/en/latest/ replaces oslo.service with a multiprocessing approach that doesn't use eventlet. great! any openstack service that rides on oslo.service would like to be able to transparently switch from eventlet to multiprocessing the same way they can more or less switch to mod_wsgi at the moment.IMO this should be part of oslo.service itself. Docs state: "oslo.service being impossible to fix and bringing an heavy dependency on eventlet, " is there a discussion thread on that? I'm finding it hard to believe that only a few years ago, everyone saw the wisdom of not re-implementing everything in their own projects and using a common layer like oslo, and already that whole situation is becoming forgotten - not just for consistency, but also when a bug is found, if fixed in oslo it gets fixed for everyone. An increase in the scope of oslo is essential to dealing with the issue of "complexity" in openstack. The state of openstack as dozens of individual software projects each with their own idiosyncratic quirks, CLIs, process and deployment models, and everything else that is visible to operators is ground zero for perceived operator complexity. Though to comment on your example, oslo.db is probably the most useful Oslo library that Gnocchi depends on and that won't go away in a snap. :-( __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Thu, May 18 2017, Mike Bayer wrote: > I'm not understanding this? do you mean this? In the long run, yes. Unfortunately, we're not happy with the way Oslo libraries are managed and too OpenStack centric. I've tried for the last couple of years to move things on, but it's barely possible to deprecate anything and contribute, so I feel it's safer to start fresh and better alternative. Cotyledon by Mehdi is a good example of what can be achieved. Though to comment on your example, oslo.db is probably the most useful Oslo library that Gnocchi depends on and that won't go away in a snap. :-( -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On 05/16/2017 05:42 AM, Julien Danjou wrote: On Wed, Apr 19 2017, Julien Danjou wrote: So Gnocchi gate is all broken (agan) because it depends on "pbr" and some new release of oslo.* depends on pbr!=2.1.0. Same things happened today with Babel. As far as Gnocchi is concerned, we're going to take the easiest route and remove all our oslo dependencies over the next months for sanely maintained alternative at this point. I'm not understanding this? do you mean this? diff --git a/gnocchi/indexer/sqlalchemy.py b/gnocchi/indexer/sqlalchemy.py index 3497b52..0ae99fd 100644 --- a/gnocchi/indexer/sqlalchemy.py +++ b/gnocchi/indexer/sqlalchemy.py @@ -22,11 +22,7 @@ import uuid from alembic import migration from alembic import operations -import oslo_db.api -from oslo_db import exception -from oslo_db.sqlalchemy import enginefacade -from oslo_db.sqlalchemy import utils as oslo_db_utils -from oslo_log import log +from ??? import ??? try: import psycopg2 except ImportError: Cheers, __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Tue, May 16 2017, Andreas Jaeger wrote: > It is needed to generate the translations, but can't we move it for > oslo-i18n into test-requirements? I've pushed this and it seems to work, pretty sure it's safe. https://review.openstack.org/#/c/465014/ If we can merge this today and then release quickly after that'd be a great help -_- > But os-testr does not need Babel at all - let's remove it, > https://review.openstack.org/465023 Arf, sure! I can only +1 though :( -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On 2017-05-16 12:10, Julien Danjou wrote: > On Tue, May 16 2017, Andreas Jaeger wrote: > >> what exactly happened with Babel? >> >> I see in global-requirements the following: >> Babel>=2.3.4,!=2.4.0 # BSD >> >> that shouldn't case a problem - or does it? Or what's the problem? > > Damn, at the moment I pressed the `Sent' button I thought "You just > complained without including much detail idiot". Sorry about that! no worries. > One of the log that fails: > > > http://logs.openstack.org/13/464713/2/check/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/db61bdf/console.html > > > Basically oslo.policy pulls oslo.i18n which pulls Babel!=2.4.0 > But Babel is already pulled by os-testr which depends on >=2.3.4. and os-testr is not importing global-requirements: https://review.openstack.org/#/c/454511/ > So pip does not solve that (unfortunately) and then the failure is: > > 2017-05-16 05:08:43.629772 | 2017-05-16 05:08:43.503 10699 ERROR gnocchi > ContextualVersionConflict: (Babel 2.4.0 > (/home/jenkins/workspace/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/upgrade/lib/python2.7/site-packages), > Requirement.parse('Babel!=2.4.0,>=2.3.4'), set(['oslo.i18n'])) > > I'm pretty sure Babel should not even be in the requirements list of > oslo.i18n since it's not a runtime dependency AFAIU. It is needed to generate the translations, but can't we move it for oslo-i18n into test-requirements? But os-testr does not need Babel at all - let's remove it, https://review.openstack.org/465023 Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Tue, May 16 2017, Andreas Jaeger wrote: > what exactly happened with Babel? > > I see in global-requirements the following: > Babel>=2.3.4,!=2.4.0 # BSD > > that shouldn't case a problem - or does it? Or what's the problem? Damn, at the moment I pressed the `Sent' button I thought "You just complained without including much detail idiot". Sorry about that! One of the log that fails: http://logs.openstack.org/13/464713/2/check/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/db61bdf/console.html Basically oslo.policy pulls oslo.i18n which pulls Babel!=2.4.0 But Babel is already pulled by os-testr which depends on >=2.3.4. So pip does not solve that (unfortunately) and then the failure is: 2017-05-16 05:08:43.629772 | 2017-05-16 05:08:43.503 10699 ERROR gnocchi ContextualVersionConflict: (Babel 2.4.0 (/home/jenkins/workspace/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/upgrade/lib/python2.7/site-packages), Requirement.parse('Babel!=2.4.0,>=2.3.4'), set(['oslo.i18n'])) I'm pretty sure Babel should not even be in the requirements list of oslo.i18n since it's not a runtime dependency AFAIU. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On 2017-05-16 11:42, Julien Danjou wrote: > On Wed, Apr 19 2017, Julien Danjou wrote: > >> So Gnocchi gate is all broken (agan) because it depends on "pbr" and >> some new release of oslo.* depends on pbr!=2.1.0. > > Same things happened today with Babel. As far as Gnocchi is concerned, > we're going to take the easiest route and remove all our oslo > dependencies over the next months for sanely maintained alternative at > this point. what exactly happened with Babel? I see in global-requirements the following: Babel>=2.3.4,!=2.4.0 # BSD that shouldn't case a problem - or does it? Or what's the problem? Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Wed, Apr 19 2017, Julien Danjou wrote: > So Gnocchi gate is all broken (agan) because it depends on "pbr" and > some new release of oslo.* depends on pbr!=2.1.0. Same things happened today with Babel. As far as Gnocchi is concerned, we're going to take the easiest route and remove all our oslo dependencies over the next months for sanely maintained alternative at this point. Cheers, -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Fri, Apr 21 2017, Doug Hellmann wrote: > My memory of the outcome of that session was that we needed to maintain > co-installability; that we could continue to keep an eye on the > container space as an alternative; and that a new team of maintainers > would take over the requirements list (which was my secret agenda for > proposing that we stop doing it at all). Just to come back to the original topic, co-installability is *not* impacted by my proposal of stopping the sync. On the opposite, last week it was very hard to install both Gnocchi and Oslo because of Oslo… So it would *improve* co-installability, not remove it. My 2c, -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Excerpts from Joshua Harlow's message of 2017-04-20 22:31:19 -0700: > Doug Hellmann wrote: > > Excerpts from gordon chung's message of 2017-04-20 17:12:26 +: > >> On 20/04/17 01:32 AM, Joshua Harlow wrote: > >>> Wasn't there also some decision made in austin (?) about how we as a > >>> group stated something along the line of co-installability isn't as > >>> important as it once was (and may not even be practical or what people > >>> care about anymore anyway)? > > > > I don't remember that, but I may not have been in the room at the > > time. In the past when we've discussed that idea, we've continued > > to maintain that co-installability is still needed for distributors > > who have packaging constraints that require it and for use cases > > like single-node deployments for POCs. > > Ya, looking back I think it was: > > https://etherpad.openstack.org/p/newton-global-requirements > > I think that was robert that lead that session, but I might be incorrect > there. That was me, though Robert was definitely present and vocal. My memory of the outcome of that session was that we needed to maintain co-installability; that we could continue to keep an eye on the container space as an alternative; and that a new team of maintainers would take over the requirements list (which was my secret agenda for proposing that we stop doing it at all). During the session in Barcelona (I previously said Austin, but misremembered the location) we agreed that we could stop syncing, as long as we maintained co-installability by ensuring that everyone's requirements lists intersect with the upper-constraints.txt list. That work has been started. As far as I know, we have never said we could drop co-installability as a requirement. We have wished we could, but have not said we can. Doug > > > > >>> With kolla becoming more popular (tripleo I think is using it, and ...) > >>> and the containers it creates making isolated per-application > >>> environments it makes me wonder what of global-requirements is still > >>> valid (as a concept) and what isn't. > > > > We still need to review dependencies for license compatibility, to > > minimize redundancy, and to ensure that we're not adding things to > > the list that are not being maintained upstream. Even if we stop syncing > > versions, official projects need to be those reviews, and having the > > global list is a way to ensure that the reviews are done. > > > >>> I do remember the days of free for all requirements (or requirements > >>> sometimes just put/stashed in devstack vs elsewhere), which I don't > >>> really want to go back to; but if we finally all agree that > >>> co-installability isn't what people actually do and/or care about > >>> (anymore?) then maybe we can re-think some things? > >> agree with all of ^... but i imagine to make progress on this, we'd have > >> to change/drop devstack usage in gate and that will take forever and a > >> lifetime (is that a chick flick title?) given how embedded devstack is > >> in everything. it seems like the solution starts with devstack. > >> > >> cheers, > >> > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
2017-04-19 23:10 GMT+08:00 Clark Boylan : > On Wed, Apr 19, 2017, at 05:54 AM, Julien Danjou wrote: > > Hoy, > > > > So Gnocchi gate is all broken (agan) because it depends on "pbr" and > > some new release of oslo.* depends on pbr!=2.1.0. > > > > Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 > > that got in banished by requirements Gods. It does not prevent it to be > > used e.g. to install the software or get version information. But it > > does break anything that is not in OpenStack because well, pip installs > > the latest pbr (2.1.0) and then oslo.* is unhappy about it. > > It actually breaks everything, including OpenStack. Shade and others are > affected by this as well. The specific problem here is that PBR is a > setup_requires which means it gets installed by easy_install before > anything else. This means that the requirements restrictions are not > applied to it (neither are the constraints). So you get latest PBR from > easy_install then later when something checks the requirements > (pkg_resources console script entrypoints?) they break because latest > PBR isn't allowed. > > What we disscuss here is the way to avoid break things, not sure we add pbr into periodic-**-with-oslo-master works in https://review.openstack.org/458753 We need to stop pinning PBR and more generally stop pinning any > setup_requires (there are a few more now since setuptools itself is > starting to use that to list its deps rather than bundling them). > > > So I understand the culprit is probably pip installation scheme, and we > > can blame him until we fix it. I'm also trying to push pbr 2.2.0 to > > avoid the entire issue. > > Yes, a new release of PBR undoing the "pin" is the current sane step > forward for fixing this particular issue. Monty also suggested that we > gate global requirements changes on requiring changes not pin any > setup_requires. > > > But for the future, could we stop updating the requirements in oslo libs > > for no good reason? just because some random OpenStack project hit a bug > > somewhere? > > > > For example, I've removed requirements update on tooz¹ for more than a > > year now, which did not break *anything* in the meantime, proving that > > this process is giving more problem than solutions. Oslo libs doing that > > automatic update introduce more pain for all consumers than anything (at > > least not in OpenStack). > > You are likely largely shielded by the constraints list here which is > derivative of the global requirements list. Basically by using > constraints you get distilled global requirements and even without being > part of the requirements updates you'd be shielded from breakages when > installed via something like devstack or other deployment method using > constraints. > > > So if we care about Oslo users outside OpenStack, I beg us to stop this > > crazyness. If we don't, we'll just spend time getting rid of Oslo over > > the long term… > > I think we know from experience that just stopping (eg reverting to the > situation we had before requirements and constraints) would lead to > sadness. Installations would frequently be impossible due to some > unresolvable error in dependency resolution. Do you have some > alternative in mind? Perhaps we loosen the in project requirements and > explicitly state that constraints are known to work due to testing and > users should use constraints? That would give users control to manage > their own constraints list too if they wish. Maybe we do this in > libraries while continuing to be more specific in applications? > > > > > My 2c, > > > > Cheers, > > > > ¹ Unless some API changed in a dep and we needed to raise the dep, > > obviously. > > > > -- > > Julien Danjou > > # Free Software hacker > > # https://julien.danjou.info > > I don't have all the answers, but am fairly certain the situation we > have today is better than the one from several years ago. It is just not > perfect. I think we are better served by refining the current setup or > replacing it with something better but not by reverting. > > Clark > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- ChangBo Guo(gcb) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Doug Hellmann wrote: Excerpts from gordon chung's message of 2017-04-20 17:12:26 +: On 20/04/17 01:32 AM, Joshua Harlow wrote: Wasn't there also some decision made in austin (?) about how we as a group stated something along the line of co-installability isn't as important as it once was (and may not even be practical or what people care about anymore anyway)? I don't remember that, but I may not have been in the room at the time. In the past when we've discussed that idea, we've continued to maintain that co-installability is still needed for distributors who have packaging constraints that require it and for use cases like single-node deployments for POCs. Ya, looking back I think it was: https://etherpad.openstack.org/p/newton-global-requirements I think that was robert that lead that session, but I might be incorrect there. With kolla becoming more popular (tripleo I think is using it, and ...) and the containers it creates making isolated per-application environments it makes me wonder what of global-requirements is still valid (as a concept) and what isn't. We still need to review dependencies for license compatibility, to minimize redundancy, and to ensure that we're not adding things to the list that are not being maintained upstream. Even if we stop syncing versions, official projects need to be those reviews, and having the global list is a way to ensure that the reviews are done. I do remember the days of free for all requirements (or requirements sometimes just put/stashed in devstack vs elsewhere), which I don't really want to go back to; but if we finally all agree that co-installability isn't what people actually do and/or care about (anymore?) then maybe we can re-think some things? agree with all of ^... but i imagine to make progress on this, we'd have to change/drop devstack usage in gate and that will take forever and a lifetime (is that a chick flick title?) given how embedded devstack is in everything. it seems like the solution starts with devstack. cheers, __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Excerpts from gordon chung's message of 2017-04-20 17:12:26 +: > > On 20/04/17 01:32 AM, Joshua Harlow wrote: > > Wasn't there also some decision made in austin (?) about how we as a > > group stated something along the line of co-installability isn't as > > important as it once was (and may not even be practical or what people > > care about anymore anyway)? I don't remember that, but I may not have been in the room at the time. In the past when we've discussed that idea, we've continued to maintain that co-installability is still needed for distributors who have packaging constraints that require it and for use cases like single-node deployments for POCs. > > With kolla becoming more popular (tripleo I think is using it, and ...) > > and the containers it creates making isolated per-application > > environments it makes me wonder what of global-requirements is still > > valid (as a concept) and what isn't. We still need to review dependencies for license compatibility, to minimize redundancy, and to ensure that we're not adding things to the list that are not being maintained upstream. Even if we stop syncing versions, official projects need to be those reviews, and having the global list is a way to ensure that the reviews are done. > > I do remember the days of free for all requirements (or requirements > > sometimes just put/stashed in devstack vs elsewhere), which I don't > > really want to go back to; but if we finally all agree that > > co-installability isn't what people actually do and/or care about > > (anymore?) then maybe we can re-think some things? > > agree with all of ^... but i imagine to make progress on this, we'd have > to change/drop devstack usage in gate and that will take forever and a > lifetime (is that a chick flick title?) given how embedded devstack is > in everything. it seems like the solution starts with devstack. > > cheers, > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On 20/04/17 01:32 AM, Joshua Harlow wrote: > Wasn't there also some decision made in austin (?) about how we as a > group stated something along the line of co-installability isn't as > important as it once was (and may not even be practical or what people > care about anymore anyway)? > > With kolla becoming more popular (tripleo I think is using it, and ...) > and the containers it creates making isolated per-application > environments it makes me wonder what of global-requirements is still > valid (as a concept) and what isn't. > > I do remember the days of free for all requirements (or requirements > sometimes just put/stashed in devstack vs elsewhere), which I don't > really want to go back to; but if we finally all agree that > co-installability isn't what people actually do and/or care about > (anymore?) then maybe we can re-think some things? agree with all of ^... but i imagine to make progress on this, we'd have to change/drop devstack usage in gate and that will take forever and a lifetime (is that a chick flick title?) given how embedded devstack is in everything. it seems like the solution starts with devstack. cheers, -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Excerpts from Matthew Oliver's message of 2017-04-20 14:41:38 +1000: > We have started this work. I've been working on: > https://review.openstack.org/#/c/444718/ Wonderful! I'm sorry I didn't realize you were working on it. Thank you! > Which will do requirement checks, as specified in the Pike PTG ehterpad for > Tuesday morning: > https://etherpad.openstack.org/p/relmgt-stable-requirements-ptg-pike (line > 40+). > > Once done, Tony and I were going to start testing it on the experimental > pipeline for Swift and Nova. That sounds like a good approach. I'll subscribe to the review and follow along. Doug > > Regards, > Matt > > On Thu, Apr 20, 2017 at 2:34 AM, Doug Hellmann > wrote: > > > Excerpts from Clark Boylan's message of 2017-04-19 08:10:43 -0700: > > > On Wed, Apr 19, 2017, at 05:54 AM, Julien Danjou wrote: > > > > Hoy, > > > > > > > > So Gnocchi gate is all broken (agan) because it depends on "pbr" > > and > > > > some new release of oslo.* depends on pbr!=2.1.0. > > > > > > > > Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 > > > > that got in banished by requirements Gods. It does not prevent it to be > > > > used e.g. to install the software or get version information. But it > > > > does break anything that is not in OpenStack because well, pip installs > > > > the latest pbr (2.1.0) and then oslo.* is unhappy about it. > > > > > > It actually breaks everything, including OpenStack. Shade and others are > > > affected by this as well. The specific problem here is that PBR is a > > > setup_requires which means it gets installed by easy_install before > > > anything else. This means that the requirements restrictions are not > > > applied to it (neither are the constraints). So you get latest PBR from > > > easy_install then later when something checks the requirements > > > (pkg_resources console script entrypoints?) they break because latest > > > PBR isn't allowed. > > > > > > We need to stop pinning PBR and more generally stop pinning any > > > setup_requires (there are a few more now since setuptools itself is > > > starting to use that to list its deps rather than bundling them). > > > > > > > So I understand the culprit is probably pip installation scheme, and we > > > > can blame him until we fix it. I'm also trying to push pbr 2.2.0 to > > > > avoid the entire issue. > > > > > > Yes, a new release of PBR undoing the "pin" is the current sane step > > > forward for fixing this particular issue. Monty also suggested that we > > > gate global requirements changes on requiring changes not pin any > > > setup_requires. > > > > > > > But for the future, could we stop updating the requirements in oslo > > libs > > > > for no good reason? just because some random OpenStack project hit a > > bug > > > > somewhere? > > > > > > > > For example, I've removed requirements update on tooz¹ for more than a > > > > year now, which did not break *anything* in the meantime, proving that > > > > this process is giving more problem than solutions. Oslo libs doing > > that > > > > automatic update introduce more pain for all consumers than anything > > (at > > > > least not in OpenStack). > > > > > > You are likely largely shielded by the constraints list here which is > > > derivative of the global requirements list. Basically by using > > > constraints you get distilled global requirements and even without being > > > part of the requirements updates you'd be shielded from breakages when > > > installed via something like devstack or other deployment method using > > > constraints. > > > > > > > So if we care about Oslo users outside OpenStack, I beg us to stop this > > > > crazyness. If we don't, we'll just spend time getting rid of Oslo over > > > > the long term… > > > > > > I think we know from experience that just stopping (eg reverting to the > > > situation we had before requirements and constraints) would lead to > > > sadness. Installations would frequently be impossible due to some > > > unresolvable error in dependency resolution. Do you have some > > > alternative in mind? Perhaps we loosen the in project requirements and > > > explicitly state that constraints are known to work due to testing and > > > users should use constraints? That would give users control to manage > > > their own constraints list too if they wish. Maybe we do this in > > > libraries while continuing to be more specific in applications? > > > > At the meeting in Austin, the requirements team accepted my proposal > > to stop syncing requirements updates into projects, as described > > in https://etherpad.openstack.org/p/ocata-requirements-notes > > > > We haven't been able to find anyone to work on the implementation, > > though. I is my understanding that Tony did contact the Telemetry > > and Swift teams, who are most interested in this area of change, > > about devoting some resources to the tasks outlined in the proposal. > > > > Doug > > > > > > > > > > > > > My 2c, > > > > > >
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Doug Hellmann wrote: Excerpts from Clark Boylan's message of 2017-04-19 08:10:43 -0700: On Wed, Apr 19, 2017, at 05:54 AM, Julien Danjou wrote: Hoy, So Gnocchi gate is all broken (agan) because it depends on "pbr" and some new release of oslo.* depends on pbr!=2.1.0. Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 that got in banished by requirements Gods. It does not prevent it to be used e.g. to install the software or get version information. But it does break anything that is not in OpenStack because well, pip installs the latest pbr (2.1.0) and then oslo.* is unhappy about it. It actually breaks everything, including OpenStack. Shade and others are affected by this as well. The specific problem here is that PBR is a setup_requires which means it gets installed by easy_install before anything else. This means that the requirements restrictions are not applied to it (neither are the constraints). So you get latest PBR from easy_install then later when something checks the requirements (pkg_resources console script entrypoints?) they break because latest PBR isn't allowed. We need to stop pinning PBR and more generally stop pinning any setup_requires (there are a few more now since setuptools itself is starting to use that to list its deps rather than bundling them). So I understand the culprit is probably pip installation scheme, and we can blame him until we fix it. I'm also trying to push pbr 2.2.0 to avoid the entire issue. Yes, a new release of PBR undoing the "pin" is the current sane step forward for fixing this particular issue. Monty also suggested that we gate global requirements changes on requiring changes not pin any setup_requires. But for the future, could we stop updating the requirements in oslo libs for no good reason? just because some random OpenStack project hit a bug somewhere? For example, I've removed requirements update on tooz¹ for more than a year now, which did not break *anything* in the meantime, proving that this process is giving more problem than solutions. Oslo libs doing that automatic update introduce more pain for all consumers than anything (at least not in OpenStack). You are likely largely shielded by the constraints list here which is derivative of the global requirements list. Basically by using constraints you get distilled global requirements and even without being part of the requirements updates you'd be shielded from breakages when installed via something like devstack or other deployment method using constraints. So if we care about Oslo users outside OpenStack, I beg us to stop this crazyness. If we don't, we'll just spend time getting rid of Oslo over the long term… I think we know from experience that just stopping (eg reverting to the situation we had before requirements and constraints) would lead to sadness. Installations would frequently be impossible due to some unresolvable error in dependency resolution. Do you have some alternative in mind? Perhaps we loosen the in project requirements and explicitly state that constraints are known to work due to testing and users should use constraints? That would give users control to manage their own constraints list too if they wish. Maybe we do this in libraries while continuing to be more specific in applications? At the meeting in Austin, the requirements team accepted my proposal to stop syncing requirements updates into projects, as described in https://etherpad.openstack.org/p/ocata-requirements-notes We haven't been able to find anyone to work on the implementation, though. I is my understanding that Tony did contact the Telemetry and Swift teams, who are most interested in this area of change, about devoting some resources to the tasks outlined in the proposal. Doug My 2c, Cheers, Wasn't there also some decision made in austin (?) about how we as a group stated something along the line of co-installability isn't as important as it once was (and may not even be practical or what people care about anymore anyway)? With kolla becoming more popular (tripleo I think is using it, and ...) and the containers it creates making isolated per-application environments it makes me wonder what of global-requirements is still valid (as a concept) and what isn't. I do remember the days of free for all requirements (or requirements sometimes just put/stashed in devstack vs elsewhere), which I don't really want to go back to; but if we finally all agree that co-installability isn't what people actually do and/or care about (anymore?) then maybe we can re-think some things? I personally still like having an ability to know some set of requirements works for certain project X for a given release Z (as tested by the gate); though I am not really concerned about if the same set of requirements works for certain project Y (also in release Z). If this is something others agree with then perhaps we just need to store those requirements and the last
Re: [openstack-dev] [oslo] Can we stop global requirements update?
We have started this work. I've been working on: https://review.openstack.org/#/c/444718/ Which will do requirement checks, as specified in the Pike PTG ehterpad for Tuesday morning: https://etherpad.openstack.org/p/relmgt-stable-requirements-ptg-pike (line 40+). Once done, Tony and I were going to start testing it on the experimental pipeline for Swift and Nova. Regards, Matt On Thu, Apr 20, 2017 at 2:34 AM, Doug Hellmann wrote: > Excerpts from Clark Boylan's message of 2017-04-19 08:10:43 -0700: > > On Wed, Apr 19, 2017, at 05:54 AM, Julien Danjou wrote: > > > Hoy, > > > > > > So Gnocchi gate is all broken (agan) because it depends on "pbr" > and > > > some new release of oslo.* depends on pbr!=2.1.0. > > > > > > Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 > > > that got in banished by requirements Gods. It does not prevent it to be > > > used e.g. to install the software or get version information. But it > > > does break anything that is not in OpenStack because well, pip installs > > > the latest pbr (2.1.0) and then oslo.* is unhappy about it. > > > > It actually breaks everything, including OpenStack. Shade and others are > > affected by this as well. The specific problem here is that PBR is a > > setup_requires which means it gets installed by easy_install before > > anything else. This means that the requirements restrictions are not > > applied to it (neither are the constraints). So you get latest PBR from > > easy_install then later when something checks the requirements > > (pkg_resources console script entrypoints?) they break because latest > > PBR isn't allowed. > > > > We need to stop pinning PBR and more generally stop pinning any > > setup_requires (there are a few more now since setuptools itself is > > starting to use that to list its deps rather than bundling them). > > > > > So I understand the culprit is probably pip installation scheme, and we > > > can blame him until we fix it. I'm also trying to push pbr 2.2.0 to > > > avoid the entire issue. > > > > Yes, a new release of PBR undoing the "pin" is the current sane step > > forward for fixing this particular issue. Monty also suggested that we > > gate global requirements changes on requiring changes not pin any > > setup_requires. > > > > > But for the future, could we stop updating the requirements in oslo > libs > > > for no good reason? just because some random OpenStack project hit a > bug > > > somewhere? > > > > > > For example, I've removed requirements update on tooz¹ for more than a > > > year now, which did not break *anything* in the meantime, proving that > > > this process is giving more problem than solutions. Oslo libs doing > that > > > automatic update introduce more pain for all consumers than anything > (at > > > least not in OpenStack). > > > > You are likely largely shielded by the constraints list here which is > > derivative of the global requirements list. Basically by using > > constraints you get distilled global requirements and even without being > > part of the requirements updates you'd be shielded from breakages when > > installed via something like devstack or other deployment method using > > constraints. > > > > > So if we care about Oslo users outside OpenStack, I beg us to stop this > > > crazyness. If we don't, we'll just spend time getting rid of Oslo over > > > the long term… > > > > I think we know from experience that just stopping (eg reverting to the > > situation we had before requirements and constraints) would lead to > > sadness. Installations would frequently be impossible due to some > > unresolvable error in dependency resolution. Do you have some > > alternative in mind? Perhaps we loosen the in project requirements and > > explicitly state that constraints are known to work due to testing and > > users should use constraints? That would give users control to manage > > their own constraints list too if they wish. Maybe we do this in > > libraries while continuing to be more specific in applications? > > At the meeting in Austin, the requirements team accepted my proposal > to stop syncing requirements updates into projects, as described > in https://etherpad.openstack.org/p/ocata-requirements-notes > > We haven't been able to find anyone to work on the implementation, > though. I is my understanding that Tony did contact the Telemetry > and Swift teams, who are most interested in this area of change, > about devoting some resources to the tasks outlined in the proposal. > > Doug > > > > > > > > > My 2c, > > > > > > Cheers, > > > > > > ¹ Unless some API changed in a dep and we needed to raise the dep, > > > obviously. > > > > > > -- > > > Julien Danjou > > > # Free Software hacker > > > # https://julien.danjou.info > > > > I don't have all the answers, but am fairly certain the situation we > > have today is better than the one from several years ago. It is just not > > perfect. I think we are better served by refining the current setup or >
Re: [openstack-dev] [oslo] Can we stop global requirements update?
Excerpts from Clark Boylan's message of 2017-04-19 08:10:43 -0700: > On Wed, Apr 19, 2017, at 05:54 AM, Julien Danjou wrote: > > Hoy, > > > > So Gnocchi gate is all broken (agan) because it depends on "pbr" and > > some new release of oslo.* depends on pbr!=2.1.0. > > > > Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 > > that got in banished by requirements Gods. It does not prevent it to be > > used e.g. to install the software or get version information. But it > > does break anything that is not in OpenStack because well, pip installs > > the latest pbr (2.1.0) and then oslo.* is unhappy about it. > > It actually breaks everything, including OpenStack. Shade and others are > affected by this as well. The specific problem here is that PBR is a > setup_requires which means it gets installed by easy_install before > anything else. This means that the requirements restrictions are not > applied to it (neither are the constraints). So you get latest PBR from > easy_install then later when something checks the requirements > (pkg_resources console script entrypoints?) they break because latest > PBR isn't allowed. > > We need to stop pinning PBR and more generally stop pinning any > setup_requires (there are a few more now since setuptools itself is > starting to use that to list its deps rather than bundling them). > > > So I understand the culprit is probably pip installation scheme, and we > > can blame him until we fix it. I'm also trying to push pbr 2.2.0 to > > avoid the entire issue. > > Yes, a new release of PBR undoing the "pin" is the current sane step > forward for fixing this particular issue. Monty also suggested that we > gate global requirements changes on requiring changes not pin any > setup_requires. > > > But for the future, could we stop updating the requirements in oslo libs > > for no good reason? just because some random OpenStack project hit a bug > > somewhere? > > > > For example, I've removed requirements update on tooz¹ for more than a > > year now, which did not break *anything* in the meantime, proving that > > this process is giving more problem than solutions. Oslo libs doing that > > automatic update introduce more pain for all consumers than anything (at > > least not in OpenStack). > > You are likely largely shielded by the constraints list here which is > derivative of the global requirements list. Basically by using > constraints you get distilled global requirements and even without being > part of the requirements updates you'd be shielded from breakages when > installed via something like devstack or other deployment method using > constraints. > > > So if we care about Oslo users outside OpenStack, I beg us to stop this > > crazyness. If we don't, we'll just spend time getting rid of Oslo over > > the long term… > > I think we know from experience that just stopping (eg reverting to the > situation we had before requirements and constraints) would lead to > sadness. Installations would frequently be impossible due to some > unresolvable error in dependency resolution. Do you have some > alternative in mind? Perhaps we loosen the in project requirements and > explicitly state that constraints are known to work due to testing and > users should use constraints? That would give users control to manage > their own constraints list too if they wish. Maybe we do this in > libraries while continuing to be more specific in applications? At the meeting in Austin, the requirements team accepted my proposal to stop syncing requirements updates into projects, as described in https://etherpad.openstack.org/p/ocata-requirements-notes We haven't been able to find anyone to work on the implementation, though. I is my understanding that Tony did contact the Telemetry and Swift teams, who are most interested in this area of change, about devoting some resources to the tasks outlined in the proposal. Doug > > > > > My 2c, > > > > Cheers, > > > > ¹ Unless some API changed in a dep and we needed to raise the dep, > > obviously. > > > > -- > > Julien Danjou > > # Free Software hacker > > # https://julien.danjou.info > > I don't have all the answers, but am fairly certain the situation we > have today is better than the one from several years ago. It is just not > perfect. I think we are better served by refining the current setup or > replacing it with something better but not by reverting. > > Clark > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Wed, Apr 19 2017, Clark Boylan wrote: > I think we know from experience that just stopping (eg reverting to the > situation we had before requirements and constraints) would lead to > sadness. Installations would frequently be impossible due to some > unresolvable error in dependency resolution. Do you have some > alternative in mind? Perhaps we loosen the in project requirements and > explicitly state that constraints are known to work due to testing and > users should use constraints? That would give users control to manage > their own constraints list too if they wish. Maybe we do this in > libraries while continuing to be more specific in applications? Most of the problem that requirements is trying to solve is related to upper-constraints, blocking new releases. And this upper constraints are used in most jobs, preventing most failure that are seen in gates. It would have "covered" the pbr issue. What I want to stop here, is the automatic push of blacklisting/capping of stuff to *everything* in OpenStack as soon as one project have a problem with something. > I don't have all the answers, but am fairly certain the situation we > have today is better than the one from several years ago. It is just not > perfect. I think we are better served by refining the current setup or > replacing it with something better but not by reverting. Agreed, I'm not suggesting to revert everything. Just the automatic push of random requirements limits and binding to Oslo. And other projects if you like, we don't do it for a good year anymore in Telemetry, and again here we saw 0 breakage due to that change. Just more easyness to install stuff. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Wed, Apr 19, 2017, at 05:54 AM, Julien Danjou wrote: > Hoy, > > So Gnocchi gate is all broken (agan) because it depends on "pbr" and > some new release of oslo.* depends on pbr!=2.1.0. > > Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 > that got in banished by requirements Gods. It does not prevent it to be > used e.g. to install the software or get version information. But it > does break anything that is not in OpenStack because well, pip installs > the latest pbr (2.1.0) and then oslo.* is unhappy about it. It actually breaks everything, including OpenStack. Shade and others are affected by this as well. The specific problem here is that PBR is a setup_requires which means it gets installed by easy_install before anything else. This means that the requirements restrictions are not applied to it (neither are the constraints). So you get latest PBR from easy_install then later when something checks the requirements (pkg_resources console script entrypoints?) they break because latest PBR isn't allowed. We need to stop pinning PBR and more generally stop pinning any setup_requires (there are a few more now since setuptools itself is starting to use that to list its deps rather than bundling them). > So I understand the culprit is probably pip installation scheme, and we > can blame him until we fix it. I'm also trying to push pbr 2.2.0 to > avoid the entire issue. Yes, a new release of PBR undoing the "pin" is the current sane step forward for fixing this particular issue. Monty also suggested that we gate global requirements changes on requiring changes not pin any setup_requires. > But for the future, could we stop updating the requirements in oslo libs > for no good reason? just because some random OpenStack project hit a bug > somewhere? > > For example, I've removed requirements update on tooz¹ for more than a > year now, which did not break *anything* in the meantime, proving that > this process is giving more problem than solutions. Oslo libs doing that > automatic update introduce more pain for all consumers than anything (at > least not in OpenStack). You are likely largely shielded by the constraints list here which is derivative of the global requirements list. Basically by using constraints you get distilled global requirements and even without being part of the requirements updates you'd be shielded from breakages when installed via something like devstack or other deployment method using constraints. > So if we care about Oslo users outside OpenStack, I beg us to stop this > crazyness. If we don't, we'll just spend time getting rid of Oslo over > the long term… I think we know from experience that just stopping (eg reverting to the situation we had before requirements and constraints) would lead to sadness. Installations would frequently be impossible due to some unresolvable error in dependency resolution. Do you have some alternative in mind? Perhaps we loosen the in project requirements and explicitly state that constraints are known to work due to testing and users should use constraints? That would give users control to manage their own constraints list too if they wish. Maybe we do this in libraries while continuing to be more specific in applications? > > My 2c, > > Cheers, > > ¹ Unless some API changed in a dep and we needed to raise the dep, > obviously. > > -- > Julien Danjou > # Free Software hacker > # https://julien.danjou.info I don't have all the answers, but am fairly certain the situation we have today is better than the one from several years ago. It is just not perfect. I think we are better served by refining the current setup or replacing it with something better but not by reverting. Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] Can we stop global requirements update?
Hoy, So Gnocchi gate is all broken (agan) because it depends on "pbr" and some new release of oslo.* depends on pbr!=2.1.0. Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 that got in banished by requirements Gods. It does not prevent it to be used e.g. to install the software or get version information. But it does break anything that is not in OpenStack because well, pip installs the latest pbr (2.1.0) and then oslo.* is unhappy about it. So I understand the culprit is probably pip installation scheme, and we can blame him until we fix it. I'm also trying to push pbr 2.2.0 to avoid the entire issue. But for the future, could we stop updating the requirements in oslo libs for no good reason? just because some random OpenStack project hit a bug somewhere? For example, I've removed requirements update on tooz¹ for more than a year now, which did not break *anything* in the meantime, proving that this process is giving more problem than solutions. Oslo libs doing that automatic update introduce more pain for all consumers than anything (at least not in OpenStack). So if we care about Oslo users outside OpenStack, I beg us to stop this crazyness. If we don't, we'll just spend time getting rid of Oslo over the long term… My 2c, Cheers, ¹ Unless some API changed in a dep and we needed to raise the dep, obviously. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev