-----Original Message----- From: Thomas Goirand <z...@debian.org> Reply: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Date: April 18, 2016 at 17:21:36 To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [release][requirements][packaging][summit] input needed on summit discussion about global requirements
> Hi Doug, > > I very much welcome opening such a thread before the discussion at the > summit, as often, sessions are too short. Taking the time to write > things down first also helps having a more constructive discussion. > > Before I reply to each individual message below, let me attempt to reply > to the big picture seen in your etherpad. I was tempted to insert > comments on each lines of it, but I'm not sure how this would be > received, and probably it's best to attempt to reply more globally. > > From what I understand, the biggesgt problems you're trying to solve is > that managing the global-reqs is really time consuming from the release > team point of view, and especially its propagation to individual > projects. There's IMO many things that we could do to improve the > situation, which would be acceptable from the package maintainers point > of view. > > First of all, from what I could see in the etherpad, I see a lot of > release work which I consider not useful for anyone: not for downstream > distros, not upstream projects. Mostly, the propagation of the > global-requirements.txt to each and every individual Python library or > service *for OpenStack maintained libs* could be reviewed. Because 1/ > distros will always package the highest version available in > upper-constraints.txt, and 2/ it doesn't really reflect a reality. As > you pointed out, project A may need a new feature from lib X, but > project B wont care. I strongly believe that we should leave lower > boundaries as a responsibility of individual projects. What important > though, is to make sure that the highest version released does work, > because that's what we will effectively package. > > What we can then later on do, at the distribution level, is artificially > set the lower bounds of versions to whatever we've just uploaded for a > given release of OpenStack. In fact, I've been doing this a lot already. > For example, I uploaded Eventlet 0.17.4, and then 0.18.4. There was > never anything in the between. Therefore, doing a dependency like: > > Depends: python-eventlet (>= 0.18.3) > > makes no sense, and I always pushed: > > Depends: python-eventlet (>= 0.18.4) > > as this reflects the reality of distros. > > If we generalize this concept, then I could push the minimum version of > all oslo libs into every single package for a given version of OpenStack. > > What is a lot more annoying though, is for packages which I do not > control directly, and which are used by many other non-OpenStack > packages inside the distribution. For example, Django, SQLAlchemy or > jQuery, to only name a few. > > I have absolutely no problem upping the lower bounds for all of > OpenStack components aggressively. We don't have gate jobs for the lower > bounds of our requirements. If we decide that it becomes the norm, I can > generalize and push for doing this even more. For example, after pushing > the update of an oslo lib B version X, I could push such requirements > everywhere, which in fact, would be a good thing (as this would trigger > rebuilds and recheck of all unit tests). Though, all of this would > benefit from a lot of automation and checks. > > On your etherpad, you wrote: > > "During the lead-up to preparing the final releases, one of the tracking > tasks we have is to ensure all projects have synced their global > requirements updates. This is another area where we could reduce the > burden on the release team." > > Well, don't bother, this doesn't reflect a reality anyway (ie: maybe > service X can use an older version of oslo.utils), so that's not really > helpful in any way. > > You also wrote: > > "Current ranges in global-requirements are large but most projects do > not actively test the oldest supported version (or other versions in > between) meaning that the requirement might result in broken packages." > > Yeah, that's truth, I've seen this and reported a few bugs (the last I > have in memory is Neutron requiring SQLA >= 1.0.12). Though that's still > very useful hints for package maintainers *for 3rd party libs* (as I > wrote, it's less important for OpenStack components). We have a few > breakage here and there, but they are hopefully fixes. > > Though having a single version that projects are allowed to test with is > super important, so we can check everything can work together. IMO, > that's the part which should absolutely not change. Dropping that is > like opening a Pandora box. Relying on containers and/or venv will > unfortunately, not work, from my stand point. > > The general rule for a distribution is that the highest version always > win, otherwise, it's never maintainable (for security and bug fixes). It > should be the case for *any program*, not even just any OpenStack > components. There's never a case where it's ok to use something older, > just because it feels like less work to do. This type of "laziness" > leads to very dangerous outcomes, always. > > Though I don't see any issue with a project willing to keep backward > compatibility with a lower version than what other project do. In fact, > that's highly desirable to always try to remain compatible with lower > versions as much as possible, *if* it doesn't bring too much burden > (think: we still have loads of Python 2.6 stuff like discover, argparse > and such that needs to be cleaned-up). > > For the projects who aren't following the release cycles, it's ok if > they bump their lower bounds at the condition that they keep what's in > the upper-constraints.txt, so that we can release them together with the > rest of OpenStack. For example, it's ok to require the latest oslo.utils > 3.8.0 for Ironic 5.1.0. > > I also see no reason to completely freeze versions to the point were it > is impossible fix bugs in the versionnings. I've seen that multiple > times. At the distro level, we do address the issue anyway, so why > bother? The only thing that counts for us, is to be able to release > everything together. A good example would be that Neutron needing SQLA > >= 1.0.12: we refuse to fix it, but the issue is still there, and we > have to fix it in distros. > > Last, before I reply to each individual message: the attempt from Robert > to test on lower bounds is really something that should be pushed > forward. I would help a lot to actually know what the reality is, rather > than doing second-guess and artificially set lower bounds for versions > which we think may probably work. > > On 04/18/2016 02:22 PM, Chris Dent wrote: > > targeting an > > all-in-one install seems only to serve the purposes of all-in-one rpm- > > or deb-based installations. > > Though that's what most everyone consumes. Or are you proposing that we > completely stop publishing OpenStack in distros? > > Remember that in distros, there's only a single version of a library at > any given time, at the exception of transitions (yes, in Red Hat it's > technically possible to install multiple versions, but the policy > strongly advocates against this). > > Also, all-in-one is what I use in Debian in my CI, to make sure that > everyone works together, with whatever is uploaded (see the > openstack-deploy package in Debian). > > > Many (most?) people won't be doing those kinds of installations. > > Most users are consuming packages from distributions. Also, if you're > using containers, probably you will also prefer using these packages to > build your containers: that's the most easy, safe and fast way to build > your containers. > > > If all-in-one installations are important to the rpm- and deb- based > > distributions then _they_ should be resolving the dependency issues > > local to their own infrastructure > > Who is "they"? Well, approximately 1 person full time for Debian, 1 for > Ubuntu if you combine Corey and David (and Debian and Ubuntu > dependencies are worked on together in Debian), so that's 2 full time > for Ubuntu/Debian. Maybe 2 and a half for RDO if what Haikel told me is > still accurate. > > So "we" wont handle it, even if "we" care, because we're already > understaffed. > > > (or realizing that it is too painful and start > > containerizing or otherwise as well). > > Distributions don't package containers, we offer packages that you can > use to build them. The way you deploy is IMO orthogonal to packaging. > > > I think making these changes will help to improve and strengthen the > > boundaries and contracts between services. If not technically then > > at least socially, in the sense that the negotiations that people > > make to get things to work are about what actually matters in their > > services, not unwinding python dependencies and the like. > > Yes, it simplifies the life of developers. But it's going to be hell on > earth for production if we discover a serious vulnerability in a > component, which will be deployed N times, with N versions. Are we then > going to provide N patches? The vulnerability management team is also > understaffed... > > > One challenge I've run into the past is when devstack > > plugin A has made an assumption about having access to a python > > script provided by devstack plugin B, but it's not on $PATH or its > > dependencies are not in the site-packages visible to the current > > context. The solution here is to use full paths _into_ virtenvs. > > Please take a step back. Devstack and virtualenv are for development, > not for production. That's not, and should not become, our target. > Applying the reasoning you have there will *not* work for distributions. > Hopefully, that's what Doug is proposing, as he would still like things > to work for distros (right?). > > On 04/18/2016 03:24 PM, Hayes, Graham wrote: > > It is also worth noting that according to the OpenStack User Survey > > [0] 56% of deployments use "Unmodifed packages from the operating > > system". > > It is my believe (and hope, as this would mean more users) that some > OpenStack users don't even know about this survey, and just install > what's in the distros. So 56% is the lowest estimate. Also, "Unmodifed > packages" is the answer. Maybe some are deploying "slightly modified > packages, but mostly what's upstream", which would bring the score even > higher. > > On 04/18/2016 07:33 PM, Doug Hellmann wrote: > > Can you truly not imagine any other useful way to package OpenStack > > other than individual packages with shared dependencies that would > > be acceptable? > > I could imagine that, but that's not at all what OpenStack is currently. > Currently, each and every service is using Oslo, clients, and so on, > plus the glue libs like os-brick, Castellan, you name it. > > Or did I misunderstood your sentence above? > > On 04/18/2016 07:49 PM, Sean Dague wrote: > > A lot of distros specifically have policies against this kind of > > bundling as well, because of security issues like this (which was so > > very bad) - http://www.zlib.net/advisory-2002-03-11.txt > > Gentoo is probably a special case (because building from source and so > on), but I'd say that *ALL* distro do have such policy in place (at > least, Red Hat, Debian, Ubuntu, SuSE...). > > On 04/18/2016 08:30 PM, Doug Hellmann wrote: > > Our upstream > > solution can be pretty light-weight, and that leaves room for > > downstream folks to make different choices. > > I'm not sure what choice we'd have, but to face loads of issues and > finally give-up packaging OpenStack if we have python module versions > that are conflicting. Hopefully, your proposal wont allow that? > > On 04/18/2016 08:40 PM, Doug Hellmann wrote: > > Some folks seem to be conflating "upstream wants to stop worrying > > about co-installability" with "upstream wants us to bundle > > dependencies". That's not what I'm proposing. > > Well, if we need multiple versions of the same Python module, what is > the solution in production then? > > > Another alternate that might work is for downstream folks to do > > their own dependency management. > > I fail to understand what you think distros could do to fix the > situation of conflicting versions, other than bundling, which isn't > acceptable. Could you explain what you mean by "downstream folks to do > their own dependency management"? > > > That's less optimal, and may not > > change the outcome for Gentoo or Debian, where the downstream teams > > are small (1 person each, I think?). > > You can also add "Ubuntu" in the list here, as absolutely all OpenStack > dependencies are maintained mostly by me, within Debian, and then later "absolutely all" of OpenStack's dependencies are not maintained by you in Debian. A significant number are maintained by the DPMT (Debian Python Modules Team). The large majority are maintained by you, but not "absolutely all". > synced to Ubuntu, with a bit of help from Ubuntu folks (but they are > mostly busy with Canonical own products, and I'm the person doing most > of the work, still). This is slowly shifting to a more spread workload, > but unfortunately, we're not there yet. > > As for Red Hat / RDO, Haikel told me they are "2 and a half", so > basically, the situation there (unless it changed) is approximately the > same amount of burnout people. > > > I started the discussion to solicit ideas, but I would prefer to > > avoid proposing what downstream should do because (a) I'm sure > > folks want options and (b) I'm sure I'm not the person to identify > > those options. > > Well, I'm doing the packaging, and I'm not sure if we have any option > either, if 2 python dependencies are conflicting. > > On 04/18/2016 09:10 PM, Jeremy Stanley wrote: > > On 2016-04-18 13:58:03 -0500 (-0500), Matthew Thode wrote: > >> Ya, I'd be happy to work more with upstream. I already review the > >> stable-reqs updates and watch them for the stable branches I package > >> for. Not sure what else is needed. > > > > Reviewing the master branch openstack/requirements repository > > changes (to make sure deps being added are going to be sane things > > for someone in your distro to maintain packages of in the long term) > > would also make sense. > > I've done some of that, and used to be reactive on it. But mostly, the > way it's been done so far is very good (thanks to a lot of folks working > on it), and it's taking too much of my time to review them. So I just > pick it up before packaging a (beta-)version of OpenStack. > > > To be clear, "Moving the burden to packagers" is not the only option > > available to us. I've proposed one option for eliminating the issue, > > which has some benefits for us upstream but obviously introduces > > some other issues we would need to resolve. Another option is for > > more people to get involved in managing the dependency list. Some > > (most? all?) of those new people may come from distros, and sharing > > the effort among them would make it easier than each of them doing > > all of the work individually. Sort of like an open source project. > > There's 2 options I see to solve the "people reviewing the > global-requirements.txt changes aren't enough". > > 1/ talk to my boss and the folks from Canonical (I can't tell for the > RPM world) to allocate more staff for packaging (yes, I'm serious here! > I can get you in touch...). > > 2/ get a lot of dependencies out. Yes, let's really do that! We're > constantly adding more, I've seen only very few going away. Currently > the global-requirements.txt for stable/mitaka is 377 lines long. If you > grep out "client", "oslo" and "os-", "openstack", comments, empty lines, > and a few others, there's about 260 dependencies left. That's insane at > multiple levels. When are we going to see a movement to clean this mess, > and define real OpenStack standards? > > Cheers, > > Thomas Goirand (zigo) > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Ian Cordasco __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev