On 08/05/2013 02:03 PM, Dean Troyer wrote: > [Moving a discussion from https://review.openstack.org/40019 to the ML > to get a wider audience] > > We've been around this block more than once so let's get it all > documented in one place and see where to go next. Skip down to > ############# for more actual discussion... > > Given: > * OpenStack now has an official list of Python package versions > required in https://review.openstack.org/p/openstack/requirements > * This list is rarely completely available in any packaged Linux distro > * Developers like new packages that fix their immediate problem > * Packagers dislike the treadmill of constantly upgrading packages for > many reasons (stability, effort, etc) > * Running OpenStack on certain long-term-stability distros (RHEL6) is > seriously a challenge due to the number of out-of-date components, > specifically here many of the python-* packages. > * Fedora and RHEL6 have a nasty configuration of telling pip to > install packages into the same location as RPM-installed packages > setting up hard-to-diagnose problems and irritating sysadmins > everywhere. FTR, Debian/Ubuntu configure pip to install into > /usr/local and put '/usr/local/lib/python2.7/dist-packages' ahead of > '/usr/lib/python2.7/dist-packages' in sys.path. > * Replacing setuptools on RHEL6 illustrated another issue: removing > python-setuptools took with it a number of other packages that > depended on it. > * OpenStack devs are not in the packaging business. This has been > decided [citation required?]. Fortunately those in the packaging > business do contribute to OpenStack (THANKS!) and do make a huge > effort to keep up with things like the Ububntu cloud Archive and Red > Hat's RDO. > > The last week or so of attempting to install Python prereqs from > requirements.txt and installing pip 1.4 to support that rather than > re-invent the wheel and all of the fallout from that illustrates > clearly that neither approach is going to solve our problem. > > Summary of the discussion in the review (paraphrased, read the review > to see where I got it wrong): > > * packages are evil: we develop and gate based on what is available in > requirements.txt and a non-zero number of those are only in PyPI > * Anvil solved this already: resolves requirements into the RPM > package list and packages anything required from PyPI > * ongoing worries about pip and apt/rpm overwriting each other as > additional things are installed > * packages are necessary: > > ############# > > My specific responses: > > * proposals to use a tool to automatically decide between package and > PyPI (harlowja, sdague): this works well on the surface, but anything > that does not take in to account the dependencies in these packages > going BOTH ways is going to fail. For example: on RHEL6 setuptools is > 0.6.10, we want 0.9.8 (the merged release w/distribute). Removing > python-setuptools will also remove python-nose, numpy and other > packages depending on what is installed. So fine, those can be > re-installed with pip. But a) we don't want to rebuild numpy (just > bear with me here), and b) the packaged python-nose 0.10.4 meets the > version requirement in requirements.txt so the package will be > re-installed, bringing with it python-setuptools 0.6.10 overwriting > the pip installation of 0.9.8. > * separate proposal to build meta-packages (iwienand, > https://review.openstack.org/39862): this would be an effective > work-around for the problems on Fedora and RHEL6 and protect against > the re-install problem, but this is solving a distro packaging problem > in an upstream testing tool. We would also need to manage these > individually and not in a single large package, or at least split > python-crypto and python-lxml out of the proposed package as they may > not be installed in all configurations. > * about-face on all-packages to all-PyPI (mtaylor): I'm still on the > use packages where possible side but DevStack specifically is not in > the packaging business. If it were we'd do what Java folk (*sorry*) > have long taken the approach for non-trivial apps to just include the > whole damn runtime. Write once duplicate everywhere. > > I want to propose moving forward with the following guidelines: > * Keep a list of specific distro packages that we want to use in favor > of PyPI for specific reasons like not recompiling native modules, i.e. > python-crypto except on RHEL, etc
Yes. I think this will be helpful. It's essentially my about-face patch, which doesn't remove the installation of packages - with a general consensus that there will be specific distro packages that are needed to be there. We're going to need to solve, particularly on RedHat, having a yum repo that someone maintains with yum packages of a couple of key things for at least a little bit. setuptools and pip are the two main ones right now. If we could have devstack on rh add a specific yum repo and yum install python-setuptools 0.9.8 and pip 1.4 early on, then I think most of our current major redhat issues would go away. I don't think we will gain much by auto-generating packages. (believe me, I REALLY REALLY want to have the opposite opinion on that, and you can go troll the -infra logs for all the times where I've gotten angry and suggested we just go back to making packages) But there is a giant rathole we can wind up in there, and we'd essentially be making our own micro-distro, and I don't think that's the business we're in. Similarly, I do not want to maintain additional repos as a general practice. Usually just pip installing the latest package works, or if it doesn't, waiting until the distro does a backport for us works. In some RARE cases (such as the current setuptools fiasco) the mechanics just do not allow us to deal with the situation in any other way. I think we need an answer for that, but I think it's VERY important that we are VERY careful with it. > * Continue to factor the prereq setup out of stack.sh such that > requirements.txt is satisfied one way or another before it begins to > install OpenStack. tools/install_prereqs.sh and tools/install_pip.sh > are the prototypes for this. Each distro gets a chance to get it right > here whether by package or by PyPI. > > tl;dr: no easy solution > > dt > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
