Hi all, Not sure which mailing list to send this too, but Plone product developers can run into this, so this looks like a fine list. :-)
Main point of the mail is to say: Hey, be aware that even if you *think* you have everything pinned in your buildout config, you might still be using unpinned eggs. And using unpinned eggs means that your buildout that runs fine today might give problems half a year later. This is similar to a problem in the plonenext buildout and the plone.recipe.zope2instance 3.8 package discussed in plone-dev this week. (3.9 has been released, solving that problem). Observe the following small buildout.cfg. It does not do anything useful, except that it showcases a possible problem: ======================================================================== [buildout] extensions = buildout-versions parts = zope2 test versions = versions # Next line is important! allow-picked-versions = false [zope2] recipe = plone.recipe.zope2install url = http://www.zope.org/Products/Zope/2.10.11/Zope-2.10.11-final.tgz fake-zope-eggs = true [test] recipe = zc.recipe.testrunner # Random egg as example eggs = pep8 [versions] buildout-versions = 1.2 distribute = 0.6.14 plone.recipe.zope2install = 3.2 zc.buildout = 1.4.3 pep8 = 0.5.0 zc.recipe.egg = 1.2.2 zc.recipe.testrunner = 1.3.0 zope.testrunner = 4.0.0b5 ======================================================================== Put that in a directory with a bootstrap.py and start the buildout process. $ python2.4 bootstrap.py ... $ bin/buildout Upgraded: zc.buildout version 1.4.3, distribute version 0.6.14; restarting. Generated script '/Users/mauritsvanrees/tmp/pin/bin/buildout'. While: Installing. Getting section test. Initializing section test. Installing recipe zc.recipe.testrunner. Getting distribution for 'zope.interface'. Error: Picked: zope.interface = 3.6.1 zc.recipe.testrunner depends on zope.testrunner, which depends on zope.interface and zope.exceptions. Since these packages have no version pin in the buildout and we have specified allow-picked-versions = false, buildout quits with an error. But, those two dependencies are in the Zope2 tarball and will be available as fake eggs. Can't buildout use those fake eggs? Yes, it can, if we set allow-picked-versions = true (or comment that line out). Rerun buildout and it works: $ bin/buildout Installing zope2. Creating fake eggs Installing test. Generated script '/Users/mauritsvanrees/tmp/pin/bin/test'. Versions had to be automatically picked. The following part definition lists the versions picked: [versions] # Required by: # zope.testrunner==4.0.0b5 zope.exceptions = 3.6.1 # Required by: # zope.testrunner==4.0.0b5 zope.interface = 3.6.1 Those last lines are output by buildout-versions (variant of buildout.dumppickedversions). It correctly reports that the two mentioned eggs have been picked. But... they are not used in the generated bin-script: $ grep zope.interface bin/* bin/test: '/Users/mauritsvanrees/tmp/pin/fake-eggs/zope.interface', $ grep zope.exceptions bin/* bin/test: '/Users/mauritsvanrees/tmp/pin/fake-eggs/zope.exceptions', Now we could add the suggested versions to the [versions] section, set allow-picked-eggs = false again, and rerun the buildout. Result: $ grep zope.interface bin/* bin/test: '.../zope.interface-3.6.1-py2.4-macosx-10.6-i386.egg', $ grep zope.exceptions bin/* bin/test: '.../zope.exceptions-3.6.1-py2.4.egg', OR: do not add those versions, but do set allow-picked-versions = false so you have the exact same buildout.cfg as in the beginning and this time bin/buildout 'magically' succeeds. It uses the fake eggs in the script and buildout-versions correctly does not report missing pins. So: the same buildout.cfg crashes at first, and later succeeds. This is because the second time around the fake eggs are already there and the buildout process makes use of them. What is the potential problem here? If you want to use the same buildout.cfg on a machine without an intermediate edit, you have some options: 1. Set allow-picked-versions = true. During the initialization phase of buildout you *do* use the latest zope.interface and zope.exceptions from pypi (which might have a bug, or for example might only work with python2.6) but at least they do not actually get used in any of the generated scripts. So you get the zope.interface and zope.exceptions from the Zope2 tarball, which is good. 2. Keep allow-picked-versions = false. Add the two version pins. Now you know exactly which eggs you use without being surprised by sudden incompatible releases. But in the resulting scripts you are not using the zope.interface and zope.exceptions form the original Zope2 tarball. So subtle things might go wrong. BTW, it might be better to use some older eggs instead. 3. Avoid using any recipes that have dependencies on fake Zope 2 eggs. Neither of the solutions is ideal I would say. With Plone 4 we use Zope 2.12 which is fully 'eggified', so no fake eggs are needed anymore, so this problem goes away then. For Plone 3 (or earlier) I think we have to live with this problem and everyone will have to choose one of the partial solutions and be smart enough to be able to handle possible future problems. Or does someone see a better, real solution? -- Maurits van Rees Programmer, Zest Software _______________________________________________ Product-Developers mailing list [email protected] http://lists.plone.org/mailman/listinfo/product-developers
