Ian Bicking wrote:
Jim Fulton wrote:
I actually tried to do this once before with zc.buildout, but I didn't
get far -- probably a result of lack of effort and lack of familiarity
with the overall stack. But I also recognize lots of the questions
about stuff like the zope.conf file and Data.fs that still seem
Certainly when you tried this, buildout was very young and we hadn't
written recipes to deal with these issues. We've made a lot of
progress since then.
Well, the last time I really used it was early December, and it still
felt slow and awkward to me at the time, with several funny quirks.
Hm, It's a bit hard to respond to "awkward" and "quirks". I'll
respond to performance issues a bit below.
And frankly I like easy_install. It's
probably 10x faster than buildout.
I doubt that that is true now. Although that probably depends on what
you are doing. Early versions of buildout did a lot of things
inefficiently as I was still learning setuptools. Because of the way
that buildout caches index information, I expect that creating a
buildout from scratch that used a lot of eggs would be much faster
than using easy_install. One difference though is that buildout
checks for the most recent compatible versions of all of the eggs it's
using every time you run it, whereas, as I understand it, with
workingenv, you'd just run easy_install manually when you want a new
Correct. The basic process with workingenv is:
1. Set it up.
2. Start installing stuff.
3. Try running stuff.
4. Realize you got it wrong, missed something, want to do more
development, return to 2.
I actually find myself doing the 2-4 loop pretty often, both in
development and when first deploying something. Just the amount of time
to do "bin/buildout -h" was substantial (though I don't really
understand why, except that buildout seemed to be working way too hard
to update itself).
Ah yes. This is a good point. By default, buildout checks for newer versions
of distributions for which there are open-ended requirements. This can take
frustratingly long -- especially because pypi is so darn slow.
One advantage of buildout over easy_install (and I assume workingenv)
is that the eggs you get are deterministic by default. They are always
the newest versions that satisfy your requirements. With easy_install,
you get the most recently installed eggs that satisfy your requirements.
This means that the eggs you have depend a lot on when you installed them.
To achieve this, buildout looks for newer distributions when a requirement
doesn't have an upper bound or when the upper bound isn't satisfied by an
I really should add a quick mode that skips looking for newer versions
of requirements are met by what's already installed. This would make
the iterative style you describe go much faster. I would certainly
appreciate this myself. I will do this soon.
You can bypass the checks by running in offline mode. Then buildout
runs very fast. Because of the ability to share eggs accross
buildouts, it is often possible to run a buidout using lots of eggs in
It has been suggested that there should be a mode for buildout that
only talks to the network when there isn't a local egg that satisfied
a requirement. This would make buildout work more like workingenv
when few if any eggs are actually needed.
Yes; more like easy_install does as well, actually. Though the way
easy_install works is hardly intuitive; I find myself frequently saying
"yes, you installed it, but did you -U install it?"
In particular, upgrading a distribution doesn't upgrade it's dependencies.
This makes it harder to control which distributions are used in an environment.
With easy_install, even through distributions are automatically included
by virtue of being dependencies, they aren't automatically updated.
There's no way to say "I want the most recent version of everything".
I wanted to make it easier to get the most recent version of the distributions
used, which is why buildout has a different policy for looking up distributions.
Plus buildout's desire to own everything and
destroy everything it does not own ;)
I'm not aware that it destroys anything. Could you be more specific?
Well, it owns parts, and the recipes control that. Doesn't it also
delete and reinstall there?
Yes. Buildout tried to make a buildout reflect it's specification.
This is an important feature. It uninstalls as well as installs.
But it isn't controlling anything it wasn't asked to control.
> How it treats each area of the buildout I'm
I can't help that. I've documented how this works in great detail.
> Simply making the file layout a bit more conventional, and
describing anything non-obvious, would make buildout feel a lot more
comfortable to the new user.
What is conventional? Python uses different layouts on different systems.
The Unix layout and Windows layout are quite different. When I came up
with the layout for Zope installations, I tried to mimic the layout
that Python used on Unix systems at the time, and then that layout
changed. We were stuck with lib/python even though we never had
anything else in lib.
I chose a shallow layout in buildout following "flat is better
* As a result buildout supports multiple things in the same buildout
that have conflicting version requirements, but where the packages
themselves don't realize this (but the deployer does). If the packages
know their requirements then setuptools' native machinery allows things
to work fine.
Yes. I expect that usually, packages won't be very specific. The
buildout configuration file provides a place to be specific.
workingenv allows this, insofar as you can be specific while installing
things, and with the requirements file. But it doesn't make the
individual scripts very specific, if for instance appfoo requires
libX>1.0, and appbar requires libX>1.1, but you actually want appfoo to
use libX==1.0 and appbar to use libX==1.1 and install them in the same
buildout. That's the only case where buildout seems to be able to
express something workingenv can't.
In practice, this can be very important. At least for us at ZC.
* Some see bin/activate as a jail. Both workingenv and buildout are
deliberately jail-like. Both Jim and I loathe the non-repeatability of
system-wide installations (at least I think I can speak for him on that
one point ;). bin/activate lets you into that jail, and lets you work
there. There is no way into a buildout.
I'm not familiar with bin/activate, but it sounds like an interpreter
script created with buildout.
It's created by workingenv, and you have to source it because basically
its only function is to add the workingenv/lib/pythonX.Y to $PYTHONPATH.
Adding that path to $PYTHONPATH is the only thing that really
"activates" a workingenv.
Ah, so it modifies the user's environment. I think that's a reasonable
approach, although not one that I care for myself. To each his own.
The important things here, IMO, is that both activate and buildout interpreter
scripts let you get a Python interactive session or run scripts with a
Neither can be entirely
compatible with a system-wide Python installation, because Python's
standard site.py f**ks up the environment really early in the process,
and avoiding that isn't all that easy.
This reminds me of a place where buildout is looser than workenv.
buildout doesn;t try to disable anything in the system python. It
just augments it. I always use a clean python, so avoiding
customizations in the Python I use isn't a problem. If I wanted to
take advantage of something in a system Python, as I occasionally do,
I can do that with buildout.
I find the isolation useful when testing things for release;
Yup. I do the same things by always using clean Python installs.
I *never* use a system Python for development. I always use a
Python that I build myself from sources.
Your approach is certainly a valid alternative.
I can be
sure that I haven't been using any packages that I don't explicitly
include in the egg requirements or instructions.
But it can be annoying
in other cases, like when there's a library that doesn't install cleanly
(of which there's still quite a few).
Anyway, if you do want to include
the global packages, --site-packages will change your workingenv to do so.
It could be argued that workingenv's default should be to include
site-packages. Another option would be to have a tool that allows you
to easily include something from the system Python (probably just a tool
to manage a custom .pth file, which works even when setuptools' fairly
heroic attempts to fix broken setup.py's doesn't work).
<shrug> My remark was not meant to criticize or to suggest a different
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
Zope-Dev maillist - Zope-Dev@zope.org
** No cross posts or HTML encoding! **
(Related lists -