At 03:20 PM 8/22/2008 -0400, David Turner wrote:
On Wed, 2008-08-20 at 18:39 -0400, Phillip J. Eby wrote:
> At 04:40 PM 8/20/2008 -0400, David Turner wrote:
> >Attached is a patch which adds download caching to setuptools.
> >
> >At TOPP (http://topp.openplans.org/), we use a system called fassembler
> >to build our opencore stack. It creates approximately a dozen
> >virtualenvs, each with their own lib/python, and then uses setuptools to
> >install lots of libraries. Some of these libraries are common among
> >multiple apps, but we install multiple copies for ease of development.
> >And every time we rebuild, we start the whole process over again. The
> >major slowdown in building is downloading a bunch of things which
> >probably haven't changed since last time we downloaded them. This
> >patch will let us maintain a cache of all downloads, and thus do builds
> >much faster.
>
> The process I'd suggest for this use case is to build the external
> libraries using:
>
> easy_install -f cache_dir -zmaxd cache_dir lib1 lib2 ...
>
> This command will NOT go to the web for new versions of libraries,
> unless you also use -U. But it will ensure that the specified
> libraries have suitable eggs in cache_dir.
>
> Then, to install a given set of libraries to a virtualenv, use:
>
> easy_install -f cache_dir lib1 lib2 ...
>
> Or, if you really insist on multiple copies of the eggs (instead of
> just linking to them), use:
>
> easy_install -af cache_dir lib1 lib2 ...
>
> (which will copy the .egg files even if they could be used in place).
>
> Unlike your caching proposal, this approach gives you finer control
> over which libraries to update, when. You can also update the cache
> without changing what's installed in a given virtualenv.
I'm having a very hard time getting it working, actually. I should note
that I'm not using easy_install directly, but through setup.py. This is
because I don't want to have to list all my dependencies twice, and
setup.py passes options on to easy_install, as I understand it.
You don't have to specify dependencies twice. Just do:
easy_install -f cache_dir -zmaxd cache_dir path_to_checkout
It will then build an egg from the checkout and copy it and all the
dependencies to the cache dir.
Here's a simple test case:
1. Create a virtualenv:
$ virtualenv.py /tmp/testve
2. Activate
$ cd /tmp/testve
$ . bin/activate
3. Check out Cabochon.
(testve)$ svn co https://svn.openplans.org/svn/Cabochon/trunk cabochon
4. Try to set up
(testve)$ mkdir /tmp/ec2 # the cache directory
Here, you should run easy_install -f /tmp/ec2 -zmaxd /tmp/ec2 cabochon
(testve)$ cd cabochon
(testve)$ python setup.py develop -f /tmp/ec2 -zmaxd /tmp/ec2
Then here, run:
python setup.py develop -af /tmp/ec2
This will then copy any dependency eggs from the cache dir to the
virtualenv, and set up the checkout for development.
running develop
Checking .pth file support in /tmp/ec2
/tmp/testve/bin/python -E -c pass
running egg_info
creating Cabochon.egg-info
writing requirements to Cabochon.egg-info/requires.txt
writing Cabochon.egg-info/PKG-INFO
writing top-level names to Cabochon.egg-info/top_level.txt
writing dependency_links to Cabochon.egg-info/dependency_links.txt
writing entry points to Cabochon.egg-info/entry_points.txt
writing manifest file 'Cabochon.egg-info/SOURCES.txt'
writing manifest file 'Cabochon.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/ec2/Cabochon.egg-link (link to .)
Installed /tmp/testve/cabochon
Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:
pkg_resources.require("Cabochon") # latest installed version
pkg_resources.require("Cabochon==0.2dev-r19871") # this exact
version
pkg_resources.require("Cabochon>=0.2dev-r19871") # this version or
higher
Note also that the installation directory must be on sys.path at runtime
for
this to work. (e.g. by being the application's script directory, by
being on
PYTHONPATH, or by being added to sys.path by your code.)
[many more lines of this as it installs all the requirements]
5. Try to run
(testve)$ paster
bash: paster: command not found
6. Hm, that's no good. Well, what if we just manually try to see if
stuff is installed:
(testve)$ python
Python 2.5.2 (r252:60911, Jul 31 2008, 17:31:22)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cabochon
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "cabochon/__init__.py", line 8, in <module>
from cabochon.config.middleware import make_app
File "cabochon/config/middleware.py", line 2, in <module>
from paste import httpexceptions
ImportError: No module named paste
Nope.
This didn't work because you only did half of what I said; you have
to do the -zmaxd step to load or update the cache, and the -af step
to actually install your target to the virtualenv.
_______________________________________________
Distutils-SIG maillist - [email protected]
http://mail.python.org/mailman/listinfo/distutils-sig
