[Distutils] Setuptools and non-system installs
I was trying to install setuptools in my home directory on a system I don't have root access to using the command python setup.py install --home=~. Of course, since it installs as an egg, I can't use it unless I *specifically* add it to my PYTHONPATH. This would be a pain should it be necessary to add for every egg I install. I've added a workaround to my ~/lib/python/sitecustomize.py (~/lib/python is in PYTHONPATH) which uses the existing machinery in site.py to process .pth files in all of the entries of sys.path, not just the default sitedirs. I've noticed, however, that Guido has rejected a patch http://www.python.org/sf/1174614 to have the default site.py search .pth files in other directories, mainly because all the extra path searching at each startup might slow down startup on some systems. The best solution I can think of is to think of is to have some directory that is explicitly searched for .pth files (or even .egg/.zip files) and has them added to the path. This would still require some changes to sitecustomize.py (and hopefully eventually site.py), so it keeps eggs from working OOTB on this type of system. What do you think is the best way to have eggs loaded from non-system installs? -- Michael Hoffman [EMAIL PROTECTED] European Bioinformatics Institute ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools and non-system installs
At 03:39 PM 7/11/2005 +0100, Michael Hoffman wrote:
I was trying to install setuptools in my home directory on a system I
don't have root access to using the command python setup.py install
--home=~. Of course, since it installs as an egg, I can't use it
unless I *specifically* add it to my PYTHONPATH. This would be a pain
should it be necessary to add for every egg I install.
It isn't necessary. All you need to do is ensure that the setuptools egg
is on PYTHONPATH; everything else can then be handled by
EasyInstall-generated script wrappers or by manual use of require().
So, basically, if you have this non-root scenario, you have to put the
setuptools egg directly on your PYTHONPATH, and everything should be fine
after that.
I've added a workaround to my ~/lib/python/sitecustomize.py
(~/lib/python is in PYTHONPATH) which uses the existing machinery in
site.py to process .pth files in all of the entries of sys.path, not
just the default sitedirs.
Unfortunately EasyInstall can't know that you're doing this, so it's not
going to generate a .pth file in your ~/lib/python directory. Basically,
you need to learn to live with --multi-version mode, which has benefits as
well as drawbacks. When EasyInstall installs in multi-version mode, you
can't just fire up the interpreter and import the package, you have to
either call require() to ask for the package by name and version, *or* you
can run any script that was installed with the package and it'll work just
fine because EasyInstall creates a wrapper around the script that does the
require() for you.
If you're using packages that depend on each other, or creating your own
packages, you can create a wrapper package that's just a setup.py and
whatever programs you need to run. In the setup.py, you declare your
dependencies on the other packages, and then you run setup.py develop to
generate wrappers for your new scripts and set up your project as an
egg. The steps are detailed in the new setuptools Developers' Guide at:
http://peak.telecommunity.com/DevCenter/setuptools
In particular, pay close attention to the sections on Basic Use,
Declaring Dependencies, Development Mode, and on the develop command.
I've noticed, however, that Guido has rejected a patch
http://www.python.org/sf/1174614 to have the default site.py search
.pth files in other directories, mainly because all the extra path
searching at each startup might slow down startup on some systems.
The best solution I can think of is to think of is to have some
directory that is explicitly searched for .pth files (or even
.egg/.zip files) and has them added to the path. This would still
require some changes to sitecustomize.py (and hopefully eventually
site.py), so it keeps eggs from working OOTB on this type of system.
What do you think is the best way to have eggs loaded from non-system
installs?
I don't think that the patch makes much difference; there's nothing
stopping web hosters or others from adding a line like this to a .pth file
in site-packages now:
import os; addsitedir(os.path.expanduser('~/lib/python2.3')) # or 2.4
Which will do the same as the patch proposes to do for every line in each
.pth file. If you can get your service provider to put this in a .pth file
in site-packages, you're pretty much set, except for the fact that
EasyInstall will still think you need to do a multi-version
install. However, you won't need PYTHONPATH and you won't need to manage
it except by manipulating .pth files in your ~/lib/python2.3 or whatever
directory. And, I could probably add an option to EasyInstall (and
setuptools install/develop commands) to force .pth usage even if you're
installing somewhere other than site-packages.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] setuptools for people behind a firewall
I've just had a look at the new documentation for setuptools. I've not read it all in detail yet, but one thing struck me regarding the automatically download dependencies feature. It isn't going to work for people (like me) stuck behind a firewall that Python doesn't support (Windows NTLM based firewall). Obviously, setuptools is never going to be able to resolve a situation like this, nor would I expect it to. But can I suggest two possible changes to make it easier for people with limited internet access? 1. A manual download mode, where setuptools lists the files which it wants you to obtain, and then leaves it to you how you get them. I'm not sure how plausible this would be, given the necessarily iterative process involved in resolving dependencies, but even a little help would be useful (a report of unresolved dependencies when run with a --no-download flag would be the most basic help). 2. A way of specifying an external command to use to download files over HTTP. This would (for example) allow me to use curl, which does support HTLM proxies, rather than relying on Python's built-in HTTP support, which doesn't. Regards, Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools for people behind a firewall
One thing you could do is keep a bunch of eggs, .tar.gz's, exe's, whatever in a directory on a web server with directory indexes turned on and then add that page to the find_links options in you ~/.pydistutils.cfg file. Here's mine:: [easy_install] find_links=http://lesscode.org/eggs/ http://peak.telecommunity.com/dist/ Then, just evolve a process for placing things on the behind-the- firewall server and you should be good. Ryan On Jul 11, 2005, at 11:58 AM, Paul Moore wrote: I've just had a look at the new documentation for setuptools. I've not read it all in detail yet, but one thing struck me regarding the automatically download dependencies feature. It isn't going to work for people (like me) stuck behind a firewall that Python doesn't support (Windows NTLM based firewall). Obviously, setuptools is never going to be able to resolve a situation like this, nor would I expect it to. But can I suggest two possible changes to make it easier for people with limited internet access? 1. A manual download mode, where setuptools lists the files which it wants you to obtain, and then leaves it to you how you get them. I'm not sure how plausible this would be, given the necessarily iterative process involved in resolving dependencies, but even a little help would be useful (a report of unresolved dependencies when run with a --no-download flag would be the most basic help). 2. A way of specifying an external command to use to download files over HTTP. This would (for example) allow me to use curl, which does support HTLM proxies, rather than relying on Python's built-in HTTP support, which doesn't. Regards, Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Ryan Tomayko [EMAIL PROTECTED] http://naeblis.cx/rtomayko/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools for people behind a firewall
Paul Moore wrote: 2. A way of specifying an external command to use to download files over HTTP. This would (for example) allow me to use curl, which does support HTLM proxies, rather than relying on Python's built-in HTTP support, which doesn't. Someone mentioned curl to me in a ticket, but didn't mention specifically why it was better. I assume now that NTLM was the issue. They submitted this patch that used pycurl: http://pythonpaste.org/trac/attachment/ticket/10/build-pkg.diff I imagine that would be an easy change to setuptools as well. Though it would be nice if setuptools could detect the case where urllib failed due to an NTLM firewall, so it could suggest (in the error message) that the user install pycurl. I don't know how easy pycurl is to install -- I wonder in practice if command-line interaction with curl would be easier to get people to install (pycurl doesn't appear to have up-to-date windows installers). -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools for people behind a firewall
Ian Bicking [EMAIL PROTECTED] writes: I don't know how easy pycurl is to install -- I wonder in practice if command-line interaction with curl would be easier to get people to install (pycurl doesn't appear to have up-to-date windows installers). I was certainly just thinking of being able to supply a command line which, given a URL, would produce the file on stdout (in the case of curl, curl %s is the basic command, possibly with proxy-type options added). I don't see the need for using pycurl - and doing so would exclude use of other utilities like wget. Paul. -- The trouble with being punctual is that nobody's there to appreciate it. -- Franklin P. Jones ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools for people behind a firewall
Ryan Tomayko [EMAIL PROTECTED] writes: One thing you could do is keep a bunch of eggs, .tar.gz's, exe's, whatever in a directory on a web server with directory indexes turned on and then add that page to the find_links options in you ~/.pydistutils.cfg file. That's pretty neat. It's obvious in retrospect, but I never thought of it... Then, just evolve a process for placing things on the behind-the- firewall server and you should be good. Ah, there's the difficulty, of course :-) That's where a utility to report the dependencies would help... (It may be pretty trivial, but I don't see immediately how to do this - it seems to me that the current setuptools documentation is more for *creators* of eggs, than for *users* of them...) Paul. -- The trouble with being punctual is that nobody's there to appreciate it. -- Franklin P. Jones ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools for people behind a firewall
At 04:58 PM 7/11/2005 +0100, Paul Moore wrote: I've just had a look at the new documentation for setuptools. I've not read it all in detail yet, but one thing struck me regarding the automatically download dependencies feature. It isn't going to work for people (like me) stuck behind a firewall that Python doesn't support (Windows NTLM based firewall). Obviously, setuptools is never going to be able to resolve a situation like this, nor would I expect it to. Have you tried APS? (i.e., http://ntlmaps.sf.net/ ) Its pages seem to suggest it can authenticate to NTLM proxy servers like the one you're dealing with, and it sounds like a general-purpose solution to the proxying problem. The only issue is that you'd need to configure your system such that urllib considers the APS address to be the proxy to use, but then *all* Python apps (or any app that reads the same proxy configuration) will be able to get out past the firewall. But can I suggest two possible changes to make it easier for people with limited internet access? 1. A manual download mode, where setuptools lists the files which it wants you to obtain, and then leaves it to you how you get them. I'm not sure how plausible this would be, given the necessarily iterative process involved in resolving dependencies, but even a little help would be useful (a report of unresolved dependencies when run with a --no-download flag would be the most basic help). EasyInstall can't *find* the files without HTTP, since that's what it uses to talk to PyPI. So manual download mode would mean you'd effectively need to find all the files yourself! I'm not sure how much help EasyInstall could actually be in such a process. 2. A way of specifying an external command to use to download files over HTTP. This would (for example) allow me to use curl, which does support HTLM proxies, rather than relying on Python's built-in HTTP support, which doesn't. This is potentially possible, but EasyInstall's PyPI searches and SourceForge download support need to be able to detect the MIME type of an HTTP response in order to decide whether it has an HTML page or not. If APS doesn't work for you, this might be an option, it's just a tricky one to implement in ez_setup.py, which wants to be small and simple and do no command-line parsing of its own, because it's also a module that gets imported by setup scripts. OTOH, perhaps an environment variable would be the way to go there, if we have to. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] easy_install - some thoughts
At 10:32 PM 7/11/2005 +0100, Paul Moore wrote: 1. It would be much better if easy_install was usable via python -m, either as well as or instead of the existing easy_install.py script. I actually goofed in removing this; I'll put it back in the next release. 2. It would be useful if the -f option worked with a file: URL (or better still, a simple directory name), to handle local repositories. A good idea, which I've been thinking about. I'll add it in the next release, such that if an argument to -f is not a URL, it gets checked for being an existing directory. If it's a directory, I'll scan the contents as if they were links on a webpage. If it's not an existing directory, I'll print a warning (since errors should not pass silently) and proceed. 3. I still dislike the fact that I can't get a listing of installed eggs. I suggest writing a simple script using pkg_resources.find_distributions(), and perhaps adding some documentation and contributing it to setuptools. :) And uninstalling by deletion isn't good, IMHO. I am *strongly* averse to using raw file operations (delete) on files in Python\lib\site-packages. Elsewhere, fine, but I view the Python standard library as a managed area, and management tools should exist for it. That's the key benefit to me of bdist_wininst, and I don't see eggs providing it. So write a pkg_delete script and contribute it, too. I've got no objection to these other tools, they just haven't been my top priority and aren't likely to be for a while. I'd like to focus on the core engine/API, in such a way that other people can easily write these sorts of management tools around it. 4. --dry-run doesn't work as expected: \Apps\Python24\Scripts\easy_install.py --dry-run dist/alarm-1.0-py2.4-win32.egg Processing alarm-1.0-py2.4-win32.egg Copying alarm-1.0-py2.4-win32.egg to C:\Apps\Python24\Lib\site-packages zipimport.ZipImportError: not a Zip file Sadly, this is about what *I* expect, because most distutils commands can only get so far into a --dry-run before some step depends on the nonexistent output of the preceding step. I've been making the assumption that it's more useful to have a dry run that tells you most of what it does before dying, than to have no dry-run capability at all. It's possible that I could make later steps say, I'm in dry run mode and I don't know what the heck to do here, but I'm really not sure how I'd start. If you or someone else has patches, I'll be happy to look at incorporating them. Again, my focus for the next few releases is going to be on hardening pkg_resources to make it more robust and to have a more usable API. A nice to have feature would be the ability for easy_install to pick apart a bdist_wininst installer and build an egg from it. Then I guess it's a good thing I implemented it already. :) Try running EasyInstall on a win32.exe file, and you'll find that it actually does a pretty good job of turning it into an egg. If you were to run easy_install pysqlite, for example, (assuming you could get past your firewall) you'd notice that it finds and downloads the bdist_wininst installer for PySQLite, converts it to an egg, and then installs it. This feature is only obscurely referenced in the docs and changelog, so I don't really blame you for missing it. :) On a related note, had you thought of having EasyInstall put eggs in a specific subdirectory? Something like site-packages/eggs? As you're using .pth files, this wouldn't be any harder to manage, would it? Yes, it would, actually, especially for non-system installs, as Michael Hoffman was asking about here recently. The main reason for putting eggs in sys.path directories, however, is to support automatic discovery and adding them to sys.path at *runtime*, dynamically. I also wanted to take advantage of certain properties of sys.path, like the fact that Python's start script has its directory added to sys.path. This then makes eggs in the script directory easily discoverable. In fact, if you set an --install-dir with EasyInstall, and set the --always-copy flag, then your installation directory is an application directory. Scripts and eggs are both copied there, including all eggs needed to run the application. Now you can just ship that directory and it can be used anywhere you have Python and setuptools installed. Anyway, I think putting eggs in directories on sys.path is a natural extension of the sys.path concept, that seems to me to avoid the mental overhead of having a separate search path that works differently. I can see a use case for a combination egg. One which packages together a group of packages. For example, if I want a standard set of packages installed on all machines, so that users can assume they are present when writing one-off scripts. If I could package *existing* eggs up into a combination site-std.egg This is called a basket (as in putting all
[Distutils] setuptools 0.5a10 released with your bugs/features
* auto-import when requested resource's module isn't imported (Kevin Dangoor) * fix detection of baskets in sys.path directories (Paul Moore) * allow directories as arguments to --find-links (Paul Moore) * fix python -m easy_install not working (Paul Moore) Enjoy. (P.S. this also fixes some breakage w/buildutils 0.1.1 trying to import stuff from the 'easy_install' module. But note that buildutils doesn't seem to be smart enough about installing dependencies yet anyway, because trying to install pudge at runtime complains about it not being found in PyPI. I'm guessing that if Ryan's not going to put these modules up on PyPI, he needs to add some --find-links to the easy_install subcommand that buildutils runs to install a command's dependencies.) (P.P.S. That reminds me... Ryan, don't bother putting http://peak.telecommunity.com/dist/ in your --find-links list; now that I've got automatic PyPI uploading figured out, I plan to put future stable releases on PyPI, and snapshot releases (of projects other than setuptools itself) will go to http://peak.telecommunity.com/snapshots/ in the future.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
