Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 21/03/2008, Terry Reedy [EMAIL PROTECTED] wrote: However, this Windows user, and I expect most, do NOT expect add-ons (things under the /Pythonx.y tree) to show up in the add/remove list. That's an interesting counterpoint to my comments. I presume from this that you dislike (and/or never use) bdist_msi and bdist_wininst precisely because they do add such items to the add/remove programs list? My argument is essentially that bdist_wininst set a de facto standard for this. Then, bdist_msi followed it. Now setuptools is trying to change that standard by ignoring it, rather than by discussion of the pros and cons. Personally, I like the current approach, but that's less relevant. The standard (and to me, preferable) way of dealing with such things is to have an 'installation manager' that can reinstall as well as delete and that has a check box for various things to delete. This is what Python needs. I'd dispute strongly that this is a standard. It may be preferable, but I'm not sure where you see evidence of it being a standard. Could I also point out that *if* such a standard is set up for Python, bdist_wininst and bdist_msi should be modified to follow it. Otherwise, it's not a standard, more of competing approach. As you can see, my main concern is for consistency :-) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:33 PM 3/21/2008 +, Paul Moore wrote: On 21/03/2008, Terry Reedy [EMAIL PROTECTED] wrote: The standard (and to me, preferable) way of dealing with such things is to have an 'installation manager' that can reinstall as well as delete and that has a check box for various things to delete. This is what Python needs. I'd dispute strongly that this is a standard. It may be preferable, but I'm not sure where you see evidence of it being a standard. I presume he means that there are a lot of entries in his Add/Remove Programs that work like that, and that it's an emerging standard for Windows. (Certainly I've seen quite a few entries like that in mine, although more often than not they only have one checkbox!) Could I also point out that *if* such a standard is set up for Python, bdist_wininst and bdist_msi should be modified to follow it. Otherwise, it's not a standard, more of competing approach. The best thing to do would be to get a standard (ala PEP 262, but modified by the benefit of experience now) for tracking installed Python package distributions. Then we can standardize on platform tools for managing this data, and include them in the relevant platform distributions. (And that would include making bdist_wininst and bdist_msi follow this installation DB standard.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
-On [20080320 19:24], Steve Holden ([EMAIL PROTECTED]) wrote: We need to stop protesting that our installation tools are easy enough and try to get behind the various platforms, be it with Windows installers, rpms, or other support. We probably aren't doing this because it's work nobody particularly relishes, and has relatively low visibility in the developer world. Non-developer Python programmers and end-users would thank us, though. FreeBSD offers through install of Perl through its ports system a Perl module called 'bsdpan' which registers every module as a package under FreeBSD's package system. Normally ports installs modules as p5-ModuleName, but now it becomes: /var/db/pkg/bsdpan-B-Lint-1.09 And from that point on I can use the pkg* tools. Quite elegant in my opinion. -- Jeroen Ruigrok van der Werven asmodai(-at-)in-nomine.org / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ To regard the fundamental as the essence, to regard things as coarse, to regard accumulation as deficiency, and to dwell quietly alone with the spiritual and the intelligent -- herein lie the techniques of Tao of the ancients. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mar 20, 2008, at 6:22 PM, Robert Brewer wrote: Phillip J. Eby wrote: The other tool that would be handy to have, would be one that unpacks eggs into standard distutils-style installation. Hear, hear. I'm an author of a couple libraries that need to interoperate with others. Of the many eggs I've downloaded over the past year, I'd say 80%+ are never installed or even built--I just want to grep the source code, and using my preferred tools, not some lame Find command in a ZIP browser menu. Um, isn't this tool called unzip? I have done this -- accessed the source code -- many times, and unzip suffices. I don't know what else would be required in order to make an egg into a standard distutils-style installation. Until PJE's comment above, I thought that unzip already accomplished exactly that. Regards, Zooko ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:53 AM 3/21/2008 -0600, zooko wrote: Um, isn't this tool called unzip? I have done this -- accessed the source code -- many times, and unzip suffices. I don't know what else would be required in order to make an egg into a standard distutils-style installation. You also have to rename the EGG-INFO directory to a .egg-info file of the same basename as the original .egg; otherwise, pkg_resources and other runtime access to the egg won't know it's installed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Phillip J. Eby [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] To Paul's question: I have only installed a couple of things (and not recently) that added their own add/remove entry. But I am not sure I would have called them add-ons as opposed to independent applications written in Python. | I presume he means that there are a lot of entries in his Add/Remove | Programs that work like that, and that it's an emerging standard for | Windows. (Certainly I've seen quite a few entries like that in mine, | although more often than not they only have one checkbox!) Yes. At least half my experience with uninstalls is removing games. Recent games typically have separate boxes for various things such as games files, save files, mods, game directory and any user added content, and icons and registry entries. Most Python add-ons I have downloaded are unziped to site-packages and only a few megabytes in size (versus gigabytes for some games). Hence there is little need to uninstall them (unless dumping everything connected with pyx.y, which is easy). Hence no desire to have add/remove slowed down and cluttered with dozens of entries for such things. I admit that my wish for a better installation manager is something I can only help with on the surface by expressing desires and testing results as a practice user. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
zooko: Um, isn't this tool called unzip? I have done this -- accessed the source code -- many times, and unzip suffices. The type of issue I ran into with eggs is when you get an exception with a trace that includes an egg, you can't use the normal means to look at the code. Instead you have to understand that its an egg, unzip the code, manually translate the path, open the file and go to the line number. Similarly, you can't easily grep the code in its egg state. If there was a global flag where I could say 'install eggs as directories of source' then I'd be much happier. Just reread the EasyInstall documentation and '--always-unzip' is portrayed as a 'don't do this' option. As it is I just avoid eggs. They may make sense for installing applications but for development they get in the way. Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 20/03/2008, zooko [EMAIL PROTECTED] wrote: On Mar 19, 2008, at 3:23 PM, Guido van Rossum wrote: If other people want to chime in please do so; if this is just a dialog between Phillip and me I might incorrectly assume that nobody besides Phillip really cares. I really care. I've used setuptools, easy_install, eggs, and pkg_resources extensively for the past year or so (and contributed a few small patches). There have been plenty of problems, but I find them to be overall useful tools. I'll chime in here, too. I really want to like setuptools/easy_install, but I don't. I'll try to be specific in my reasons, in the hope that they can be addressed. I know some of these are known about, but one of my meta-dislikes of setuptools is that known issues never seem to get addressed (I know, patches accepted, but I haven't got the time either...) 1. No integration with the system packager (Windows, in my case). If I do easy_install nose, then nose does not show up in add/remove programs. That significantly affects the way I manage my PC. 2. No uninstaller. After easy_install nose, how do I get rid of it later? Searching for files to delete (even if there are only a few obviously named ones) is not good enough. 3. The pkg_resources documentation (in particular, that's the one I've tried to follow) is extremely hard to read. Partly this is just style, but it's partly because it is couched in very unfamiliar terms (distributions, working sets, interfaces, providers, etc). It's also *huge*. A tutorial style overview, supported by API detail, would be far better. 4. Hard to use with limited connectivity. At work, I *only* have access to the internet via Internet Explorer (MS based proxy). There are workarounds, but ultimately download an installer, then run it is a far simpler approach for me. 5. Auto-discovery doesn't always work. I'm sorry, I really can't recall the example at the moment, but sometimes easy_install says it can't find a package I *know* is available. 6. Splitting the community. Windows users rely heavily on binary installers (at least, I do). We're starting to get a situation where some projects provide .egg files, and some provide traditional (bdist_wininst/bdist_msi) installers. This is bad. One way to do it, and all that :-) But if these problems are solved, then I have no problem with seeing the features of setuptools added to the standard library - resource APIs, plugin/entry point APIs, ways to create executable scripts, and such things *should* be standardised. Dependency resolution and automatic installation isn't something I like (probably because as a Windows user I've never used such a system, so I mistrust it) but if it works *with* the system and not against it, I don't mind. I hope this helps, Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 09:33 am, [EMAIL PROTECTED] wrote:
I'll chime in here, too. I really want to like
setuptools/easy_install, but I don't. I'll try to be specific in my
reasons, in the hope that they can be addressed. I know some of these
are known about, but one of my meta-dislikes of setuptools is that
known issues never seem to get addressed (I know, patches accepted,
but I haven't got the time either...)
I agree with almost everything that Paul says, and he put it quite well,
so I'll spare the me too, but I do have some additional gripes to add.
setuptools (or, for that matter, distutils) has no business installing
stuff in the system directory on a Linux box with a package manager.
The *major* benefit I can see to a tool like easy_install is providing a
feature that system packagers do not: allowing developers to quickly
pull down all their dependencies into a *user directory* without
worrying about system administration. However, not only does setuptools
not do this on its own, it actively fights me when I try to do it.
Admittedly, my user directory is a little messed up. Combinator, the
Divmod path management / developer deployment tool, does some
inadvisable things to attempt to trick distutils into doing local
installation. However, setuptools does have some pretty clear bugs in
this area; for example, it hard-codes a copy of a list that's present in
site.py to try to figure out where .pth files will be honored, rather
than looking at what's actually on sys.path. Every time I've tried to
install a package for development using setuptools - and I am speaking
across a wide range of versions here, so this isn't a specific bug
report - it's either emitted a completely inscrutable traceback or
printed an error message telling me that it couldn't or wouldn't install
to the provided location.
But if these problems are solved, then I have no problem with seeing
the features of setuptools added to the standard library - resource
APIs, plugin/entry point APIs, ways to create executable scripts, and
such things *should* be standardised. Dependency resolution and
automatic installation isn't something I like (probably because as a
Windows user I've never used such a system, so I mistrust it) but if
it works *with* the system and not against it, I don't mind.
This is more of a vague impression than a specific technical criticism,
but it really seems like the implementation of these features face a lot
of unnecessary coupling in setuptools. Twisted (Hey, did you guys know
I work on Twisted? It seems I hardly ever mention it!), for example,
implements resource APIs (twisted.python.modules), plugins
(twisted.plugin, which is a bit like some of the uses of entrypoints),
and the zip-file agnosticism of both (twisted.python.zipstream) without
needing any packaging metadata or ini files. It just introspects the
Python path and adds a little frosting to importers.
I could be wrong about setuptools' actual design; this could be a
documentation or UI issue, because I haven't read the code. But when
interacting with it as a user and perusing its API, it definitely seems
as though things are woven too tightly together, and the whole thing is
very centered around the concept of a build, i.e. running some kind of
compilation or deployment step with a setup.py. One of my favorite
things about python is that stuff just runs without needing that
normally. I know that setup.py develop is supposed to avoid that -
but even that itself is a deployment step which generates some metadata.
With the setuptools-free libraries I use, it's just check out, then run;
no other intermediary step. I'm spoiled, of course, having apt to do
the package-management work for me on the majority of my dependencies,
and combinator mostly handling the rest.
easy_install also definitely has problems with security. It
automatically downloads links from plain-HTTP connections, some of them,
I believe, from publicly editable wiki pages, and installs them with no
verification of signatures (as root! because how else are you going to
get them to the only supported installation directory!). I believe that
this is possibly the easiest issue to fix though, and I hope that my
information here is already out of date. I realize that people are
already doing this (insecure installation) with their web browsers, but
there are tons of UI cues in a web browser looking at a link on a wiki
page which you don't get from an automated command-line tool.
As others have said, I wanted to like setuptools. I wanted to believe
we could be saved from the unfortunate issues with distutils. But the
extremely high degree of frustration I've encountered every time I've
tried to use it, its general inscrutability, its awful UI (python -c
import setuptools; execfile('setup.py') develop, seriously? you
couldn't write a command-line tool to make that look like 'setuptool
develop' or something?) and now the availability of
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul Moore wrote: I'll chime in here, too. I really want to like setuptools/easy_install, but I don't. I'll try to be specific in my reasons, in the hope that they can be addressed. I know some of these are known about, but one of my meta-dislikes of setuptools is that known issues never seem to get addressed (I know, patches accepted, but I haven't got the time either...) Thanks for feedback from the Windows world, from whence I have been blissfully exiled now for years. 1. No integration with the system packager (Windows, in my case). If I do easy_install nose, then nose does not show up in add/remove programs. That significantly affects the way I manage my PC. Point taken. Of course, it isn't really a program at that point: it is an installed add-on to Python. However, if Windows users expect such add-ons to show up in the system list, that is good to know. I'll note that I use easy_install *only* to work in *non-system* locations: if I want to install Python packages to /usr/lib/python2.x/, I use the standard system installer, e.g. 'apt-get install python-frobnatz'. But I routinely create non-system Python environments for development, using either alternate Pythons or virtualenv: in those environments, it works very well for me. 2. No uninstaller. After easy_install nose, how do I get rid of it later? Searching for files to delete (even if there are only a few obviously named ones) is not good enough. People ask for this on Unix platforms as well, often adding a request that pacakges installed only as dependencies of the package-being-removed go away as well. If you install everything in a way that works with system package manager, of course, you don't need this. ;) Deleting the 'lib/python2.x/site-packages/foo-X.Y.X.egg' directory is all that is actually required to uninstall an egg that was previouly added via easy_install. Cleaning out the equivalent entry in 'easy_install.pth' in that directory is not strictly required. I wonder if a GUI for managing the add-ons would fit the bill, as an alternative to packaging them as though they were standalone programs? 3. The pkg_resources documentation (in particular, that's the one I've tried to follow) is extremely hard to read. Partly this is just style, but it's partly because it is couched in very unfamiliar terms (distributions, working sets, interfaces, providers, etc). It's also *huge*. A tutorial style overview, supported by API detail, would be far better. Many of those terms are distutils jargon, actually. I think Jeff Rush' recent work looks like a good start here. 4. Hard to use with limited connectivity. At work, I *only* have access to the internet via Internet Explorer (MS based proxy). There are workarounds, but ultimately download an installer, then run it is a far simpler approach for me. I don't know how to make this requirement compatible with using shared dependencies, except to make it easier for folks to download *all* the requirements, and later install from the local distribution cache (a directory full of .zip / .egg / .tgs files). It does turn out to be quite easy to build a PyPI-style simple index for such a cache. Your use case would then require: 1. Run some command to fetch the desired package and the transitive closure of its dependencies into a working directory (the cache). 2. Run another command to build an index for that directory. 3. Run 'easy_install', pointing to the local index. 5. Auto-discovery doesn't always work. I'm sorry, I really can't recall the example at the moment, but sometimes easy_install says it can't find a package I *know* is available. Usually this indicates that there are incompatible dependencies between packages already installed and those on the index. E.g., if I already have package foo installed, but its version is not compatible with the requirements for package bar, then I can't install bar, even though the distribution is available. Because PyPI is not a centrally-managed index of packages, such conflicts are pretty much inevitable over time for those who don't subset it in some form (what we've been calling the known good set strategy in Zope-land). 6. Splitting the community. Windows users rely heavily on binary installers (at least, I do). We're starting to get a situation where some projects provide .egg files, and some provide traditional (bdist_wininst/bdist_msi) installers. This is bad. One way to do it, and all that :-) If it weren't for the Add / Remove Programs requirement you mentioned above, we would be better off if authors of pure Python packages uploaded only 'sdist' distributions, which can be cleanly converted to platform-local eggs at install time, even on Windows. Packages which contain C extensions typically must upload the 'bdist_win' version for the benefit of the vast majority of Windows users who can't bulid the extensions locally. Uploading any
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Tres Seaver wrote: Point taken. Of course, it isn't really a program at that point: it is an installed add-on to Python. However, if Windows users expect such add-ons to show up in the system list, that is good to know. Are things really that different in the non-Windows worlds? If I want python-nose, I run sudo apt-get install python-nose (and that means I can always remove it with sudo apt-get remove ...). Seems more similar than different (ignoring the silliness of Microsoft's insistence on the GUI is the OOWTDI even for such administrative tasks as installing system-wide software). -- Bob Kline http://www.rksystems.com mailto:[EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Bob Kline wrote: Are things really that different in the non-Windows worlds? If I want python-nose, I run sudo apt-get install python-nose (and that means I can always remove it with sudo apt-get remove ...). Seems more similar than different (ignoring the silliness of Microsoft's insistence on the GUI is the OOWTDI even for such administrative tasks as installing system-wide software). I was going to -- pointedly -- drop in here the help output for msiexec, which is the commandline version of the MSI installation graphical stuff. Only... when I did msiexec /?, the result was that a Window popped up with the information in it. (Sort of agrees with your point a bit!) Still, here's the info (cut-and-pasted from that window): - Windows ® Installer. V 3.01.4000.1823 msiexec /Option Required Parameter [Optional Parameter] Install Options /package | /i Product.msi Installs or configures a product /a Product.msi Administrative install - Installs a product on the network /ju|m Product.msi [/t Transform List] [/g Language ID] Advertises a product - m to all users, u to current user /uninstall | /x Product.msi | ProductCode Uninstalls the product [... snip lots of other options ...] TJG ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Guido van Rossum schrieb: I was using the human interface at python.org/pypi. There are two prominent links at the top of the page: Browse the tree of packages and Submit package information followed by the 30 most recently changed packages. Ah, ok. In PyPI parlance, these are classifiers (Trove classifiers, although the word trove means nothing to me), not keywords. They are different from keywords in the sense that they form a hierarchy. I personally consider trove classifiers over-valued, but apparently, some people really love them (probably the ones who are more organized than I am). Developers continuously request addition of new classifiers; I don't have any statistics whether users actually use them to locate stuff. What I was looking for was the page for a specific package. The Browse the tree of packages link was no help. Finally I realized that in the side bar, in a small unobtrusive font, is a link to List packages which links to a list of *all* packages, in alphabetical order. I found my package there. I think repeating that link right below browse the tree would have been sufficient. I can't change that right now, but created http://sourceforge.net/tracker/index.php?func=detailaid=1921108group_id=66150atid=513503 But it would have been cool if there had been a search box (also in the start page) where I could type (part of) the name of the package and it would have given me the nearest matches. Did you try the search box in the top-right, and did did not work? What search term did you enter, and what package did you expect to get? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:33 AM 3/20/2008 +, Paul Moore wrote: 1. No integration with the system packager (Windows, in my case). If I do easy_install nose, then nose does not show up in add/remove programs. That significantly affects the way I manage my PC. The long-term fix here is probably to have a platform-specific installer capable of either turning eggs into .msi or .exe installers, or of doing the add/remove programs integration directly. (Someone, of course, will have to step up to create such a tool.) 5. Auto-discovery doesn't always work. I'm sorry, I really can't recall the example at the moment, but sometimes easy_install says it can't find a package I *know* is available. Sometimes it does that to me, too. But then I look at the project's page in PyPI, and they don't have a link to a download page. Usually, they've got a link to a page on their site with instructions about downloading, but that doesn't directly link to any tarballs or anything. So I grab the link of the real download page and paste it into a -f option to easy_install, so it knows where to find the link from. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:44 AM 3/20/2008 -0400, Tres Seaver wrote: I don't know how to make this requirement compatible with using shared dependencies, except to make it easier for folks to download *all* the requirements, and later install from the local distribution cache (a directory full of .zip / .egg / .tgs files). It does turn out to be quite easy to build a PyPI-style simple index for such a cache. Your use case would then require: 1. Run some command to fetch the desired package and the transitive closure of its dependencies into a working directory (the cache). 2. Run another command to build an index for that directory. 3. Run 'easy_install', pointing to the local index. Actually, if someone were to develop a patch for PyPI to do this, we could perhaps have a display download dependencies link for eggs shown on PyPI. That way, someone who wants to do a manual download could get a page with links for all the required eggs, and manually download them. (Of course, the other alternative would be for someone to provide an IE-controlling extension to urllib2 so that easy_install wouldn't be proxy-bound on such machines.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
-On [20080320 15:29], Martin v. Löwis ([EMAIL PROTECTED]) wrote: (Trove classifiers, although the word trove means nothing to me) Isn't that something lifted from SourceForge? -- Jeroen Ruigrok van der Werven asmodai(-at-)in-nomine.org / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ The Eyes of Truth are always watching you... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
-On [20080320 05:58], Tres Seaver ([EMAIL PROTECTED]) wrote: I think that, warts an all, setuptools is a *huge* improvement over bare distutils for nearly every use case I know about. Agreed. I see setuptools (along with PyPI - hopefully much better in near future though) as the Python equivalent to CPAN and RubyGems. -- Jeroen Ruigrok van der Werven asmodai(-at-)in-nomine.org / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ Sadness is the inner beauty of the untouched tear... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
I'll note that I use easy_install *only* to work in *non-system* locations: if I want to install Python packages to /usr/lib/python2.x/, I use the standard system installer, e.g. 'apt-get install python-frobnatz'. This is probably not the Windows way of doing things (at least not how I use Windows). Windows doesn't really have the notion of system location (or multiple levels of them, where \Windows and \Windows\system32 is more system than \Program Files, say). Windows users typically view the entire system as theirs, and have no concerns at all about putting things into Program Files, system32, or, for that matter, \python25. In fact, they don't care or even know where stuff ends up - they expect that the system will find it, and they expect that they can remove anything they installed even without known where it is - because there is a standard place to look for uninstalling things. Of course, setuptools is not the only piece of software that doesn't play well, so Windows users collect all kinds of cruft over time. Eventually, C: will run out of disk space, and they either get a new machine, or reinstall from scratch. I wonder if a GUI for managing the add-ons would fit the bill, as an alternative to packaging them as though they were standalone programs? On Windows, it is fairly easy to have an uninstaller registered. There are wrappers to managing that (such as MSI), but it's really only a set of registry keys that needs to get written on installation time, one of them being the command to run on uninstallation. Assuming that you uninstall the package before uninstalling Python, that uninstall program could be a Python script (although using a cmd.exe batch file would probably be more resilient). The concern with you just need to delete the folder is how am I supposed to know that? and can I be really sure?. If you run the official uninstall procedure, and it messes things up, you can complain to setuptools, or the package author that uninstallation doesn't work. If you delete stuff manually, and you forgot to remove something in a remote location you didn't even know it existed, you still think it's your own fault. So people are hesitant to actually execute the procedure. Of course, once you *do* provide an entry to Add/Remove Programs, uninstalling won't be mere deletion, as mere deletion would still leave these registry keys behind (although Windows got more resilient over time to provide cleanup in that case: I believe it offers to remove the ARP entry if the uninstall program has been removed) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Actually, if someone were to develop a patch for PyPI to do this, we could perhaps have a display download dependencies link for eggs shown on PyPI. That way, someone who wants to do a manual download could get a page with links for all the required eggs, and manually download them. Just to make this position a bit more official (as one of the PyPI maintainers): it would be fully within the scope of PyPI to integrate dependency tracking into its database, and present it in any form that is desired. Any such feature would have to be contributed. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Martin v. Löwis wrote: Martin v. Löwis wrote: I'll note that I use easy_install *only* to work in *non-system* locations: if I want to install Python packages to /usr/lib/python2.x/, I use the standard system installer, e.g. 'apt-get install python-frobnatz'. This is probably not the Windows way of doing things (at least not how I use Windows). Windows doesn't really have the notion of system location (or multiple levels of them, where \Windows and \Windows\system32 is more system than \Program Files, say). Windows users typically view the entire system as theirs, and have no concerns at all about putting things into Program Files, system32, or, for that matter, \python25. In fact, they don't care or even know where stuff ends up - they expect that the system will find it, and they expect that they can remove anything they installed even without known where it is - because there is a standard place to look for uninstalling things. Of course, setuptools is not the only piece of software that doesn't play well, so Windows users collect all kinds of cruft over time. Eventually, C: will run out of disk space, and they either get a new machine, or reinstall from scratch. I wonder if a GUI for managing the add-ons would fit the bill, as an alternative to packaging them as though they were standalone programs? On Windows, it is fairly easy to have an uninstaller registered. There are wrappers to managing that (such as MSI), but it's really only a set of registry keys that needs to get written on installation time, one of them being the command to run on uninstallation. Assuming that you uninstall the package before uninstalling Python, that uninstall program could be a Python script (although using a cmd.exe batch file would probably be more resilient). The concern with you just need to delete the folder is how am I supposed to know that? and can I be really sure?. If you run the official uninstall procedure, and it messes things up, you can complain to setuptools, or the package author that uninstallation doesn't work. If you delete stuff manually, and you forgot to remove something in a remote location you didn't even know it existed, you still think it's your own fault. So people are hesitant to actually execute the procedure. Of course, once you *do* provide an entry to Add/Remove Programs, uninstalling won't be mere deletion, as mere deletion would still leave these registry keys behind (although Windows got more resilient over time to provide cleanup in that case: I believe it offers to remove the ARP entry if the uninstall program has been removed) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org I'll note that I use easy_install *only* to work in *non-system* locations: if I want to install Python packages to /usr/lib/python2.x/, I use the standard system installer, e.g. 'apt-get install python-frobnatz'. This is probably not the Windows way of doing things (at least not how I use Windows). Windows doesn't really have the notion of system location (or multiple levels of them, where \Windows and \Windows\system32 is more system than \Program Files, say). Windows users typically view the entire system as theirs, and have no concerns at all about putting things into Program Files, system32, or, for that matter, \python25. In fact, they don't care or even know where stuff ends up - they expect that the system will find it, and they expect that they can remove anything they installed even without known where it is - because there is a standard place to look for uninstalling things. In point of fact, for an *end user* it makes increasing sense to use application installers that automatically install a correct-version interpreter and all dependencies in a stand-alone manner (i.e. explicitly *not* sharing anything with any other installed application. This makes uninstall much easier, as the lack of external dependencies eases version lock-step problems. It would pain me, as a computer scientist, to do this, but I honestly believe it may be the way forward -- just think, it wouldn't even matter whether an application (and all its extension modules) had been built with VS2003, VS2008 or Mingw. People misunderstood when Mike Driscoll started to provide pure-Python modules as Windows installers, but increasingly your naive Windows programmer is going to be happier doing that. I'm not sure whether that provides easy_f**king_uninstall (Zed Shaw will live on in my memory for that particular PyCon moment), but it (ought to be) relatively easy to do so. Extension modules for programmers still offer more of a challenge, but a build-farm for extension module writers could help there. Of course,
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 05:55 PM 3/20/2008 +, Paul Moore wrote: It's not that I object to the existence of automatic dependency management, I object to being given no choice, as if my preference for handling things manually is unacceptable. Note that easy_install has a --no-deps option, and you can make it the default in your distutils.cfg file. Also, setuptools-based packages *can* build bdist_wininst installers. (In fact, if memory serves, I added that feature at your request.) Personally, I'm not very thrilled with the number of complaints on this thread that could be resolved by RTFMing. There are extensive manuals, and they do contain the information that some people are saying isn't there. In several cases that I've seen here today alone, there are actually *entries in the tables of contents* that name the precise thing people here are characterizing as undocumented or even *impossible*, like: * Making your package available for EasyInstall * Installing on Un-networked Machines * Custom Installation Locations * Restricting Downloads with --allow-hosts It's easy to get the impression that people not only didn't RTFM, they didn't even Read The Friendly Table Of Contents of the said M. Nor, when, they found something in the manual that they didn't understand, write to the distutils-sig to ask anybody to explain, and perhaps suggest ways the FM's could be improved. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 20/03/2008, Phillip J. Eby [EMAIL PROTECTED] wrote: At 05:55 PM 3/20/2008 +, Paul Moore wrote: It's not that I object to the existence of automatic dependency management, I object to being given no choice, as if my preference for handling things manually is unacceptable. Note that easy_install has a --no-deps option, and you can make it the default in your distutils.cfg file. Sorry, I wasn't clear. There's context that helps, but even with it I wasn't clear. Tres told me how to download all the dependencies for use offline (which I actually knew, but that's not the point here). I clarified that I didn't want to use dependency management but preferred to manage dependencies manually. I then went on to say that putting dependency information in setup.exe and expecting users to use automatic dependency resolution encourages developers to omit dependency details from documentation (to an extent I can't quantify, but I believe is non-zero). That lack of documentation forces me to rely on the automatic process. THAT is the thing that removes my choice, not easy_install's ability to skip dependency checking. I'm sorry I wasn't clearer - but in my defence, my posting was pretty long already and I was trying to be brief... Also, setuptools-based packages *can* build bdist_wininst installers. (In fact, if memory serves, I added that feature at your request.) I know. python setup.py bdist_wininst. And thank you for adding it. But again you miss my point. People are starting to omit distributing bdist_wininst installers in favour of eggs only. And you cannot (to my knowledge) convert an egg into a bdist_wininst installer, and if you can't compile from source (a C extension with complex dependencies, for example) you're stuck (in the sense that you're forced to use eggs without add/remove programs support). Personally, I'm not very thrilled with the number of complaints on this thread that could be resolved by RTFMing. There are extensive manuals, and they do contain the information that some people are saying isn't there. In several cases that I've seen here today alone, there are actually *entries in the tables of contents* that name the precise thing people here are characterizing as undocumented or even *impossible*, like: * Making your package available for EasyInstall * Installing on Un-networked Machines * Custom Installation Locations * Restricting Downloads with --allow-hosts It's easy to get the impression that people not only didn't RTFM, they didn't even Read The Friendly Table Of Contents of the said M. Nor, when, they found something in the manual that they didn't understand, write to the distutils-sig to ask anybody to explain, and perhaps suggest ways the FM's could be improved. As I said, I know there is documentation, but (a) it's very long, and (b) it's full of jargon that you need to understand before you can follow it. Believe me, I've tried to read it. But ultimately, all I'm trying to do is work out how to do something that is as simple as download exe, run it (on a PC with browser-only access to the internet) in the bdist_wininst world. That's all. I'm equally not very thrilled at having to read many pages of dense documentation to find out how to do this. Heck, I read the documentation twice, and asked on the distutils-sig, before I worked out how to do easy_install -zmax (and the only reason I can remember that now without looking it up is that zmax is memorable - z plus the word max - I have no idea without going back to the manual what the individual options do). I'd say that the documentation needs to be better. (And I said how - a tutorial-style summary. What more should I do short of writing it myself?) Honestly, I'm trying to help improve (by my measure of improvement, certainly) setuptools. I've done as much (more!) homework as I feel is appropriate (no, I haven't studied the whole manual all the way through). Being treated as if it's my fault, and I haven't done enough, is both discouraging and to be honest, somewhat offensive. I'm going to quit this thread for a while now, as I don't want to turn things into a flamewar. I believe Tres took my points on board. I hope others have, too. I certainly don't expect everything I say to be taken as gospel, so that's enough for now. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 08:34 PM 3/20/2008 +, Paul Moore wrote: I then went on to say that putting dependency information in setup.exe and expecting users to use automatic dependency resolution encourages developers to omit dependency details from documentation (to an extent I can't quantify, but I believe is non-zero). That lack of documentation forces me to rely on the automatic process. THAT is the thing that removes my choice, not easy_install's ability to skip dependency checking. Ah. Fair enough. So, if we get PyPI to display that information, that should fix this problem for you? People are starting to omit distributing bdist_wininst installers in favour of eggs only. You mean, they're shipping a .win32.egg, but not an .exe? And you cannot (to my knowledge) convert an egg into a bdist_wininst installer, Not at the moment, no. It seems like it ought to be *possible*, though, since the reverse translation can be done. Eggs are more restrictive in what they can include, so the reverse step actually ought to be relatively easy. Indeed, I would think that it could be done by a standalone tool without even using setuptools. All that really needs to happen (I believe) is that the zipfile directory needs all its names prepended with PURELIB or PLATLIB, and then add the appropriate prefix .exe and bdist_wininst extra data on the front of the restructured zip file. In fact, it should probably be possible to write such a tool by subclassing the distutils bdist_wininst command and overriding the run() and get_inidata() methods, using the existing create_exe() method to do that part of the magic. The other tool that would be handy to have, would be one that unpacks eggs into standard distutils-style installation. Personally, I'm not very thrilled with the number of complaints on this thread that could be resolved by RTFMing. ... Honestly, I'm trying to help improve (by my measure of improvement, certainly) setuptools. I've done as much (more!) homework as I feel is appropriate (no, I haven't studied the whole manual all the way through). Being treated as if it's my fault, and I haven't done enough, is both discouraging and to be honest, somewhat offensive. My comment wasn't aimed specifically at you; you're only one of many people today who have appeared to state that something or other wasn't possible or documented, described optional behavior as required, etc. Addressing each and every one point by point looks petty, but then lumping them together like that makes it look like I'm picking on you specifically. Sorry about that. In any event, I'm not saying that anyone hasn't done enough or that it's their fault. The fact that I'm not thrilled about some of the things said in the thread doesn't somehow magically invalidate other people's frustrations, nor was it my intent to accuse you (or anyone) of making up their problems. I'm just expressing *my* frustration. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 2008-03-20 21:34, Paul Moore wrote: Also, setuptools-based packages *can* build bdist_wininst installers. (In fact, if memory serves, I added that feature at your request.) I know. python setup.py bdist_wininst. And thank you for adding it. But again you miss my point. People are starting to omit distributing bdist_wininst installers in favour of eggs only. And you cannot (to my knowledge) convert an egg into a bdist_wininst installer, and if you can't compile from source (a C extension with complex dependencies, for example) you're stuck (in the sense that you're forced to use eggs without add/remove programs support). You might want to look at the eGenix pre-built packages as an alternative: they include all the information necessary to let standard distutils continue its works *after* the build step. It's basically a distribution of the package as it looks after the build step has run, but before the package is wrapped up using a packager like bdist_wininst or bdist_msi or installed into the system. You can download the pre-built package and create e.g. an MSI installer or a wininst EXE without needing a compiler - in addition to providing all the options of the standard distutils install command (which makes repackaging them as part of larger applications easy as well). All the logic for this is included in mxSetup.py which ships with the pre-built packages. http://www.egenix.com/products/python/mxBase/#Download http://www.egenix.com/products/python/mxBase/#Installation The current version we have is not yet perfect. The next iteration will also play nice with distutils extensions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 20 2008) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:58 AM 3/20/2008 -0400, Tres Seaver wrote: A lot of setuptools warts are driven by related design problems in the distutils, such as the choice to use imperative / procedural code for everything If a distutils replacement is ever written, I'd like to see it structured as a dependency graph, like a makefile, with each node in the graph knowing how to transform its inputs into its outputs. That would make it a lot easier to extend to accommodate new things like Pyrex. You'd just have to write a new node class that knows how to turn .pyx files into .c files, and the existing machinery would take it from there. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Phillip J. Eby wrote: The other tool that would be handy to have, would be one that unpacks eggs into standard distutils-style installation. Hear, hear. I'm an author of a couple libraries that need to interoperate with others. Of the many eggs I've downloaded over the past year, I'd say 80%+ are never installed or even built--I just want to grep the source code, and using my preferred tools, not some lame Find command in a ZIP browser menu. Robert Brewer [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Tres Seaver [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] | -BEGIN PGP SIGNED MESSAGE- | Hash: SHA1 | | Paul Moore wrote: | | 1. No integration with the system packager (Windows, in my case). If I | do easy_install nose, then nose does not show up in add/remove | programs. That significantly affects the way I manage my PC. | | Point taken. Of course, it isn't really a program at that point: it | is an installed add-on to Python. However, if Windows users expect | such add-ons to show up in the system list, that is good to know. However, this Windows user, and I expect most, do NOT expect add-ons (things under the /Pythonx.y tree) to show up in the add/remove list. Please do not do that. On my system, it already takes several seconds to 'populate the list'. It is bad enough that Windows Update occasionally adds entries like 'Windows Update KB284798324' instead of adding something to the separate 'Manage Windows components' subsystem with readable names and explanations. The standard (and to me, preferable) way of dealing with such things is to have an 'installation manager' that can reinstall as well as delete and that has a check box for various things to delete. This is what Python needs. Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mar 20, 2008, at 7:44 AM, Tres Seaver wrote: Paul Moore wrote: 4. Hard to use with limited connectivity. At work, I *only* have access to the internet via Internet Explorer (MS based proxy). There are workarounds, but ultimately download an installer, then run it is a far simpler approach for me. I don't know how to make this requirement compatible with using shared dependencies, We've done something like this. The http://allmydata.org project bundles its easy_installable dependencies. If you get the current trunk from our darcs repository [1], or get a release tarball or a snapshot tarball from [2], then it comes with a directory named misc/dependencies which has the source tarballs of our easy_installable dependencies. You can browse this directory on the web: [3]. Therefore, if you manually satisfy the non-easy_installable dependencies, you can download an allmydata.org tarball, disconnect from the Internet (which we call moving to a Desert Island), and install it. This is, as you say, compatible with using shared dependencies because setuptools will detect if you already have sufficiently new versions of some of these dependencies installed (for example, if they are installed in Debian packages), and then skip the step of installing that dependency from its source tarball. The remaining dependencies that cannot be satisfied automatically by our setup.py are listed in the install.html [4]. They are: 1. g++ = v3.3 -- the Cygwin version of gcc/g++ works for Cygwin and for Windows 2. GNU make 3. Python = v2.4.2 including development headers i.e. Python.h 4. Twisted = v2.4.0 -- from the Twisted sumo source tarball 5. OpenSSL = v0.9.7, including development headers 6. PyOpenSSL == v0.6 7. Crypto++ = v5.2.1, including development headers I am hoping that in the future Twisted (see twisted #1286 [5]) and pyOpenSSL will be easy_installable, and that our use of setuptools plugins will eventually replace our GNUmakefile and thus remove our dependency on GNUmake. That will leave only g++, Python, OpenSSL, and Crypto++ as dependencies that a user has to manually deal with in order to build allmydata.org from source. Regards, Zooko [1] http://allmydata.org/source/tahoe/trunk/ [2] http://allmydata.org/source/tahoe/tarballs/ [3] http://allmydata.org/trac/tahoe/browser/misc/dependencies [4] http://allmydata.org/source/tahoe/trunk/docs/install.html [5] http://twistedmatrix.com/trac/ticket/1286 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Guido van Rossum wrote: On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: [a long message] I'm back at Google and *really* busy for another week or so, so I'll have to postpone the rest of this discussion for a while. If other people want to chime in please do so; if this is just a dialog between Phillip and me I might incorrectly assume that nobody besides Phillip really cares. Since there seems to be a fair number of negative responses to setuptools, I just wanted to add a bit of positive counterbalance. I'm just a random python user that happens to track python-dev a bit, so take all this with the realization that I probably shouldn't have much input into anything. ;) I've been using python for somewhere around 10 years to write various random small scripts, gui applications and recently web applications. For me setuptools is the best thing to happen to python since I've been using it. I develop and deploy on a seemingly constantly changing mix of various flavors of windows and linux. Unlike for others, I love that once I get setuptools installed I can just use the same commands to get the things I need. I guess the contrast for me is that python is the common base that I tend to work from not the underlying OS. So I don't know if I'm part of a large number of quiet users or just happen to be an odd case that works really well with setuptools. I was disappointed when setuptools didn't make it into 2.5 and I really hope it or something very much like it can make it into a release in the near future. Because while setuptools certainly isn't perfect, for me at least, it is much, much better than nothing at all. Brian Haskin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Janzert wrote: Since there seems to be a fair number of negative responses to setuptools, I just wanted to add a bit of positive counterbalance. I'm just a random python user that happens to track python-dev a bit, so take all this with the realization that I probably shouldn't have much input into anything. ;) I've been using python for somewhere around 10 years to write various random small scripts, gui applications and recently web applications. For me setuptools is the best thing to happen to python since I've been using it. I develop and deploy on a seemingly constantly changing mix of various flavors of windows and linux. Unlike for others, I love that once I get setuptools installed I can just use the same commands to get the things I need. I guess the contrast for me is that python is the common base that I tend to work from not the underlying OS. So I don't know if I'm part of a large number of quiet users or just happen to be an odd case that works really well with setuptools. I was disappointed when setuptools didn't make it into 2.5 and I really hope it or something very much like it can make it into a release in the near future. Because while setuptools certainly isn't perfect, for me at least, it is much, much better than nothing at all. My interpretation of this is that setuptools suffers from the same malaise all flexible apps do (but especially CLI apps it seems): frequent users love the power and high volume of options, infrequent users despise it. If you're installing apps all day, you probably use it a lot more often than library devs like me who use it once every other month (if we're forced to). Robert Brewer [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Tue, Mar 18, 2008 at 3:36 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote: Only very few people would care about writing a setup script that works with this bootstrap module; basically only package manager implementers. That's true today, sure, but as soon as it is widely available, others are sure to want to use it too. I just want a bright-line distinction between what is and isn't bootstrappable, rather than a murky region of maybe, if you're not doing anything too complicated. How about anything that uses only distutils in its setup.py and doesn't have external dependencies? See a (horribly incomplete) prototype I added as sandbox/bootstrap/bootstrap.py. I wrote this on the plane last night and have only tested it with file:/// URLs; it needs to add the ability to consult PyPI to find the download URL, and probably more. (PS: just now I also managed to successfully install setuptools from source by giving it the URL to the gar.gz file.) There seems to be a misunderstanding about what I am proposing we do instead. The boostrap installer should only be powerful enough to allow it to be used to install a real package manager like setuptools. Which is why PEP 365 proposed only downloading an archive to a cache directory, and optionally running something from it. It explicitly disavows installation of anything, since the downloaded archive wouldn't have been added to sys.path except for the duration of the bootstrap process, and no scripts were to be installed. (Indeed, apart from the methods it would have used to locate the archive on PyPI, and to determine what to run from inside it, there was nothing particularly egg-specific about the proposed bootstrapping process.) My bootstrap.py does exactly that: it downloads and unzips/untars a file and runs its setup.py with install as the only command line argument. (It currently looks for setup.py at the toplevel and one level deep in the unpacked archive.) Of course you will likely have to be root or administrator to run it effectively. So, to fully egg-neutralize the bootstrapping approach, we need only know how to locate an appropriate archive, and how to determine what to run from it. Right. For the latter, we could use the already-in-2.6 convention of running __main__ from a zipfile or directory. (Too bad distutils source distributions have an extra directory name embedded in them, so one can't just execute them directly. Otherwise, we could've just let people drop in a __main__.py next to setup.py. OTOH, maybe it would be enough to use setuptools' algorithm for finding setup.py to locate __main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.) What's wrong with just running setup.py install? I'd rather continue existing standards / conventions. Of course, it won't work when setup.py requires setuptools; but old style setup.py files that use only distutils work great (I managed to install Django from a file:/// URL). The other open question is a naming convention and version detection, so that the bootstrap tool can identify which of the files listed on PyPI is suitable for its use. (Both with regard to the version selection, and file type.) However, if PyPI were to grow support for designating the appropriate files and/or versions in some other way, we wouldn't need a naming convention as such. I don't understand PyPI all that well; it seems poor design that the browsing via keywords is emphasized but there is no easy way to *search* for a keyword (the list of all packages is not emphasized enough on the main page -- it occurs in the side bar but not in the main text). I assume there's a programmatic API (XML-RPC?) but I haven't found it yet. Without one or the other, the bootstrap tool would have to grow a version parsing scheme of some type, and play guessing games with file extensions. (Which is one reason I limited PEP 365's scope to downloading eggs actually *uploaded* to PyPI, rather than arbitrary packages *linked* from PyPI.) There are two version parsers in distutils, referenced by PEP 345, the PyPI 1.2 metadata standard. So, if I had to propose something right now, I would be inclined to propose: * using setuptools' version parsing semantics for interpretation of alpha/beta/dev/etc. releases Can you point me to the code for this? What is its advantage over distutils.version? * having a bdist_bootstrap format that's essentially a bdist_dumb .zip file with the internal path prefixes stripped off, making it an importable .zip with a different file extension. (Or maybe just .pyboot.zip?) The filename convention would use setuptools' canonicalization and escaping of names and version numbers, to allow unambiguous machine parsing of the filename. A __main__ module would have to be present for the archive to be run, as opposed to just being downloaded
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 10:48 AM 3/19/2008 -0700, Guido van Rossum wrote: I don't understand PyPI all that well; it seems poor design that the browsing via keywords is emphasized but there is no easy way to *search* for a keyword (the list of all packages is not emphasized enough on the main page -- it occurs in the side bar but not in the main text). I assume there's a programmatic API (XML-RPC?) but I haven't found it yet. http://wiki.python.org/moin/CheeseShopXmlRpc There's also a REST API that setuptools uses: http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api The API was originally designed for screen-scraping an older version of PyPI, but that has been replaced with a lite version served from: http://pypi.python.org/simple/ The lite version is intended for tools such as easy_install to process, as it consists strictly of links and can be statically cached. Zope Corp., for example, maintains a static mirror of this API, to guard themselves against PyPI outages and slowdowns, since their buildouts can involve huge numbers of eggs, both their own and external dependencies. I'd love it if you could write or point me to code that takes a package name and optional version and returns the URL for the source archive, and the type (in case it can't be guessed from the filename or the Content-type header). You can probably do that with the XML-RPC API. There's a function to get the versions of a package, given a (case-sensitive) name, and there's a function to get information for uploaded archives, given a name and a version. I originally intended to use it for the PEP 365 approach, but you can get the necessary information in just one static roundtrip using the REST (/simple) HTML API, if you're willing to parse the URLs for version information. (The catch of course being that distutils source distributions don't have unambiguously parseable filenames.) Hm. Why not just use the existing convention for running setup.py after unpacking? This works great in my experience, and has the advantage of having an easy fallback if you end up having to do this manually for whatever reason. Because I want bootstrap-ees to be able to use the bootstrap mechanism. For example, I expect at some point that setuptools will use other, non-self-contained packages, and other package managers such as zc.buildout et al also want to depend on setuptools without bundling it. * calling the bootstrap module 'bootstrap', as in 'python -m bootstrap projectname optionalversion'. The module would expose an API to allow it to be used programmatically as well as the command line, so that bootstrapped packages can use the bootstrap process to locate dependencies if they so desire. (Today's package management tools, at least, are all based on setuptools, so if it's not present they'll need to download that before beginning their own bootstrapping process.) This sounds like going beyond bootstrapping. My vision is that you use the bootstrap module (with the command line you suggest above) once to install setuptools or the alternate package manager of your choice, and then you can use easy_install (or whatever alternative) to install the rest. Well, I noticed that the other package managers were writing bootstrap scripts that then download setuptools' bootstrap script and run it as part of *their* bootstrap process... and then I got to thinking that it sure would be nice for setuptools to not have to be a giant monolithic download if I wanted to start using other packages in it... and that it sure would be nice to get rid of all these bootstrap scripts downloading other bootstrap scripts... and then I wrote PEP 365. :) One other thing that PEP 365 does for these use cases that your approach doesn't, is that pkg_resources could detect whether a desired package of a usable version was *already* installed, and skip it if so. So, we've already scaled back the intended use cases quite a bit, as people will have to write their own is it already there? and is it the right version? checks. Without one or the other, the bootstrap tool would have to grow a version parsing scheme of some type, and play guessing games with file extensions. (Which is one reason I limited PEP 365's scope to downloading eggs actually *uploaded* to PyPI, rather than arbitrary packages *linked* from PyPI.) There are two version parsers in distutils, referenced by PEP 345, the PyPI 1.2 metadata standard. Yes, and StrictVersion doesn't parse release candidates. And neither LooseVersion nor StrictVersion supports handling multiple pre/post-release tags correctly. (E.g. 1.1a1dev-r2753) So, if I had to propose something right now, I would be inclined to propose: * using setuptools' version parsing semantics for interpretation of alpha/beta/dev/etc. releases Can you point me to the code for this? What is its advantage over distutils.version? It implements
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: [a long message] I'm back at Google and *really* busy for another week or so, so I'll have to postpone the rest of this discussion for a while. If other people want to chime in please do so; if this is just a dialog between Phillip and me I might incorrectly assume that nobody besides Phillip really cares. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
I don't understand PyPI all that well; it seems poor design that the browsing via keywords is emphasized but there is no easy way to *search* for a keyword (the list of all packages is not emphasized enough on the main page -- it occurs in the side bar but not in the main text). I don't understand. What is browsing via keywords and how is that emphasized? (one I know that, I can look into ways for searching for keywords) I assume there's a programmatic API (XML-RPC?) but I haven't found it yet. The recommended programmatic API is http://pypi.python.org/simple/ Not sure what you were trying to achieve programmatically; typically people know what they want to install (e.g. threadedcomments), and then the tool goes directly to http://pypi.python.org/simple/threadedcomments/ Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mar 19, 2008, at 3:23 PM, Guido van Rossum wrote: If other people want to chime in please do so; if this is just a dialog between Phillip and me I might incorrectly assume that nobody besides Phillip really cares. I really care. I've used setuptools, easy_install, eggs, and pkg_resources extensively for the past year or so (and contributed a few small patches). There have been plenty of problems, but I find them to be overall useful tools. It is a great boon to a programming community to lower the costs of re-using other people's code. The Python community will benefit greatly once a way to do that becomes widely enough accepted to reach a tipping point and become ubiquitous. Setuptools is already the de facto standard, but it hasn't become ubiquitous, possibly in part because of egg hatred, about which more below. I've interviewed several successful Python hackers who hate eggs in order to understand what they hate about them, and I've taken notes from some of these interviews. (The list includes MvL, whose name was invoked earlier in this thread.) After filtering out yer basic complaining about bugs (which complaints are of course legitimate, but which don't indict setuptools as worse than other software of comparable scope and maturity), their objections seem to fall into two categories: 1. The very notion of package dependency resolution and programmable or command-line installation of packages at the language level is a bad notion. This can't really be the case. If the existence of such functionality at the programming language level were an inherently bad notion, then we would be hearing some complaints from the Ruby folks, where the Gems system is standard and ubiquitous. We hear no complaints -- only murmurs of satisfaction. One person recently reported to me that while there are more packages in Python, he finds himself re-using other people's code more often when he works in Ruby, because almost all Ruby software is Gemified, but only a fraction of Python software is Eggified. Often this complaint comes with the idea that eggs conflict with their system-level package management tools. (These are usually Debian/Ubuntu users.) Note that Ruby software is not too hard to include in operating system packaging schemes -- my Ubuntu Hardy apt-cache shows plenty of Ruby software. A sufficiently mature and widely supported setuptools could actually make it easier to integrate Python software into Debian -- see stdeb [1]. 2. Setuptools/eggs give me grief. What can really be the case is that setuptools causes a host of small, unnecessary problems for people who prefer to do things differently than PJE does. Personally, I prefer to use GNU stow, and setuptools causes unnecessary, but avoidable, problems for me. Many people object (rightly enough) to a ./setup.py install automatically fetching new software over the Internet by default. The fact that easy_install creates a site.py that changes the semantics of PYTHONPATH is probably the most widely and deservedly hated example of this kind of thing [2]. I could go on with a few other common technical complaints of this kind. These type-2 problems can be fixed by changing setuptools or they can be grudgingly accepted by users, while retaining compatibility with the large and growing ecosystem of eggy software. Certainly fixing setuptools to play better with others is a more likely path to success than setting out to invent a non-egg-compatible alternative. Such a project might never be implemented well enough to serve, and if it were it would probably never overtake eggs's lead in the Python ecosystem, and if it did it would probably not turn out to be a better tool. So, since you asked for my chime, I advise you to publically bless eggs, setuptools, and easy_install as plausible future standards and solicit patches which address the complaints. For that matter, soliciting specific complaints would be a good start. I've done so in private many times with only partial success as to the specific part. One promising approach is to request objections in the form of automated tests that setuptools fails, e.g. [3]. Regards, Zooko O'Whielacronx [1] http://stdeb.python-hosting.com/ [2] http://www.rittau.org/blog/20070726-02 And no, PJE's suggested trivial fix does not satisfy the objectors, as it can't support the use case of cd somepkg ; python ./ setup.py install ; cd .. ; python -c 'import somepkg'. [3] http://twistedmatrix.com/trac/ticket/2308#comment:5 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido van Rossum wrote: On Wed, Mar 19, 2008 at 12:54 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: [a long message] I'm back at Google and *really* busy for another week or so, so I'll have to postpone the rest of this discussion for a while. If other people want to chime in please do so; if this is just a dialog between Phillip and me I might incorrectly assume that nobody besides Phillip really cares. I care, a lot, enough to have volunteered to help with maintenance / development of setuptols back in September 2007. I think that, warts an all, setuptools is a *huge* improvement over bare distutils for nearly every use case I know about. A lot of setuptools warts are driven by related design problems in the distutils, such as the choice to use imperative / procedural code for everything: a declarative approach, with hooks for cases which actually need them (likely 5% of existing packages) would have made writing tools on top of the framework much simpler. It is ironic that Python is *too powerful* a tool for the tasks normally done by distutils / setuptools: a more restricted, and thererfore introspectable, configuration-driven approoach seems much cleaner. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH4e7m+gerLs4ltQ4RAt+hAKDBqIrashlgf8U6XRtfMHjTOaiy4gCeO1Zn UfdjDYIb2P6vDCcUGSjITTo= =JTok -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
I was using the human interface at python.org/pypi. There are two prominent links at the top of the page: Browse the tree of packages and Submit package information followed by the 30 most recently changed packages. What I was looking for was the page for a specific package. The Browse the tree of packages link was no help. Finally I realized that in the side bar, in a small unobtrusive font, is a link to List packages which links to a list of *all* packages, in alphabetical order. I found my package there. I think repeating that link right below browse the tree would have been sufficient. But it would have been cool if there had been a search box (also in the start page) where I could type (part of) the name of the package and it would have given me the nearest matches. On Wed, Mar 19, 2008 at 6:05 PM, Martin v. Löwis [EMAIL PROTECTED] wrote: I don't understand PyPI all that well; it seems poor design that the browsing via keywords is emphasized but there is no easy way to *search* for a keyword (the list of all packages is not emphasized enough on the main page -- it occurs in the side bar but not in the main text). I don't understand. What is browsing via keywords and how is that emphasized? (one I know that, I can look into ways for searching for keywords) I assume there's a programmatic API (XML-RPC?) but I haven't found it yet. The recommended programmatic API is http://pypi.python.org/simple/ Not sure what you were trying to achieve programmatically; typically people know what they want to install (e.g. threadedcomments), and then the tool goes directly to http://pypi.python.org/simple/threadedcomments/ Regards, Martin -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:31 AM 3/18/2008 -0500, Guido van Rossum wrote: I am hoping that someone will create a simpler bootstrap module that is able to download a file of pure Python code and install it, perhaps by running its setup.py, assuming that it only depends on distutils (or other things previously installed). I will welcome such a module into the stdlib. I'm not sure a PEP is even needed, though interested parties are certainly welcome to write a PEP specifying the behavior first. With 2.6 and 3.0 slated for release in September, there should be enough time to get this done before then. Unfortunately, as I've already tried to explain, download a file ... and install it is not a sufficiently well-specified requirement to implement a robust tool. Even if it is not to support arbitrary existing distutils sources, there still needs to be a way to document precisely what the tool does and does not support installing, so that users can produce correct files for it to consume, register them properly with PyPI, etc. And as I said before (perhaps not very well) the distutils documentation has already proven to be inadequate as such a specification, both for users to create these files -- and even more important -- for programs to consume them. (For example, distutils source distribution tarball filenames are not unambiguously machine-parseable.) That's why I think some specific format (i.e. conventions) have to be defined for this to work, even if it's merely a set of documented restrictions on a distutils-based layout, file naming conventions, versioning, etc. In other words, you can't have your cake and eat it, too. If there's to be a bootstrap tool, you must bless *some* set of packaging conventions, including file naming, version parsing, and so on. Can we use setuptools' version parsing scheme to identify the latest stable version, for example? What about setuptools' filename component canonicalization and escaping rules? Frankly, I don't care what the conventions are, only that they be unambiguously defined and reasonably implementable for producers and consumers alike. I just want there to be *some* sort of robust, documented, standard installation bootstrap vector in the stdlib, so that setuptools, zc.buildout, and virtualenv don't have to maintain their own (or depend on setuptools maintaining its own). But not only have you rejected the *only* existing robust and well-documented conventions for automated processing of Python libraries, you say you have no time for this part of the thread when I ask what conventions you want to bless *instead*. So I'm at a bit of a loss for what we're supposed to do now. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Guido van Rossum writes: I am hoping that someone will create a simpler bootstrap module FWIW (I've never tried to implement one of these things) I agree with Phillip. This is not possible in the sense you are advocating. Anything simpler is simply an invitation to an unending stream of issues, far more so than adopting eggs as a best current practice would. Better to Just Say No to an installer in the stdlib. I'm not sure a PEP is even needed This I simply don't understand. I *have* participated in bolting on features to such systems, and it's a mess. As features are added, existing users will demand that old packages install exactly as they ever did, except that sometimes (but not always!) they want them upgraded to put things in newly blessed places. Features are easy to add, since on the main thread of control installers are basically just a sequence of single commands, sometimes optional. python-dev has some pretty good controls that will help slow such trends. Nonetheless, PEP-less development of an installer system is scary. Installer complexity is a creeping horror, worse than kudzu.[1] Footnotes: [1] http://en.wikipedia.org/wiki/Kudzu#Invasive_species ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Tue, Mar 18, 2008 at 11:23 AM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 12:31 AM 3/18/2008 -0500, Guido van Rossum wrote: I am hoping that someone will create a simpler bootstrap module that is able to download a file of pure Python code and install it, perhaps by running its setup.py, assuming that it only depends on distutils (or other things previously installed). I will welcome such a module into the stdlib. I'm not sure a PEP is even needed, though interested parties are certainly welcome to write a PEP specifying the behavior first. With 2.6 and 3.0 slated for release in September, there should be enough time to get this done before then. Unfortunately, as I've already tried to explain, download a file ... and install it is not a sufficiently well-specified requirement to implement a robust tool. Even if it is not to support arbitrary existing distutils sources, there still needs to be a way to document precisely what the tool does and does not support installing, so that users can produce correct files for it to consume, register them properly with PyPI, etc. And as I said before (perhaps not very well) the distutils documentation has already proven to be inadequate as such a specification, both for users to create these files -- and even more important -- for programs to consume them. (For example, distutils source distribution tarball filenames are not unambiguously machine-parseable.) That's why I think some specific format (i.e. conventions) have to be defined for this to work, even if it's merely a set of documented restrictions on a distutils-based layout, file naming conventions, versioning, etc. In other words, you can't have your cake and eat it, too. If there's to be a bootstrap tool, you must bless *some* set of packaging conventions, including file naming, version parsing, and so on. Can we use setuptools' version parsing scheme to identify the latest stable version, for example? What about setuptools' filename component canonicalization and escaping rules? Frankly, I don't care what the conventions are, only that they be unambiguously defined and reasonably implementable for producers and consumers alike. I just want there to be *some* sort of robust, documented, standard installation bootstrap vector in the stdlib, so that setuptools, zc.buildout, and virtualenv don't have to maintain their own (or depend on setuptools maintaining its own). But not only have you rejected the *only* existing robust and well-documented conventions for automated processing of Python libraries, you say you have no time for this part of the thread when I ask what conventions you want to bless *instead*. So I'm at a bit of a loss for what we're supposed to do now. You're welcome to write a PEP as long as it is self-contained (at best referencing other accepted PEPs like the PyPI metadata PEPs). And the PEP better not be 2500 lines long. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
There seems to be a misunderstanding about what I am proposing we do instead. The boostrap installer should only be powerful enough to allow it to be used to install a real package manager like setuptools. Maybe my use of Django as an example was confusing; I didn't actually mean that it should be possible to support installing Django directly (although I don't understand all the issure that would make it impossible). Only very few people would care about writing a setup script that works with this bootstrap module; basically only package manager implementers. --Guido On Tue, Mar 18, 2008 at 3:45 PM, Stephen J. Turnbull [EMAIL PROTECTED] wrote: Guido van Rossum writes: I am hoping that someone will create a simpler bootstrap module FWIW (I've never tried to implement one of these things) I agree with Phillip. This is not possible in the sense you are advocating. Anything simpler is simply an invitation to an unending stream of issues, far more so than adopting eggs as a best current practice would. Better to Just Say No to an installer in the stdlib. I'm not sure a PEP is even needed This I simply don't understand. I *have* participated in bolting on features to such systems, and it's a mess. As features are added, existing users will demand that old packages install exactly as they ever did, except that sometimes (but not always!) they want them upgraded to put things in newly blessed places. Features are easy to add, since on the main thread of control installers are basically just a sequence of single commands, sometimes optional. python-dev has some pretty good controls that will help slow such trends. Nonetheless, PEP-less development of an installer system is scary. Installer complexity is a creeping horror, worse than kudzu.[1] Footnotes: [1] http://en.wikipedia.org/wiki/Kudzu#Invasive_species -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 03:43 PM 3/18/2008 -0500, Guido van Rossum wrote: Only very few people would care about writing a setup script that works with this bootstrap module; basically only package manager implementers. That's true today, sure, but as soon as it is widely available, others are sure to want to use it too. I just want a bright-line distinction between what is and isn't bootstrappable, rather than a murky region of maybe, if you're not doing anything too complicated. There seems to be a misunderstanding about what I am proposing we do instead. The boostrap installer should only be powerful enough to allow it to be used to install a real package manager like setuptools. Which is why PEP 365 proposed only downloading an archive to a cache directory, and optionally running something from it. It explicitly disavows installation of anything, since the downloaded archive wouldn't have been added to sys.path except for the duration of the bootstrap process, and no scripts were to be installed. (Indeed, apart from the methods it would have used to locate the archive on PyPI, and to determine what to run from inside it, there was nothing particularly egg-specific about the proposed bootstrapping process.) So, to fully egg-neutralize the bootstrapping approach, we need only know how to locate an appropriate archive, and how to determine what to run from it. For the latter, we could use the already-in-2.6 convention of running __main__ from a zipfile or directory. (Too bad distutils source distributions have an extra directory name embedded in them, so one can't just execute them directly. Otherwise, we could've just let people drop in a __main__.py next to setup.py. OTOH, maybe it would be enough to use setuptools' algorithm for finding setup.py to locate __main__.py, and I'm fairly sure *that* can be briefly expressed in the PEP.) The other open question is a naming convention and version detection, so that the bootstrap tool can identify which of the files listed on PyPI is suitable for its use. (Both with regard to the version selection, and file type.) However, if PyPI were to grow support for designating the appropriate files and/or versions in some other way, we wouldn't need a naming convention as such. Without one or the other, the bootstrap tool would have to grow a version parsing scheme of some type, and play guessing games with file extensions. (Which is one reason I limited PEP 365's scope to downloading eggs actually *uploaded* to PyPI, rather than arbitrary packages *linked* from PyPI.) So, if I had to propose something right now, I would be inclined to propose: * using setuptools' version parsing semantics for interpretation of alpha/beta/dev/etc. releases * having a bdist_bootstrap format that's essentially a bdist_dumb .zip file with the internal path prefixes stripped off, making it an importable .zip with a different file extension. (Or maybe just .pyboot.zip?) The filename convention would use setuptools' canonicalization and escaping of names and version numbers, to allow unambiguous machine parsing of the filename. A __main__ module would have to be present for the archive to be run, as opposed to just being downloaded to a temporary directory. * calling the bootstrap module 'bootstrap', as in 'python -m bootstrap projectname optionalversion'. The module would expose an API to allow it to be used programmatically as well as the command line, so that bootstrapped packages can use the bootstrap process to locate dependencies if they so desire. (Today's package management tools, at least, are all based on setuptools, so if it's not present they'll need to download that before beginning their own bootstrapping process.) Apart from keeping the PEP self-contained and short, is there anything in this that you think you would object to? (You may reserve the right, of course, to later not like something in the details of setuptools' version/filename rules, after I've put them into the PEP, or really, anything else. I'm just asking if there's anything that's obviously offensive at this point, before I spend time on a new PEP.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Sun, Mar 16, 2008 at 7:06 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: So, if the consensus is that it would be better to have a module that only does bootstrap installs of pure-Python eggs from PyPI, I'm totally fine with that. Let's just do this; it will avoid a protracted discussion of the merits of eggs, pkg_resources, and setuptools. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 08:48 AM 3/17/2008 -0500, Guido van Rossum wrote: On Sun, Mar 16, 2008 at 7:06 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: So, if the consensus is that it would be better to have a module that only does bootstrap installs of pure-Python eggs from PyPI, I'm totally fine with that. Let's just do this; it will avoid a protracted discussion of the merits of eggs, pkg_resources, and setuptools. Well, it might be replaced by a protracted discussion of how the module should work and what its API should be, but perhaps that would be a better one to have. :) So, the original proposal (from the previous thread about this) was that the module be named easy_install, and that it simply downloads setuptools and delegates to the real easy_install. That way, people can simply use python -m easy_install ..., without worrying about whether setuptools has been installed yet. IIRC, other package management tools such as zc.buildout and workingenv can then be installed using easy_install. Any objections? Should I revise the PEP? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Well, it might be replaced by a protracted discussion of how the module should work and what its API should be, but perhaps that would be a better one to have. :) Indeed, that's likely to happen :-) So, the original proposal (from the previous thread about this) was that the module be named easy_install, and that it simply downloads setuptools and delegates to the real easy_install. That way, people can simply use python -m easy_install ..., without worrying about whether setuptools has been installed yet. I thought the original proposal was to install a *binary* easy_install that takes that function. IIRC, other package management tools such as zc.buildout and workingenv can then be installed using easy_install. Any objections? Should I revise the PEP? I'm fine with the module, but would really like to see a command line utility in addition. This, of course, would raise the issue who owns the easy_install script name; ideally, the script would not have to be overwritten when setuptools gets installed. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 09:45 AM 3/17/2008 -0500, Martin v. Löwis wrote: Well, it might be replaced by a protracted discussion of how the module should work and what its API should be, but perhaps that would be a better one to have. :) Indeed, that's likely to happen :-) So, the original proposal (from the previous thread about this) was that the module be named easy_install, and that it simply downloads setuptools and delegates to the real easy_install. That way, people can simply use python -m easy_install ..., without worrying about whether setuptools has been installed yet. I thought the original proposal was to install a *binary* easy_install that takes that function. What do you mean by binary? I thought we were talking about a module. Do you mean a script to be installed alongside Python itself in e.g. /usr/bin? In the original discussion, it was a module to be added alongside pkg_resources, which would use pkg_resources to find and/or install setuptools. I also personally like the use of -m instead of a script because it makes it quite clear that this is a Python-specific installation tool, and *which* version of Python, as well, without having to have easy_install-2.5, easy_install-2.6, etc. IIRC, other package management tools such as zc.buildout and workingenv can then be installed using easy_install. Any objections? Should I revise the PEP? I'm fine with the module, but would really like to see a command line utility in addition. This, of course, would raise the issue who owns the easy_install script name; ideally, the script would not have to be overwritten when setuptools gets installed. It won't have to. The module will attempt to import the setuptools-supplied version of easy_install, and delegate to it if possible, before trying to do any download. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
I thought the original proposal was to install a *binary* easy_install that takes that function. What do you mean by binary? I thought we were talking about a module. Do you mean a script to be installed alongside Python itself in e.g. /usr/bin? Exactly so. In the original discussion, it was a module to be added alongside pkg_resources, which would use pkg_resources to find and/or install setuptools. I also personally like the use of -m instead of a script because it makes it quite clear that this is a Python-specific installation tool, and *which* version of Python, as well, without having to have easy_install-2.5, easy_install-2.6, etc. If that becomes the official interface to easy_install, that's fine with me. I'm worried about web instructions that tell people that there is an easy_install utility, so that people never find out the module actually exists. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
I don't think this should play games with scripts being overridden or whatever. If a bootstrap script is to be installed it should have a separate name. I'm not sure what the advantage is of a bootstrap script over python -m bootstrap_module ... though. The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools? That sounds like the bootstrap script would make setuptools the de-facto standard, which I'd like to avoid given the politics around this topic. I'd also like to avoid the specific name easy_install for any of this. That's a brand name (and a misleading one if you ask me, but that's politics again :-). --Guido On Mon, Mar 17, 2008 at 10:10 AM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 09:45 AM 3/17/2008 -0500, Martin v. Löwis wrote: Well, it might be replaced by a protracted discussion of how the module should work and what its API should be, but perhaps that would be a better one to have. :) Indeed, that's likely to happen :-) So, the original proposal (from the previous thread about this) was that the module be named easy_install, and that it simply downloads setuptools and delegates to the real easy_install. That way, people can simply use python -m easy_install ..., without worrying about whether setuptools has been installed yet. I thought the original proposal was to install a *binary* easy_install that takes that function. What do you mean by binary? I thought we were talking about a module. Do you mean a script to be installed alongside Python itself in e.g. /usr/bin? In the original discussion, it was a module to be added alongside pkg_resources, which would use pkg_resources to find and/or install setuptools. I also personally like the use of -m instead of a script because it makes it quite clear that this is a Python-specific installation tool, and *which* version of Python, as well, without having to have easy_install-2.5, easy_install-2.6, etc. IIRC, other package management tools such as zc.buildout and workingenv can then be installed using easy_install. Any objections? Should I revise the PEP? I'm fine with the module, but would really like to see a command line utility in addition. This, of course, would raise the issue who owns the easy_install script name; ideally, the script would not have to be overwritten when setuptools gets installed. It won't have to. The module will attempt to import the setuptools-supplied version of easy_install, and delegate to it if possible, before trying to do any download. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote: I don't think this should play games with scripts being overridden or whatever. If a bootstrap script is to be installed it should have a separate name. I'm not sure what the advantage is of a bootstrap script over python -m bootstrap_module ... though. And -m also makes explicit: 1. that it's a Python-specific tool 2. which Python version it will apply to The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools? Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :) I'd also like to avoid the specific name easy_install for any of this. That's a brand name (and a misleading one if you ask me, but that's politics again :-). Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mon, Mar 17, 2008 at 11:12 AM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote: I don't think this should play games with scripts being overridden or whatever. If a bootstrap script is to be installed it should have a separate name. I'm not sure what the advantage is of a bootstrap script over python -m bootstrap_module ... though. And -m also makes explicit: 1. that it's a Python-specific tool 2. which Python version it will apply to Right! The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools? Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :) On how much of pkg_resources do they depend? Or is that an unanswerable question? I'd also like to avoid the specific name easy_install for any of this. That's a brand name (and a misleading one if you ask me, but that's politics again :-). Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.) Perhaps it can be called package_bootstrap? I don't know enough about the required functionality to propose an API, but here goes anyway. Would be reasonable for it to have a command line that allows you to specify a package name, optionally a version string, and (very) optionally a server to contact (specified as an URL?). It should default to the highest non-experimental version that's applicable to the current Python version (assuming there's an easy way to determine this; I don't want it to try and download different versions until one works). It should not handle any kind of dependency management or package management administration. It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs. If the contents of the tar/zip file has a toplevel directory with version numbers in its name (e.g. Django-0.96.1) it should skip that and just use the subdirectory whose name is the pure package name (e.g. django). Does this make sense? I'm happy to take push-back, this is just something I cooked up off the top of my head based on my experience with manually installing packages. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Guido van Rossum wrote: It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs. Do you mean existing eggs or does that include the (potential .egg) package that is being installed? If I understood correctly, this bootstrap module currently supports installing eggs (although I'm not sure how they are supposed to work without the current way of keeping a .pth file). Is it *wanted* that eggs are being supported by core Python? Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mon, Mar 17, 2008 at 11:55 AM, Stefan Behnel [EMAIL PROTECTED] wrote: Guido van Rossum wrote: It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs. Do you mean existing eggs or does that include the (potential .egg) package that is being installed? The latter. If I understood correctly, this bootstrap module currently supports installing eggs (although I'm not sure how they are supposed to work without the current way of keeping a .pth file). My proposal is that the boostrap module only be able to install non-egg archives. Is it *wanted* that eggs are being supported by core Python? No. There will be no egg support in the standard library. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: There will be no egg support in the standard library. Are there any qualifications on that statement, or is this in the same category as from __future__ import braces? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 17/03/2008, Guido van Rossum [EMAIL PROTECTED] wrote: On Mon, Mar 17, 2008 at 11:55 AM, Stefan Behnel [EMAIL PROTECTED] wrote: Is it *wanted* that eggs are being supported by core Python? No. There will be no egg support in the standard library. This bothers me somewhat. At a certain level, all that egg files are is zip files with a different extension - and the zipimport module and PEP 302 certainly do support them. There is a lot more conceptual baggage associated with the egg brand name, but unless that extra is clearly specified, a statement like there will be no egg support doesn't mean much. My view on PEP 365 is that it offers a standard way of doing from pkg_resources import resource_string foo_config = resource_string(__name__, 'foo.conf') which is basically a version of foo_config = open(os.path.join(os.path.dirname(__file__),'foo.conf').read() which also supports PEP 302 importers such as zipimport. This has nothing to do with eggs, and everything to do with solidifying the support for cleanly handling PEP 302 importers. For me, that would be far more useful that a package downloadinstaller (as I'm on Windows, I tend to use bdist_wininst installers, and a bootstrap module which gave no uninstall capability would suck for me). The sticking point here, is deciding what parts of pkg_resources are OK, and which constitute egg support. It may not be possible to come to a clear understanding of this. But I remain -1 on any module that just does download and install, with no uninstall capability. I don't like *eggs* for precisely this reason! Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: There will be no egg support in the standard library. Are there any qualifications on that statement, or is this in the same category as from __future__ import braces? IIUC eggs are a method of package management that includes support for dependencies, multiple versions, and C extensions in zip files, as well as conventions for naming these and for encoding metadata (e.g. how to find out the version or the dependencies). This whole set of conventions is IMO too much to include into the stdlib ATM -- if only because it has proved controversial in the past. Maybe a few years from now it will be no longer controversial and then my objections will disappear. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote: On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: There will be no egg support in the standard library. Are there any qualifications on that statement, or is this in the same category as from __future__ import braces? IIUC eggs are a method of package management that includes support for dependencies, multiple versions, and C extensions in zip files, as well as conventions for naming these and for encoding metadata (e.g. how to find out the version or the dependencies). This whole set of conventions is IMO too much to include into the stdlib ATM -- if only because it has proved controversial in the past. Maybe a few years from now it will be no longer controversial and then my objections will disappear. So, does this mean that the bootstrap tool must not use eggs? That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs... So, it might be simpler all around to just clear up the controversy. To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs. Note: I'm specifically segregating egg support from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5. However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.) That leaves MAL, whose objections to PEP 365 centered on the API (he said he was +1 on the concepts being added to the stdlib, -1 on adding the module in its current state). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new egglib module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.) I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :) So, was there anyone else you were counting towards controversy? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general. In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mon, Mar 17, 2008 at 1:45 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 12:59 PM 3/17/2008 -0500, Guido van Rossum wrote: On Mon, Mar 17, 2008 at 12:45 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 12:17 PM 3/17/2008 -0500, Guido van Rossum wrote: There will be no egg support in the standard library. Are there any qualifications on that statement, or is this in the same category as from __future__ import braces? IIUC eggs are a method of package management that includes support for dependencies, multiple versions, and C extensions in zip files, as well as conventions for naming these and for encoding metadata (e.g. how to find out the version or the dependencies). This whole set of conventions is IMO too much to include into the stdlib ATM -- if only because it has proved controversial in the past. Maybe a few years from now it will be no longer controversial and then my objections will disappear. So, does this mean that the bootstrap tool must not use eggs? Correct. That seems a little bit odd, in that setuptools will at least need its .egg-info directory to get installed, and all of the people who'll be using this initially will be using it precisely in order to have support for eggs... I'm okay if setuptools, once it's been installed, runs some setup code that creates the .egg-info directory and whatever else. This means I'm also okay with the bootstrap module finding and invoking that setup code. But I'm *not* okay with building any kind of egg management into the bootstrap module. The bootstrap module must be be neutral w.r.t. the package management style. So, it might be simpler all around to just clear up the controversy. To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs. You can add my name to the list. I've heard plenty of people speak highly of eggs, but I've *also* heard from plenty of people (besides MAL and MvL) who have serious difficulties with the concept of eggs. I have certainly personally encountered plenty of situations where I wasn't able to complete an egg-based install because some dependency was broken (e.g. not available for the Python version I was using). Note: I'm specifically segregating egg support from the topic of including setuptools or easy_install in the stdlib directly. There are many legitimate reservations and open questions about the latter, including availability of volunteer support, choice of defaults, whether to replace distutils with setuptools, etc. etc. I recognize and respect the validity of those issues, which is precisely why I withdrew setuptools from inclusion in Python 2.5. However, regarding support for eggs, my understanding is that there were only two objections to eggs -- even at the time of the 2.5 setuptools discussions. And even though MvL objects to the idea of eggs in *principle*, I didn't read his recent posts as objecting to having the bootstrap tool download and install eggs in *practice*. (Although I hope he will clarify that stance one way or the other.) You can do it in two stages. The bootstrap can download and install egg support, even though the bootstrap itself knows nothing about eggs. Then another bootstrap can download and install eggs. That leaves MAL, whose objections to PEP 365 centered on the API (he said he was +1 on the concepts being added to the stdlib, -1 on adding the module in its current state). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new egglib module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.) Well, you've heard my position now. I would guess, however, that MAL's issues with the pkg_resources API would not apply to a bootstrap module whose sole purpose was to download eggs and put them on sys.path. Or, perhaps he would object *more*, I don't know. We could certainly ask him, though. :) No need. I object to this myself. So, was there anyone else you were counting towards controversy? The only other person I recall objecting to setuptools in any way on Python-Dev was effbot, and IIUC his objections were practical/administrative re: supporting easy_install and setuptools, not to the idea of .egg support in general. In summary, I think the controversy on Python-Dev regarding .egg support has actually been over for some time now. Not really. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Guido van Rossum wrote: On Mon, Mar 17, 2008 at 11:12 AM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote: I don't think this should play games with scripts being overridden or whatever. If a bootstrap script is to be installed it should have a separate name. I'm not sure what the advantage is of a bootstrap script over python -m bootstrap_module ... though. And -m also makes explicit: 1. that it's a Python-specific tool 2. which Python version it will apply to Right! The PEP suggests that other package managers also benefit. How do they benefit if the bootstrap script installs setuptools? Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :) On how much of pkg_resources do they depend? Or is that an unanswerable question? I'd also like to avoid the specific name easy_install for any of this. That's a brand name (and a misleading one if you ask me, but that's politics again :-). Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.) Perhaps it can be called package_bootstrap? I don't know enough about the required functionality to propose an API, but here goes anyway. Would be reasonable for it to have a command line that allows you to specify a package name, optionally a version string, and (very) optionally a server to contact (specified as an URL?). It should default to the highest non-experimental version that's applicable to the current Python version (assuming there's an easy way to determine this; I don't want it to try and download different versions until one works). It should not handle any kind of dependency management or package management administration. It should be able to download a Python-only module or package and install it into site-packages (or perhaps elsewhere if so directed via another optional command line flag). It should support zip, tar and tar.gz/tgz files (and perhaps tar.bz2). It should simply unpack the zip/tar file using the zip or tar modules, and extract the package/module into site-packages in such a way that it can be imported directly without messing with sys.path. It should not mess with .pth files, setup.py scripts, or eggs. If the contents of the tar/zip file has a toplevel directory with version numbers in its name (e.g. Django-0.96.1) it should skip that and just use the subdirectory whose name is the pure package name (e.g. django). Does this make sense? I'm happy to take push-back, this is just something I cooked up off the top of my head based on my experience with manually installing packages. I would prefer to see it just: - Find a source distribution (the variants you specified) on the given server which matches the supplied version string, using the same semantics as the current 'pkg_resources' constraints. - Unpack the source distribution to a temp directory. - cd into that directory and use sys.executable to invoke 'setup.py install'. Any extra command-line arguments beyond those used to find the source distribution would be passed on to the 'install' command, which would allow alternate locations, etc. That makes the installation as much like a manual one as possible, and means that existing pacakges will be installable whether or not they know about setuptools, etc. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH3sDr+gerLs4ltQ4RAhjWAKCbFP87mJN4UnVt0pzDs81JovVpoACdGI7A tohpRJnXah0/QnyQeYiqoYY= =9Cif -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
At 01:59 PM 3/17/2008 -0500, Guido van Rossum wrote: I have certainly personally encountered plenty of situations where I wasn't able to complete an egg-based install because some dependency was broken (e.g. not available for the Python version I was using). That's odd -- setuptools-based installs should be able to find and install packages from source. I have noticed a recent phenomenon where new developers upload *only* an egg to PyPI, without the source, but that's usually short-lived until someone points it out to them. Do you happen to know what packages you had this problem with? I'm okay if setuptools, once it's been installed, runs some setup code that creates the .egg-info directory and whatever else. This means I'm also okay with the bootstrap module finding and invoking that setup code. But I'm *not* okay with building any kind of egg management into the bootstrap module. The bootstrap module must be be neutral w.r.t. the package management style. Ok, well then we'll have to invent a new kind of binary package, whose name isn't 'egg'. Supporting distutils source packages is almost certainly a non-starter, if you want to avoid bringing the rest of setuptools into play. The only way to correctly determine what a source package contains is to run its setup script... and running unboxed setup scripts isn't safe because there are people who hardcode paths (or more precisely, use bad ways of computing them) in their setup scripts. I'm not saying the tool needs to guard against *malicious* scripts, just badly-written ones. (Setuptools does this with its sandboxing module, when running source packages' setup scripts.) So, if source is out, then some binary format is needed, which means defining the conventions for said format... i.e. eggs lite or egg substitutes. :) So, it might be simpler all around to just clear up the controversy. To the best of my recollection, only MAL and MvL have ever objected on Python-Dev to the idea of supporting eggs. You can add my name to the list. I've heard plenty of people speak highly of eggs, but I've *also* heard from plenty of people (besides MAL and MvL) who have serious difficulties with the concept of eggs. I did say on Python-Dev, and you implied that it was not controversial with you, except for the maintenance-related concerns. I'm not fighting about this, but I would rather you were straight-up with your objections rather than deferring it to a controversy that might go away in a few years. That way, I could at least attempt to do something about the concerns. OTOH, if your objections were non-specific and likely to stay that way, then I could have at least not wasted your time with any of this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On 17/03/2008, Phillip J. Eby [EMAIL PROTECTED] wrote: That leaves MAL, whose objections to PEP 365 centered on the API (he said he was +1 on the concepts being added to the stdlib, -1 on adding the module in its current state). Among other concerns, he wanted pkg_resources to be split into pkgutil and a new egglib module. I don't have a problem with this in principle, if there were a pkg_resources module that reconstituted the merged API. (But there are some practical problems with that approach, such as trying to split namespace package support between two theoretically-unrelated modules.) I would personally like to see such a separation. As one of the authors of PEP 302 (well, the documentation - Just did all of the implementation) I have an interest in strengthening the standard library's support for transparent use of PEP 302 importers. To that end, I would like to see such functions as pkg_resources.resource_string() standardised. That covers the pkgutil aspect of pkg_resources. The egglib side of things is where the controversy lies, IMHO. I have a (somewhat unfocussed) dislike of eggs, largely because of the lack of a package management tool to handle them. I can live with them or without them, and it doesn't bother me if others use them, but the thing that really, really bothers me is that the controversy (and yes, there is such) over eggs is hampering the adoption of the pkgutil side of pkg_resources. So from me, there's a strong +1 for the split of pkg_resources into additions to the existing pkgutil module, and an independent egglib. You say there are practical problems to doing this. OK, but could we not have a discussion on how to resolve those issues as they come up? Could the split not be initially into 3 parts - pkgutil (eg, resource_string), egglib, and difficult? Then there would be something concrete to discuss and resolve. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
After reading all this, I really don't believe that adding egg support to the stdlib at this time is the right thing to do. I am therefore rejecting the PEP. I am hoping that someone will create a simpler bootstrap module that is able to download a file of pure Python code and install it, perhaps by running its setup.py, assuming that it only depends on distutils (or other things previously installed). I will welcome such a module into the stdlib. I'm not sure a PEP is even needed, though interested parties are certainly welcome to write a PEP specifying the behavior first. With 2.6 and 3.0 slated for release in September, there should be enough time to get this done before then. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
Quick summary of the below: I'm definitely fine with doing a simpler, pure-bootstrap module, if there's some consensus on what should go in it. I just wish we could've had this discussion last year, when OSAF was still able to fund the work... ;-) At 06:13 PM 3/16/2008 -0500, Guido van Rossum wrote: Phillip asked me to give an opinion on his pkg_resources PEP. While the PEP is short and sweet, the pkg_resources module itself is huge (1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67 names in total according to __all__). And pkg_resources.txt is another 1700 lines of documentation. I find that hard to swallow. Is there anyone besides Phillip who can claim he understands this module? Bob Ippolito actually wrote the very first version of pkg_resources. Others, such as Philip Jenvey of the Jython project, have provided patches. From previous discussions on the distutils-sig, I know that Jim Fulton has in-depth knowledge of both pkg_resources and easy_install. Of course, that's not the same as any of these guys volunteering to be maintainers. :) If its inclusion is really meant just as a bootstrap to simplify installing other package management solutions, as the PEP claims, I would prefer to see something with a much smaller footprint. Actually, the PEP says: pkg_resources is a module used to find and manage Python package/version dependencies and access bundled files and resources, including those inside of zipped .egg files. Currently, pkg_resources is only available through installing the entire setuptools distribution, but it does not depend on any other part of setuptools; in effect, it comprises the entire runtime support library for Python Eggs, and is independently useful. This kind of glosses over the part where this is also for runtime support of projects that use eggs. Which, these days, is, well, almost any large Python project, from Chandler to Enthought to Zope. Surely there is no need for example to have support for C extensions inside zip files *as part of the bootstrap module*? It's a runtime; the PEP actually merely proposes that a further addition to be made to support bootstrapping, *also*. Otherwise, the PEP would be even shorter. :) The reason I proposed it this way was for simplicity -- and politics. Currently, people using setuptools in their setup.py have to include a similar bootstrap module to download setuptools if it's not available, and pkg_resources already has version checking logic and everything needed to find dependencies and download them. (Plus, I figured it'd be easier to just use what was already there and stable, rather than creating something different.) That was the simplicity part. The politics part was that: 1. I thought it would be less controversial to include the runtime for eggs than to include something that's just a bootstrapper for setuptools. However, MvL surprised me by actually being in *favor* of including a setuptools bootstrapper. 2. I thought that it would have broader acceptance if it was oriented towards bootstrapping *any* package, not just setuptools. So, if the consensus is that it would be better to have a module that only does bootstrap installs of pure-Python eggs from PyPI, I'm totally fine with that. Unless I find someone besides Phillip who is interested in having this included and is willing to help maintain it, I don't really think it would be wise to accept this into the standard library. Phillip, in the PEP you mention that there are several other package management tools that also like to use pkg_resources. Maybe you can get some folks from those tools to speak up and explain what pkg_resources means to them, and maybe even volunteer to co-own it once it's in the standard library? The distutils-sig is the de facto place for discussions regarding those tools, so I've cc'd this there. Hopefully, one or more volunteers will step up if they want this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 365 (Adding the pkg_resources module)
On Mar 16, 2008, at 8:06 PM, Phillip J. Eby wrote: Quick summary of the below: I'm definitely fine with doing a simpler, pure-bootstrap module, if there's some consensus on what should go in it. I just wish we could've had this discussion last year, when OSAF was still able to fund the work... ;-) At 06:13 PM 3/16/2008 -0500, Guido van Rossum wrote: Phillip asked me to give an opinion on his pkg_resources PEP. While the PEP is short and sweet, the pkg_resources module itself is huge (1800 non-blank lines; 16 classes plus 5 exceptions; it exports 67 names in total according to __all__). And pkg_resources.txt is another 1700 lines of documentation. I find that hard to swallow. Is there anyone besides Phillip who can claim he understands this module? Bob Ippolito actually wrote the very first version of pkg_resources. Others, such as Philip Jenvey of the Jython project, have provided patches. From previous discussions on the distutils-sig, I know that Jim Fulton has in-depth knowledge of both pkg_resources and easy_install. Of course, that's not the same as any of these guys volunteering to be maintainers. :) If its inclusion is really meant just as a bootstrap to simplify installing other package management solutions, as the PEP claims, I would prefer to see something with a much smaller footprint. Actually, the PEP says: pkg_resources is a module used to find and manage Python package/version dependencies and access bundled files and resources, including those inside of zipped .egg files. Currently, pkg_resources is only available through installing the entire setuptools distribution, but it does not depend on any other part of setuptools; in effect, it comprises the entire runtime support library for Python Eggs, and is independently useful. This kind of glosses over the part where this is also for runtime support of projects that use eggs. Which, these days, is, well, almost any large Python project, from Chandler to Enthought to Zope. Surely there is no need for example to have support for C extensions inside zip files *as part of the bootstrap module*? It's a runtime; the PEP actually merely proposes that a further addition to be made to support bootstrapping, *also*. Otherwise, the PEP would be even shorter. :) The reason I proposed it this way was for simplicity -- and politics. Currently, people using setuptools in their setup.py have to include a similar bootstrap module to download setuptools if it's not available, and pkg_resources already has version checking logic and everything needed to find dependencies and download them. (Plus, I figured it'd be easier to just use what was already there and stable, rather than creating something different.) That was the simplicity part. The politics part was that: 1. I thought it would be less controversial to include the runtime for eggs than to include something that's just a bootstrapper for setuptools. However, MvL surprised me by actually being in *favor* of including a setuptools bootstrapper. 2. I thought that it would have broader acceptance if it was oriented towards bootstrapping *any* package, not just setuptools. So, if the consensus is that it would be better to have a module that only does bootstrap installs of pure-Python eggs from PyPI, I'm totally fine with that. Unless I find someone besides Phillip who is interested in having this included and is willing to help maintain it, I don't really think it would be wise to accept this into the standard library. Phillip, in the PEP you mention that there are several other package management tools that also like to use pkg_resources. Maybe you can get some folks from those tools to speak up and explain what pkg_resources means to them, and maybe even volunteer to co-own it once it's in the standard library? The distutils-sig is the de facto place for discussions regarding those tools, so I've cc'd this there. Hopefully, one or more volunteers will step up if they want this. I'd like to see it in and am willing to help maintain it. Especially if it only does the bootstrap installs of pure-Python egg bit. I've dug into pkg_resource some, but can't claim to understand it all. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
