Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Phillip J. Eby wrote: I assumed that it would be more controversial to merge setuptools into distutils, than to provide it as an enhanced alternative. It is this assumption in setuptools that seems to have guided many design decisions: that it is completely unacceptable to change implementation details, because somebody might rely on them. I firmly believe that this position is misguided, and that decisions resulting from it should be corrected (over time, of course). Beautiful is better than ugly: if you believe that the distutils code is wrong in some respect, then change it. Of course, things are more complicated in this approach: you have to actively consider the likelyhood of breakage, and you have to a) clearly document the potential for breakage: the more likely the breakage, the more visible the documentation should be b) try to come up with recommendations for users should the any code actually break: users then want to know how they should change their code so that it works with previous versions of Python still. c) ask for consent in advance to making a potentially-breaking change. 1. How to activate or deactivate backward compatibility for packages or people relying on intimate details of current distutils behaviors See above: on a case-by-case basis. 2. Maintaining the existing version of setuptools to work with Python 2.3 and 2.4, which would not have this integration with the distutils. One way would be to make another distutils release, and require setuptools users to install this distutils release as a prerequisite. Another solution would be to fork setuptools, in a pre-2.5 branch and a post-2.5 branch. Over time, the pre-2.5 branch could be abandoned. A third solution likely exists on a case-by-case basis: conditionalize code inside setuptools, depending on Python version (or other criteria). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Martin c) ask for consent in advance to making a potentially-breaking Martinchange. Doesn't that potentially extend the release time for an enhanced distutils across multiple Python releases? With both distutils and setuptools available simultaneously, setuptools can be designed and implemented to suit the needs of its constituency while distutils remains avilable and compatible for those people using it. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib
Phillip J. Eby [EMAIL PROTECTED] wrote: If so, can't we have some kind of versioning system? We do: import setuptools. We could perhaps rename it to import distutils2 if you prefer, but it would mean essentially the same thing. :) I believe the naming is important, though. I'd rather it be called distutils2, or from distutils.core import setup2 or something like that. setuptools *is* a new version of distutils, so it shouldn't have a different name. Then, about new commands. Why should I need to do import distutils2 to do, eg, setup.py develop? This doesn't break backward compatibility. Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib
At 08:11 AM 4/19/2006 +0200, Giovanni Bajo wrote: Then, about new commands. Why should I need to do import distutils2 to do, eg, setup.py develop? This doesn't break backward compatibility. The develop command uses the egg_info command. egg_info uses the setuptools enhanced MANIFEST scheme. Both make use of extended setup() arguments, and the entry points feature that allows distutils plugins to co-operate. Develop also uses easy_install... and so on. I'm not saying it would be impossible to merge this stuff into the distutils, just that it's not a trivial undertaking. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 -python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Neal Norwitz wrote: I was also working under the assumption that people would complain if they didn't like something. What do people think should happen for the Possible features section? Should I ask if there are any objections to each item? some discussion on python-dev for each non-trivial item should be required. larger items may need discussion + grace period + discussion before they've checked in. the increasing amount of but I've discussed this in some other forum worries me. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Phillip J. Eby wrote: I was surprised that MAL didn't comment *then*, actually, and mistakenly thought it meant that our last discussion on the distutils-sig (and my attempts to deal with the problems) had been successful. Between that and MvL's mild response to the explicit discussion of supporting setuptools, I thought their votes had effectively moved from -1 to -0. Off-list discussion with Fredrik suggested that he too had shifted from -1 to -0, I'm +1 on adding stuff to distutils (and other install tools) to make it *easier* for setuptools (and other install tools) to make a good job. I'm -1 on adding tools to the core that changes the structure of an installed Python system, without a full PEP process. If nobody can point to (or produce) a technical document that, in full detail, describes the mechanisms *used* by setuptools, including what files it creates, what the files contain, how they are used during import, how non-setuptools code can manipulate (or at least in- spect) the data, etc, setuptools should *not* go into 2.5. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 08:38 AM 4/19/2006 +0200, Fredrik Lundh wrote: I'm -1 on adding tools to the core that changes the structure of an installed Python system, without a full PEP process. If nobody can point to (or produce) a technical document that, in full detail, describes the mechanisms *used* by setuptools, including what files it creates, what the files contain, how they are used during import, how non-setuptools code can manipulate (or at least in- spect) the data, etc, setuptools should *not* go into 2.5. And that is a mostly-specific, very fair, and completely reasonable objection. And I think a significant portion of it is answered by the existing documentation, at least with respect to the runtime. The pkg_resources API module includes all of the discovery, dependency resolution and introspection other facilities used by setuptools, and it does not depend on the rest of setuptools, which is directed primarily at building and installing eggs. A number of users have written simple Python scripts using the documented API in order to list installed packages and that sort of thing. The current API reference documentation is available at http://peak.telecommunity.com/DevCenter/PkgResources .) I do think that changes the structure of an installed Python system is rather vague, though. Python supports .pth files, for example, so is writing to a .pth file changing the structure of an installed Python system? Python supports installing modules in directories on PYTHONPATH or specifying zip files on it, so are these operations changing structure? I ask not to argue, but to make sure I know what else it is that you want documented, so I can be specific about what is and isn't documented. My vague guess at the moment is that the only things that setuptools does which could be considered changing the structure of an installed Python system are: 1. If installing to a PYTHONPATH directory (instead of site-packages), it adds a special 'site.py' file so that .pth files are processed in PYTHONPATH directories. (Python does not normally process .pth files on PYTHONPATH, but this is necessary to support dynamic installation of packages without modifying PYTHONPATH itself. 'site' is hooked instead of 'sitecustomize' to avoid interfering with a user-defined 'sitecustomize'.) 2. If installing a package in compatibility mode (aka --single-version-externally-managed, or if --root was specified), it adds an .egg-info directory to hold setuptools-specific metadata. This will be named something like 'FooBar-1.0.egg-info', and placed in the target directory alongside the installed package. So if the package were installed as Lib/foobar, the egg-info would be stored in Lib/FooBar-1.0.egg-info/. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
On 18-apr-2006, at 23:10, Phillip J. Eby wrote: There aren't all that many things that are wrong in setuptools, but some of them are essential: * setuptools should not monkey patch distutils on import Please propose an alternative, or better still, produce a patch. Be sure that it still allows distutils extensions like py2exe to work. The only real alternative to monkeypatching that I'm aware of for that issue is to actually merge setuptools with the distutils, but my guess is that you'll like that even less. :) Have you considered such a merger? It's rather odd to include a package in the stdlib that monkeypatches another part of the stdlib. For the record: I like setuptools and eggs. Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 08:33 PM 4/19/2006 +0200, Ronald Oussoren wrote: Have you considered such a merger? It's rather odd to include a package in the stdlib that monkeypatches another part of the stdlib. I assumed that it would be more controversial to merge setuptools into distutils, than to provide it as an enhanced alternative. Also, there are two practical issues: 1. How to activate or deactivate backward compatibility for packages or people relying on intimate details of current distutils behaviors 2. Maintaining the existing version of setuptools to work with Python 2.3 and 2.4, which would not have this integration with the distutils. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Phillip J. Eby wrote: a technical document that, in full detail, describes the mechanisms *used* by setuptools, including what files it creates, what the files contain, how they are used during import, how non-setuptools code can manipulate (or at least inpect) the data, etc, setuptools should *not* go into 2.5. And that is a mostly-specific, very fair, and completely reasonable objection. And I think a significant portion of it is answered by the existing documentation, at least with respect to the runtime. The pkg_resources API module includes all of the discovery, dependency resolution and introspection other facilities used by setuptools, and it does not depend on the rest of setuptools, which is directed primarily at building and installing eggs. A number of users have written simple Python scripts using the documented API in order to list installed packages and that sort of thing. The current API reference documentation is available at http://peak.telecommunity.com/DevCenter/PkgResources .) What I want is a PEP that explains the things that are manipulated by this API. What things are stored where, what do they contain, and how are they are used by the various tools ? The API in itself isn't very interesting; I'm interested in the underlying design. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 11:36 PM 4/19/2006 +0200, Fredrik Lundh wrote: Phillip J. Eby wrote: a technical document that, in full detail, describes the mechanisms *used* by setuptools, including what files it creates, what the files contain, how they are used during import, how non-setuptools code can manipulate (or at least inpect) the data, etc, setuptools should *not* go into 2.5. And that is a mostly-specific, very fair, and completely reasonable objection. And I think a significant portion of it is answered by the existing documentation, at least with respect to the runtime. The pkg_resources API module includes all of the discovery, dependency resolution and introspection other facilities used by setuptools, and it does not depend on the rest of setuptools, which is directed primarily at building and installing eggs. A number of users have written simple Python scripts using the documented API in order to list installed packages and that sort of thing. The current API reference documentation is available at http://peak.telecommunity.com/DevCenter/PkgResources .) What I want is a PEP that explains the things that are manipulated by this API. What things are stored where, what do they contain, and how are they are used by the various tools ? The API in itself isn't very interesting; I'm interested in the underlying design. But the API *documentation* is where you'll find: * The external formats understood for version numbers and version requirements * The escaping and comparison rules for project names and version numbers * The external formats used for dependency listings * The external formats used for declaring plugins * The conceptual overview for the design, including terminology and the design concepts exposed by the objects and operations * A link to the original design overview, which I wrote with a mind to it becoming part of a PEP someday: http://mail.python.org/pipermail/distutils-sig/2005-June/004652.html And probably many more things that I don't recall off the top of my head. If what you want is an implementer's guide to which actual files contain what data in which of the above formats, I'll admit there isn't such a roadmap. I could probably write one up by tomorrow, since it would mostly be a list of here are the filenames, this is what format in the existing documentation is used for this file, and here's what the contents are used for, and here are some pointers to relevant parts of the user and API docs that you should read in order to understand what this is for. However, if you don't read at least the API docs, you're definitely going to be missing most of the design picture. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Phillip J. Eby wrote: At 10:55 AM 4/18/2006 +0200, M.-A. Lemburg wrote: Phillip.eby wrote: Author: phillip.eby Date: Tue Apr 18 02:59:55 2006 New Revision: 45510 Modified: python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py Log: Second phase of refactoring for runpy, pkgutil, pydoc, and setuptools to share common PEP 302 support code, as described here: http://mail.python.org/pipermail/python-dev/2006-April/063724.html Shouldn't this new module be named pkglib to be in line with the naming scheme used for all the other utility modules, e.g. httplib, imaplib, poplib, etc. ? It's not a new module; it was previously a module with only one function in it, introduced in Python 2.3. If it were a new module, I'd have inquired about a name for it first. Didn't realize that. Too bad the time machine didn't work on this one :-/ pydoc now supports PEP 302 importers, by way of utility functions in pkgutil, such as 'walk_packages()'. It will properly document modules that are in zip files, and is backward compatible to Python 2.3 (setuptools installs for Python 2.5 will bundle it so pydoc doesn't break when used with eggs.) Are you saying that the installation of setuptools in Python 2.3 and 2.4 will then overwrite the standard pydoc included with those versions ? Yes. As far as I can tell, there were no API changes to pydoc during this time, so this is effectively a hot fix. Why should a 3rd party extension be hot-fixing the standard Python distribution ? This hot-fixing doesn't apply to setuptools system packages built with --root or --single-version-externally-managed, however, so OS vendors who build packages that wrap setuptools will have to make an explicit decision whether to also apply any fixes. If they do not, an end-user can of course install setuptools in their local PYTHONPATH and the hotfix will take effect. What does setuptools have to do with pydoc ? Why should a user installing setuptools assume that some arbitrary stdlib modules get (effectively) replaced by installing setuptools ? If you want to provide a hot fix for Python 2.3 and 2.4, you should make this a separate install, so that users are aware that their Python distribution is about to get modified in ways that have nothing to do with setuptools. I bothered by the fact that installing setuptools actually changes the standard Python installation by either overriding stdlib modules or monkey-patching them at setuptools import time. Please feel free to propose alternative solutions that will still ensure that setuptools just works for end-users. Both this and the pydoc hotfix are practicality beats purity issues. Not really. I'd consider them design flaws. distutils is built to be extended without having to monkey-patch it, e.g. you can easily override commands with your own variants by supplying them via the cmdclass and distclass keyword arguments to setup(). By monkey patching distutils during import of setuptools, you effectively *change* distutils at run-time and not only for the application space that you implement in setuptools, but for all the rest of the application. If an application wants to use setuptools for e.g. plugin management, then standard distutils features will get replaced by setuptools implementations which are not compatible to the standard distutils commands, effectively making it impossible to access the original versions. Monkey patching is only a last resort in case nothing else works. In this case, it's not needed, since distutils provides the interfaces needed to extend its command classes via the setup() call. See e.g. mxSetup.py in the eGenix extensions for an example of how effective the distutils design can be put to use without having to introduce lots of unwanted side-effects. Add setuptools to the stdlib ? I'm still missing the PEP for this along with the needed discussion touching among other things, the change of the distutils standard python setup.py install to install an egg instead of a site package. Setuptools in the stdlib simply means that people wanting to use it can import it. It does not affect programs that do not import it. It also means that python -m easy_install is available without having to first download ez_setup.py. Doesn't really seem like much of an argument for the addition... the above is true for any 3rd party module. As for discussion, Guido originally brought up the question here a few months ago, and it's been listed in PEP 356 for a while. I've also posted things related to the inclusion both here and in distutils-sig. I know, but the discussions haven't really helped much in getting the setuptools design compatible with standard distutils. Unless that's being put in place, I'm -1 on the addition, due to the invasive nature of setuptools and its various side-effects on systems management. Note that it only takes one single module in an application doing import
Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: Why should a 3rd party extension be hot-fixing the standard Python distribution ? Because setuptools installs things in zip files, and older versions of pydoc don't work for packages zip files. If you want to provide a hot fix for Python 2.3 and 2.4, you should make this a separate install, so that users are aware that their Python distribution is about to get modified in ways that have nothing to do with setuptools. Their Python distribution is not modified -- new modules are merely placed on sys.path ahead of the stdlib. (Which, I might add, is a perfectly normal process in Python -- nothing stops users from installing their own version of pydoc or any other module via PYTHONPATH.) Note also that uninstalling setuptools by removing the .egg file or directory will effectively remove the hot fix, since the modules live in the .egg, not in the stdlib. If an application wants to use setuptools for e.g. plugin management, then standard distutils features will get replaced by setuptools implementations which are not compatible to the standard distutils commands, effectively making it impossible to access the original versions. Please do a little research before you spread FUD. The 'pkg_resources' module is used for runtime plugin management, and it does not monkeypatch anything. Monkey patching is only a last resort in case nothing else works. In this case, it's not needed, since distutils provides the interfaces needed to extend its command classes via the setup() call. The monkeypatching is there so that the easy_install command can build eggs for packages that use the distutils. It's also there so that other distutils extensions that monkeypatch distutils (and there are a few of them out there) will have a better chance of working with setuptools. I originally took a minimally-invasive approach to setuptools-distutils interaction, but it was simply not possible to provide a high-degree of backward and sideward compatibility without it. In fact, I seem to recall finding some behaviors in some versions of distutils that can't be modified without monkeypatching, although the details are escaping me at this moment. As for discussion, Guido originally brought up the question here a few months ago, and it's been listed in PEP 356 for a while. I've also posted things related to the inclusion both here and in distutils-sig. I know, but the discussions haven't really helped much in getting the setuptools design compatible with standard distutils. That's because the job was already done. :) The setuptools design bends over backwards to be compatible with Python 2.3 and 2.4 versions of distutils, not to mention py2exe, Pyrex, and other distutils extensions, along with the quirky uses of distutils that exist in dozens of distributed Python packages. However, I think you and I may perhaps have different definitions of compatibility. Mine is that things just work and users don't have to do anything special. For that definition, setuptools is extremely compatible with the standard distutils. In many situations it's more compatible than the distutils themselves, in that more things just work. ;) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Phillip J. Eby wrote: At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: Why should a 3rd party extension be hot-fixing the standard Python distribution ? Because setuptools installs things in zip files, and older versions of pydoc don't work for packages zip files. That doesn't answer my question. If you want to provide a hot fix for Python 2.3 and 2.4, you should make this a separate install, so that users are aware that their Python distribution is about to get modified in ways that have nothing to do with setuptools. Their Python distribution is not modified -- new modules are merely placed on sys.path ahead of the stdlib. (Which, I might add, is a perfectly normal process in Python -- nothing stops users from installing their own version of pydoc or any other module via PYTHONPATH.) Note also that uninstalling setuptools by removing the .egg file or directory will effectively remove the hot fix, since the modules live in the .egg, not in the stdlib. Whether you place a module with the same name in front of the stdlib path in PYTHONPATH (e.g. copy it into site-packages) or replace the file in the Python installation is really the same thing to the user. Third-party extension *should not do this* ! It's OK to have private modified copies of a module inside a package or used inside an application, but python setup.py install should never (effectively) replace a Python stdlib module with some modified copy without explicit user interaction. If an application wants to use setuptools for e.g. plugin management, then standard distutils features will get replaced by setuptools implementations which are not compatible to the standard distutils commands, effectively making it impossible to access the original versions. Please do a little research before you spread FUD. The 'pkg_resources' module is used for runtime plugin management, and it does not monkeypatch anything. I'm talking about the setuptools package which does apply monkey patching and is needed to manage the download and installation of plugin eggs, AFAIK. Monkey patching is only a last resort in case nothing else works. In this case, it's not needed, since distutils provides the interfaces needed to extend its command classes via the setup() call. The monkeypatching is there so that the easy_install command can build eggs for packages that use the distutils. It's also there so that other distutils extensions that monkeypatch distutils (and there are a few of them out there) will have a better chance of working with setuptools. I originally took a minimally-invasive approach to setuptools-distutils interaction, but it was simply not possible to provide a high-degree of backward and sideward compatibility without it. In fact, I seem to recall finding some behaviors in some versions of distutils that can't be modified without monkeypatching, although the details are escaping me at this moment. That's a very vague comment. The distutils mechanism for providing your own command classes lets you take complete control over distutils if needed. What's good about it, is that this approach doesn't modify anything inside distutils at run-time, but does these modifications on a per-setup()-call basis. As for setuptools, you import the package and suddenly distutils isn't what's documented on python.org anymore. As for discussion, Guido originally brought up the question here a few months ago, and it's been listed in PEP 356 for a while. I've also posted things related to the inclusion both here and in distutils-sig. I know, but the discussions haven't really helped much in getting the setuptools design compatible with standard distutils. That's because the job was already done. :) Not much of an argument, if you ask me. Some of the design decisions you made in setuptools are simply wrong IMHO and these need to be discussed in a PEP process. The setuptools design bends over backwards to be compatible with Python 2.3 and 2.4 versions of distutils, not to mention py2exe, Pyrex, and other distutils extensions, along with the quirky uses of distutils that exist in dozens of distributed Python packages. However, I think you and I may perhaps have different definitions of compatibility. Mine is that things just work and users don't have to do anything special. For that definition, setuptools is extremely compatible with the standard distutils. In many situations it's more compatible than the distutils themselves, in that more things just work. ;) You've implemented your own view of just works. This is fine, but please remember that Python is a collaborative work, so design decisions have to be worked out in collaboration as well. There aren't all that many things that are wrong in setuptools, but some of them are essential: * setuptools should not monkey patch distutils on import * the standard python setup.py install should continue to
Re: [Python-Dev] setuptools in the stdlib ([Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 09:02 PM 4/18/2006 +0200, M.-A. Lemburg wrote: Phillip J. Eby wrote: At 07:15 PM 4/18/2006 +0200, M.-A. Lemburg wrote: Why should a 3rd party extension be hot-fixing the standard Python distribution ? Because setuptools installs things in zip files, and older versions of pydoc don't work for packages zip files. That doesn't answer my question. That is the answer to the question you asked: why hot-fix? Because setuptools uses zip files, and older pydocs crash when trying to display the documentation for a package (not module; modules were fine) that is in a zip file. Third-party extension *should not do this* ! If you install setuptools, you presumably would like for things to work, and the hot fix eliminates a bug that interferes with them working. I'm not sure, however, what you believe any of that has to do with python-checkins or python-dev. The version of setuptools that will do this is not yet released, and the hotfix aspect will be documented. If it causes actual problems for actual setuptools users, it will be actually fixed or actually removed. :) (Meanwhile, the separately-distributed setuptools package is not a part of Python, any more than the Optik is, despite Python including the 'optparse' package spun off from Optik.) I'm talking about the setuptools package which does apply monkey patching and is needed to manage the download and installation of plugin eggs, AFAIK. In which case the monkeypatching is needed, to handle situations like building eggs for third-party non-setuptools libraries listed as dependencies. You can't have it both ways; either you need setuptools or you don't. If you do, you will almost certainly need the monkeypatching, and you'll definitely need it to build eggs for non-setuptools-based packages. What's good about it, is that this approach doesn't modify anything inside distutils at run-time, but does these modifications on a per-setup()-call basis. What's bad about it, is that it provides no options for virtualizing the execution of other packages' setup scripts. The 'run_setup' mechanism was horribly broken in the versions of the distutils that I wrestled with. (I also don't see anybody stepping up to provide alternative implementations or patches to third-party setup scripts or distutils extensions to deal with this issue.) There aren't all that many things that are wrong in setuptools, but some of them are essential: * setuptools should not monkey patch distutils on import Please propose an alternative, or better still, produce a patch. Be sure that it still allows distutils extensions like py2exe to work. The only real alternative to monkeypatching that I'm aware of for that issue is to actually merge setuptools with the distutils, but my guess is that you'll like that even less. :) * the standard python setup.py install should continue to work as documented in the Python documentation; any new install command should use a different name, e.g. install_egg * placing too many ZIP files on PYTHONPATH is a bad idea since it slows down import searches for all Python applications, not just ones relying on eggs; a solution to this would be to have a PYTHONEGGPATH which is then only scanned by egg-aware modules/applications * the user should have freedom of choice in whether to have her Python installation rely on eggs or not (and not only --by-using-some-complicated-options) These questions have been hashed to death on the distutils-sig already, but I will try to summarize my responses here as briefly as practical while still covering the full spectrum of issues. I would begin by saying that the tradeoffs I've made favor inexperienced users at the expense of experts (who I assume are capable of learning to use --root or --single-version-externally managed in order to have finer-grained control). Your proposals, however, generally favor experts at the expense of the average user, and treat it as axiomatic that the benefits provided by setuptools are not worth having, no matter how small the cost. By contrast, users who *want* to use setuptools for either distribution or installation of packages, would rather it just work, no matter how much complexity is required behind-the-scenes. They don't care about my cost to implement, they just care that they type easy_install foo to get a package from PyPI or setup.py sdist bdist_egg upload to put it there. Therefore, it makes no sense to apply your design approach to setuptools, since by your criteria it wouldn't exist in the first place! After all, expert users can munge distutils extensions to their hearts' content (and you certainly have done so already). Next, your own measurements posted to the distutils-sig debunked the PYTHONPATH performance question, IMO: if you installed *every Python package listed on PyPI at that time* as a zip file (assuming that the average zipfile directory size was the same as in
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
Fredrik Lundh wrote: it's still listed under possible additions in the release PEP, and there don't seem to be a PEP or any other easily located document that explains exactly how it works, what's required from library developers, how it affects existing toolchains, etc. is this really ready for inclusion ? does anyone but Phillip understand how it works ? I don't understand it. My concern is that it appears to involve a lot of magic. This magic might do the right thing in many cases, and it might indeed help the user that the magic is present, yet I still do believe that magic is inherently evil: explicit is better than implicit. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 11:55 PM 4/18/2006 +0200, Fredrik Lundh wrote: who decided that setuptools should be added to 2.5, btw? Guido proposed it on Python-dev when the 2.5 schedule was first being discussed. I discussed it with him off-list, to ensure that it could be done in a way that wouldn't interfere with existing setuptools users or affect Python itself in a negative way. (For example, it needed to be upgradeable in the field in case users wanted/needed a later version than the one included in 2.5.) He then mentioned it in his 2.5 slideshow at PyCon. This is the first anyone's objected to it, however, at least that I'm aware of. it's still listed under possible additions in the release PEP, I imagine that might be why nobody raised any objections sooner, although I understood the possibility to mean if nobody objects. :) I also posted on Python-dev repeatedly in recent weeks, referring to how the various PEP 302 fixes and updates would interact with setuptools when it got in for 2.5. Also, Neal emailed me the week before last, asking when I would be getting setuptools checked in, and I told him April 17th - i.e., yesterday. So, I was under the impression this was all a done deal. and there don't seem to be a PEP or any other easily located document that explains exactly how it works, what's required from library developers, how it affects existing toolchains, etc. The setuptools manual is currently at: http://peak.telecommunity.com/DevCenter/setuptools pending conversion to the standard Pythondoc format. I posted earlier today asking about how it, and the other related manuals should be included in the overall Python documentation structure: http://mail.python.org/pipermail/python-dev/2006-April/063846.html The reST source of these manuals is in trunk/sandbox/setuptools, where it has been evolving over the last year. is this really ready for inclusion ? Please define ready. I don't mean that in a flippant way, I just don't know what you mean. does anyone but Phillip understand how it works ? Does anybody besides Thomas understand how ctypes works? ;) Rhetorical jokes aside, every time I've made a significant change to how setuptools works, I've posted it to the distutils-sig -- and usually I make proposals in advance to get feedback or stimulate discussion. I regularly post explanations there in response to questions from people who are integrating it with system packaging tools, or creating various other customizations. And there are other people on distutils-sig who can answer questions about it. The TurboGears community is proficient enough with it that it's only once every few months now that a question gets kicked upstairs to me to answer. A number of people have contributed patches, including Ian Bicking and Tres Seaver. Bob Ippolito was a significant participant in the original design and wrote some of the initial code for the runtime. A *considerable* number of distutils-sig participants have had design input, either through direct suggestions, or through their giving more use case examples that I needed to make just work. So, I'm not too pleased by insinuations that setuptools is anything other than a Python community project. But MAL and MvL are the only folks from Python-Dev who I've seen over there arguing for changes to setuptools -- and I actually made changes based on their input, although they rarely got 100% of what they asked for. The --really-long-option MAL is complaining about was put in to provide a feature that *he and MvL wanted me to include*; I just don't want that behavior to be the default behavior for setuptools. (And neither do package developers who have to support non-root users on virtual hosting systems, or other environments where system packaging tools aren't available.) So, it seems to me that MAL claiming that nobody got to participate in the design process is rather misleading. It's like somebody who wanted decorators in Python and then gripes about the '@' syntax. Everybody's got to compromise a bit. I put their feature in months ago, and it is even the default when you use --root with install. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib
On Tuesday 18 April 2006 19:00, Phillip J. Eby wrote: He then mentioned it in his 2.5 slideshow at PyCon. This is the first anyone's objected to it, however, at least that I'm aware of. Until the past week, I wasn't aware it was being considered. But then, I've not been paying a lot of attention lately, so I suspect that's my fault. The setuptools manual is currently at: http://peak.telecommunity.com/DevCenter/setuptools pending conversion to the standard Pythondoc format. I posted earlier today asking about how it, and the other related manuals should be included in the overall Python documentation structure: Saw that; hopefully I'll have a chance to look at it soon. I wonder, generally, if it should be merged into the distutils documentation. Those documents happen to be distutils-centric now, because that's what's been provided. Their titles should be the guide to their content, however. So, I'm not too pleased by insinuations that setuptools is anything other than a Python community project. I've no doubt about that at all, FWIW. I think you've put a lot of effort into discussing it with the community, and applaud you for that as well as your implementation efforts. -Fred -- Fred L. Drake, Jr. fdrake at acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 12:49 AM 4/19/2006 +0200, Martin v. Löwis wrote: Fredrik Lundh wrote: it's still listed under possible additions in the release PEP, and there don't seem to be a PEP or any other easily located document that explains exactly how it works, what's required from library developers, how it affects existing toolchains, etc. is this really ready for inclusion ? does anyone but Phillip understand how it works ? I don't understand it. Have you read the manuals? My concern is that it appears to involve a lot of magic. Please define magic. Better, please point to the API functions or classes, or the setup commands documented in the manual, to show me what these things are that appear to be magic. There do exist implementation details that are not included in user documentation, but this is going to be true of any system. These details are sometimes necessarily complex due to distutils limitations, behavioral quirks of deployed packages using distutils, and the sometimes baroque variations in sys.path resolution across platforms, Python versions, and invocation methods. That these details are not discussed at length in the user documentation is because they are boring hideous things that I wish I didn't have to know myself, and that few people using setuptools would ever want to know. They *do*, however, get discussed (or at least explained by me) at length on the distutils-sig. I explain the quirks and the tradeoffs for each bit of implementation in detail there, and have been doing so for nearly a year now. Anybody who wants to know what's going on has had plenty of opportunity to learn. This magic might do the right thing in many cases, and it might indeed help the user that the magic is present, yet I still do believe that magic is inherently evil: explicit is better than implicit. Are documented defaults implicit or magic? To the extent that there is anything that may be called magic in setuptools, it exists only because the necessary infrastructure isn't already present in Python, or because it was required to work around the quirks of other systems. Setuptools chooses to work at all costs because backward-compatibility skirts the chicken-and-egg problems that would otherwise exist. If setuptools *weren't* so highly backward-compatible, its use would never have caught on enough to allow this discussion to be taking place in the first place. :) The primary place where it *isn't* backward compatible is if somebody does setup.py install *without* using --root, *and* they care about the exact file layout of the result, *and* they are installing a package that explicitly uses setuptools... in which case their complaint is with the author of the package, if the author didn't explain that they used setuptools or point them to relevant documentation. Setuptools' manual prominently explains what developers should tell their users, if they use setuptools in their setup.py: To keep these users happy, you should review the following topics in your project's installation instructions, if they are relevant to your project and your target audience isn't already familiar with setuptools and easy_install. http://peak.telecommunity.com/DevCenter/setuptools#what-your-users-should-know With respect to you and MAL, I think that some of your judgments regarding setuptools may have perhaps been largely formed at a time last year when, among other things: * That documentation section I just referenced didn't exist yet * Many common installation scenarios (e.g. custom PYTHONPATH) didn't just work without special setup steps * --single-version-externally-managed was barely a proposal, and it wasn't automatically activated when --root was used These are significant changes that are directly relevant to the objections that you and he raised (regarding startup time, path length, tools compatibility, etc.), and which I added because of those objections. I think that they are appropriate responses to the issues raised, in that they allow the audience that cares about them (you, MAL, and system packagers in general) to avoid the problems that those features' absence caused. It would probably helpful if you would both be as specific as possible in your objections so that they can be addressed individually. If you don't want setuptools in 2.5, let's be clear either on the specific objections. If the objection is to the very *idea* of setuptools, that's fine too, as long as we're clear that's what the objection is. So I would ask that you please make a list of what you would have to see changed in setuptools before you would consider it viable for stdlib inclusion, or simply admit that there are no changes that would satisfy you, or that you don't know enough about it to say, or that you'd like it to be kicked back to distutils-sig for more discussion ad infinitum, or whatever your actual objections are. Then I will make my responses to your
Re: [Python-Dev] setuptools in the stdlib
At 07:08 PM 4/18/2006 -0400, Fred L. Drake, Jr. wrote: Saw that; hopefully I'll have a chance to look at it soon. I wonder, generally, if it should be merged into the distutils documentation. Those documents happen to be distutils-centric now, because that's what's been provided. Their titles should be the guide to their content, however. No doubt that's the proper thing to do in the long term, when/if setuptools is the official One Obvious Way To Do It. I was wondering, however, if perhaps Python 2.5 should include them as Building and Distributing Python Eggs (for what's now setuptools.txt) and Installing Python Eggs (for what's now EasyInstall.txt). I've no doubt about that at all, FWIW. I think you've put a lot of effort into discussing it with the community, and applaud you for that as well as your implementation efforts. Thank you. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib
Fred L. Drake, Jr. [EMAIL PROTECTED] wrote: So, I'm not too pleased by insinuations that setuptools is anything other than a Python community project. I've no doubt about that at all, FWIW. I think you've put a lot of effort into discussing it with the community, and applaud you for that as well as your implementation efforts. I agree but I have a question for Phil though: why can't many of the setuptools feature be simply integrated within the existing distutils? I have been fighting with distutils quite some time and have had to monkey-patch it somehow to fit my needs. I later discovered that setuptools included many of those fixes already (let alone the new features). I actually welcome all those setuptools fixes in the Just Works(TM) principle with which I totally agree. But, why can't setuptools be gradually merged into distutils, instead of being kept as a separate package? Let's take a real example: setuptools' sdist is much enhanced, has integration with CVS/SVN, uses MANIFEST in a way that it really works, etc. Why can't it be merged into the original distutils? Is it just for backward compatibility? If so, can't we have some kind of versioning system? Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib
At 02:12 AM 4/19/2006 +0200, Giovanni Bajo wrote: But, why can't setuptools be gradually merged into distutils, instead of being kept as a separate package? Let's take a real example: setuptools' sdist is much enhanced, has integration with CVS/SVN, uses MANIFEST in a way that it really works, etc. Why can't it be merged into the original distutils? Is it just for backward compatibility? This specific issue was discussed last year on the distutils-sig, and the issue is indeed one of compatibility. Setuptools' behavior for MANIFEST generation definitely matches new or infrequent users' expectations 1000% better than the distutils, and requires much less work to get right, even for experts. But for anybody who has extended the distutils using external tools, it would not necessarily work. MAL gave the example of someone who has written other scripts or Makefile rules to add things to MANIFEST or use it to do other things. They might be relying on quirks of the existing behaviors, in other words, and thus it should not be changed without explicit action on their part. And I agree with his reasoning, although I also think that any distutils 2 should have only One Obvious Way to send input to that process, and it should be via MANIFEST.in, not MANIFEST. Likewise, it should have only one way to get the output. However, unless somebody explicitly chooses to use distutils 2, they should get the old behavior. This unfortunately means that we can't backport most of setuptools' enhancements to the existing distutils without breaking backward compatibility for people who may have made extensive investment in integrating with the distutils. (Of course, how many of these people exist I don't know; in my personal experience it seems rare for people to integrate with external tools in this fashion, versus simply subclassing things in Python or abandoning distutils altogether. But that's a separate question.) If so, can't we have some kind of versioning system? We do: import setuptools. We could perhaps rename it to import distutils2 if you prefer, but it would mean essentially the same thing. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
On 4/18/06, Phillip J. Eby [EMAIL PROTECTED] wrote: At 11:55 PM 4/18/2006 +0200, Fredrik Lundh wrote: who decided that setuptools should be added to 2.5, btw? Guido proposed it on Python-dev when the 2.5 schedule was first being discussed. I discussed it with him off-list, ... I thought more was discussed on-list, but apparently not. I searched my mail and could find no direct discussion on python-dev. I saw quite a few references to setuptools being included in 2.5, but nothing explicit. That's unfortunate. I think I talked to Guido and probably Anthony off-list to see if they objected to setuptools in 2.5. My only mail to Phillip was to see if it was going in. it's still listed under possible additions in the release PEP, I imagine that might be why nobody raised any objections sooner, although I understood the possibility to mean if nobody objects. :) I was also working under the assumption that people would complain if they didn't like something. What do people think should happen for the Possible features section? Should I ask if there are any objections to each item? In the current list of possible features, I think the only new features that stand a chance in 2.5 are: wsgiref and python pgen. Ronald is working on the fat Mac binaries, it's only in this section because it's not complete, but anticipated. All the other things are small cleanups (icons, Demo, file/open) or things that have to happen (ssize_t cleanup). @decorator and functools are possible, but no one is doing anything about them, so I suspect they will not go in unless someone cares enough to do the work and Guido agrees. Since he's still on vacation, these won't be decided for another week or so. Probably these will be decided after a2 goes out (hopefully around the 25th). If you have any questions about the status of 2.5, speak up. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] setuptools in the stdlib ( r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py)
At 10:14 PM 4/18/2006 -0700, Neal Norwitz wrote: On 4/18/06, Phillip J. Eby [EMAIL PROTECTED] wrote: At 11:55 PM 4/18/2006 +0200, Fredrik Lundh wrote: who decided that setuptools should be added to 2.5, btw? Guido proposed it on Python-dev when the 2.5 schedule was first being discussed. I discussed it with him off-list, ... I thought more was discussed on-list, but apparently not. I searched my mail and could find no direct discussion on python-dev. I saw quite a few references to setuptools being included in 2.5, but nothing explicit. That's unfortunate. Here are the threads that Guido started; the longer one includes a number of discussions about setuptools-related features. MvL raised an objection to the whole idea of a Python-specific packaging format, which I responded to, and there was some other discussion about the relative utility of setuptools' approach versus learning (and building) a half-dozen bdist_* formats for different platforms: http://mail.python.org/pipermail/python-dev/2006-February/060723.html http://mail.python.org/pipermail/python-dev/2006-February/060869.html I was also working under the assumption that people would complain if they didn't like something. What do people think should happen for the Possible features section? Should I ask if there are any objections to each item? Well, Guido asked about including setuptools, and it doesn't seem to have done any good in this case. :) I'm not sure how much more explicit you can get. Yes, he called it setuplib in the first thread, but Georg knew what he meant anyway, and then Guido corrected himself in the second thread -- which also had a title that should've caught the eye of anybody interested in distutils-related things. (bdist_* to stdlib.) I was surprised that MAL didn't comment *then*, actually, and mistakenly thought it meant that our last discussion on the distutils-sig (and my attempts to deal with the problems) had been successful. Between that and MvL's mild response to the explicit discussion of supporting setuptools, I thought their votes had effectively moved from -1 to -0. Off-list discussion with Fredrik suggested that he too had shifted from -1 to -0, and since those were the only core developers that I knew of who had ever said anything the least bit negative about setuptools, I assumed this meant that Guido's motion had carried, so to speak. I mention this mainly to clarify that this was not some attempt by me to slip setuptools in past opposition; I genuinely thought the objectors had gone from don't bring that stuff anywhere near me to I don't like it but other people seem to, so whatever. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
