Re: [Distutils] Dynamic linking on Linux, Mac OS, et al
Phillip J. Eby wrote: Hi folks. I just implemented preliminary shared library build support for setuptools, but I ran into a bit of a snag. I was operating under the assumption that you could simply put shared libraries in the directory alongside the extensions that use them, but in practice it turns out that this only works on Windows. On Linux I found that you could only put shared libraries in a directory on LD_LIBRARY_PATH, and I suspect a similar issue will be found under OS X and other Unixes. This seems to be a bit of a problem, in that it appears to mean there's no sane way to include shared libraries in an egg on any platform other than Windows. It's common practice for Python extensions built for Windows to include shared libraries to avoid having to install them in the Windows system directories, but for Unix-like OSes this isn't really the normal practice anyway. But, in the case where a developer wants to include a shared library as part of the distribution, this remains somewhat problematic. One can of course install the library to a platform-appropriate location ($prefix/lib on most platforms), but to do it from an egg is harder. You can't simply unzip the egg any more, you would have to also move the libraries to a standard installation location, breaking any hope of multi-version support. (And this would have to be able to happen on the fly, when shared libraries and extensions are extracted to the cache directory, if running zipped.) I don't suppose you can modify LD_LIBRARY_PATH on the fly? Any other options? Should I just give up on supporting shared libraries on anything but Windows, perhaps silently converting the shared libraries to static ones? On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the linker at build time that causes the run-time linker to use specific paths to load libraries. This feature is supported by disutils through its --rpath/-R build_ext options. This doesn't work on Mac OS X, but it sounds like someone else may know how to make this work. BTW, I often have need to be able to create what we call application buildouts that contain various facilities needed by a particular application. These facilities are mostly Python-based but often include things like relational databases, ICU, ldap servers, etc. We want to be able to have buildouts for multiple applications (or multiple application versions) on a single development machine. Each buildout needs to be able to control versions of things it uses (including Python versions). We try to minimize the dependence on system libraries, as these are usually different on development and deployment systems and because we may need different versions for different applications on a development machine. I'm just beginning to learn about eggs. I certainly hope I'll have the same flexibility when installing eggs that I have now when using distutils. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] License for virtual-python.py?
virtual-python.py looks very useful. It lacks a license/copyright statement though. Can we get this? Are there plans to include this script in 2.5? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Dynamic linking on Linux, Mac OS, et al
Phillip J. Eby wrote: At 10:06 AM 1/14/2006 -0500, Jim Fulton wrote: On many Linux (Unix?) platforms, you can pass an -rpath/-R option to the linker at build time that causes the run-time linker to use specific paths to load libraries. This feature is supported by disutils through its --rpath/-R build_ext options. But that's not relocatable (to different directories), and eggs are intended to be zero-install for simple use cases. So, the solution I came up with uses '.' in the rpath, and a stub loader that changes directories Changes directories? Uh ... this sounds brittle. long enough to get the extension to load and find its library (or libraries). This appears to work decently enough on Linux. BTW, I often have need to be able to create what we call application buildouts that contain various facilities needed by a particular application. These facilities are mostly Python-based but often include things like relational databases, ICU, ldap servers, etc. We want to be able to have buildouts for multiple applications (or multiple application versions) on a single development machine. Each buildout needs to be able to control versions of things it uses (including Python versions). We try to minimize the dependence on system libraries, as these are usually different on development and deployment systems and because we may need different versions for different applications on a development machine. I'm just beginning to learn about eggs. I certainly hope I'll have the same flexibility when installing eggs that I have now when using distutils. I'm not sure I can answer that question, or that I even know what your question is, actually. :) Eggs don't really try to encompass building arbitrary executables, only Python code, C/C++/Pyrex/SWIG extensions, and libraries linked to the extensions. It sort of sounds like you're saying you'd like to be able to build a bunch of non-Python stuff too, which is likely to remain out of scope for the foreseeable future. No. I want to be able to get a Python egg built in such a way that it uses a library in a location I tell it to, rather than in the standard system libraries. For example, suppose I have an egg with an extension that uses ICU, and that I've installed the ICU libraries in a non-standard place. When I build the egg, I want to tell it to use the location that I specify for ICU. I can do this with setup now with the -R option to build_ext. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] setuptools in 2.5?
Has a decision been made whether to include setuptools in 2.5? If it is going to be included, then I'll update doctest to use pkg_resources to find doctest text files. If it isn't going to be included then I'll have a quandry. :) I guess that in that case, I'd have to use some conditional logic. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] eggs in Python path
The eggs quick guide http://peak.telecommunity.com/DevCenter/PythonEggs#using-eggs says that eggs can be installed by just putting them on sys.path. This doesn't seem to be enough though. A .pth file seem to also be necessary. Am I missing something? Or is the documentation incorrect? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Easy install, making Traditional PYTHONPATH-based Installation work better
I'be been learning about setuptools and eggs and am very pleased. There is one area where I'd like to see a small tweak. In: http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations four options are presented for installing outside of Python installation. Naturally, I prefer the the last option, which the documentation recommends against. :) Here's why. An important use case to me is the ability for a developer to work in a development sandbox without affecting anything outside the sandbox. The solution needs to be cross platform. The Administrator Installation option doesn't work for me because it requires modifying the Python setup. Neither the Mac OS X User Installation nor the 'Creating a Virtual Python' options work for me because they are not cross platform. I also find the Virtual Python approach a bit heavy handed. In playing a bit, I find the Traditional PYTHONPATH-based Installation works pretty well except for the problem of having to set the path manually when invoking scripts. This problem could be addressed by having the generated scripts include the custom path settings. I recommend adding two new options to the easy_install section of the configuration file: script_path A list of paths, separated by commas to be added to sys.path by generated scripts. script_pth_files A list of pth files, separated by commas to be added be processed by script files. Generated scripts would include code to manipulate sys.path based on this information. So, for example, in a workspace with a 'lib' directory containing eggs, I'd have: [easy_install] site_dirs = '/here/lib' script_path = '/here/lib' script_pth_files = '/here/lib/easy-install.pth' The script_pth_files option is needed, I think, to avoid hardcoding the setuptools egg path in the generated scripts. Note that it could be argued that the site_dirs option should be enough and that generated scripts should simply reflect this option. Thoughts? For now, I can create a wrapper around easy_install that does custom script generation, which could serve as a prototype for the proposed change. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API for finding plugins
Phillip J. Eby wrote: I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements: I don't fully understand the goal here. From later discussion, I think you envision a model where people drop eggs into some directory and an application should be able to analyze this directory to determine which ones to use, meaning which to include in some working set. In addition to automatically determining the working set, the application should be able to find the entry points in that working set. Does that capture what you want to do? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A prefix option?
Phillip J. Eby wrote: At 02:42 PM 2/8/2006 -0800, Ben Bangert wrote: I'm wondering if there's any plans for a prefix option, which functions like the common makefile prefix option for where the lib/ bin dir will then be, etc. My main reason for asking is this blog entry regarding setuptools installs: http://bitworking.org/news/ Please_stop_using_setuptools__at_least_exclusively__for_now Rather than stop using setuptools, package authors should consult: http://peak.telecommunity.com/DevCenter/setuptools#what-your-users-should-know which has a (hopefully) comprehensive list of things that you may need or want to inform your users about. A --prefix option wouldn't have removed the need for *any* of them, I'm afraid. Note also that among the installation options offered at: http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations is the option to create a virtual Python installation, and the supplied virtual-python.py script *does* accept a --prefix option, and would have allowed him to do exactly what he wanted to do by simply using the ~/bin/python created automatically by virtual-python.py. I really don't think the virtual python approach is viable, at a minimum because it doesn't work on windows. It is also unacceptably heavy IMO. I think we need a simple way to handle custom directories that is: - cross platform - doesn't require modifying the Python install - allows chained/multile custom directories, which means the trick of putting eggs and scripts in the same directory doesn't work, - is simple. :) It appears that this has something to do with your plugin proposal. I'll try to participate in that discussion to tease this out a bit more. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A prefix option?
Phillip J. Eby wrote: At 07:31 AM 2/9/2006 -0500, Jim Fulton wrote: I really don't think the virtual python approach is viable, at a minimum because it doesn't work on windows. It is also unacceptably heavy IMO. What do you mean by heavy? It essentially duplicates the python install via symbolic links. I think we need a simple way to handle custom directories that is: - cross platform - doesn't require modifying the Python install - allows chained/multile custom directories, which means the trick of putting eggs and scripts in the same directory doesn't work, PYTHONPATH-based installs allow all of this, as long as you add the setuptools .egg to PYTHONPATH. Agreed. I think PYTHONPATH-based installations are close to what some of us need. I think that having to set the path is a hassle that is possible to overcome and worth doing. As I've suggested before, since easy install generates startup scripts, it should be possible to for it to generate startup scripts that take care of setting the path. I think it should be possible to have Python startup scripts invoke setuptools' site.py to add necessary .pth files to the path. If I'm wrong, then another option is for setuptools to generate sh scripts on 'nix platforms, much as it generates exe's on Windows. (I presume that the generates exe files on windows then invoke Python and can also manipulate the environment.) - is simple. :) PYTHONPATH is simple everywhere but Windows, and it's complex there only because you have to right click a bunch of icons to get to the editing facility. Eek. I would never consider setting the PYTHONPATH that way on Windows to satisfy this use case. The fact that you suggest this suggests that you missunderstand my use case. OK, I left something out of my bulleted list: - Doesn't require changes to a user's profile or anything else outside the working directory tree. ISTM that what's missing from your requirements statement is how you would like to activate or deactivate these various custom directories. An alternate Python executable? Script wrappers? Commands you run to set what directories are active? I worry that I'm not setuptools zenfull enough to make the best proposal yet, but here's what I envision so far. To create a workspace, you create the directory and put a setup.cfg in it. You then install easy_install into that directory (typically into that directory's bin directory). Then when you run that directory's easy_install script, eggs are installed into the workspace (e.g. into some sort of library directory within the workspace). In addition, wrapper scripts build during the installation set the path and invoke setuptools' site.py to reflect settings in setup.cfg. You might be able to think of a better way to do this. My goal is to have self contained working directories. In addition, I want to have chained self-contained working directories, which is why the excellent put scripts and eggs in the same place solution doesn't work for me (more below). It appears that this has something to do with your plugin proposal. Not that I'm aware of, actually. The plugin facility is intended for an application instance path, distinct from a general/global installation location. But that's exactly what I'm talking about. To make this more concrete: My hope is that Zope 3.3 will be highly eggs based. People expect to be able to install Zope without odifying Python. Basically, a Zope install is a separate aplication instance. (Note that to leverage distutils, Zope 3 windows releases have been installed into windows Python installations. This installation experience has been unpopular.) I would want a Zope install to be largely a collection of eggs. I would also want a Zope install to support easy installayion of additional eggs into the Zope install or into a directory associated with the Zope install (probably the later). A Zope install provides a simple application for creating Zope instances. A Zope instance is what actually runs a Zope application server. Multiple Zope instances can share a single Zope software install. (Of course, multiple Zope sotware installs can share a single Python install.) Of course, I want people to be able to install additional eggs, including eggs with scripts, into the Zope instance. I don't want users to have to manually manipulate paths to be able to use these. Traditionally, in Zope 2, packages installed into Zope instances have been plug-in-like in that they could just be dropped into a directory and magically used. In Zope 3, we've gone for a more explicit approach, which I think I prefer. More on that in the plugin thread, which, sadly, I probably won't have time to contribute more to until this weekend. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com
Re: [Distutils] A prefix option?
Phillip J. Eby wrote:
At 06:40 AM 2/10/2006 -0500, Jim Fulton wrote:
Phillip J. Eby wrote:
At 07:31 AM 2/9/2006 -0500, Jim Fulton wrote:
I really don't think the virtual python approach is viable, at a
minimum
because it doesn't work on windows. It is also unacceptably heavy IMO.
What do you mean by heavy?
It essentially duplicates the python install via symbolic links.
That's just a statement of what it does. I was looking for an
explanation of what makes that heavy. :) One assumes that on Unix,
directory entries and inodes are relatively cheap.
heavy is qualitative. I was explaining what it did that made it seem
heavy to me. Perhaps I should have said heavy handed. So, to be more
specific, in addition to the fatal flaw of not working on windows:
- It copies, not links, the python executable. On my system, that's
a 3 meg file. Is that heavy? Some would say yes. With disks
measured in hundreds of gigabytes these days, it doesn't get me
very excited. I do think it is pretty heavy handed.
- While it links individual files it copies the directory structure,
including include and site-packages. This means that new packages
installed into the base python won't be reflected in the virtual python.
Aside from the lack of windows suport, I might be willing to live with this
approach if we couldn't do better, but I think we can find better solutions.
PYTHONPATH-based installs allow all of this, as long as you add the
setuptools .egg to PYTHONPATH.
Agreed. I think PYTHONPATH-based installations are close to what some
of us
need. I think that having to set the path is a hassle that is possible
to overcome and worth doing. As I've suggested before, since easy
install generates startup scripts, it should be possible to for it
to generate startup scripts that take care of setting the path.
I think it should be possible to have Python startup scripts invoke
setuptools' site.py to add necessary .pth files to the path.
The regular site.py suffices for that. What it doesn't suffice for is
getting those directories on *ahead* of the system-provided libraries.
If you don't care about that, then adding a series of statements like
this to the script would suffice:
__import__('site').addsitedir(some target dir)
Where the series of directories represents your instance, in the order
of highest to lowest precedence.
Are you worried about system modules that might already have been imported?
I presume that site.py could put the local directories in front of system
directories in sys.path.
Of course, this is unnecessary *except* for obtaining the location of
setuptools itself, unless the goal is to have plugins installed using
.pth files, and to manage that setup externally to the actual programs.
(By which I mean, the programs themselves have no notion of a plugin
directory, they simply expect everything to be already set up on
sys.path when they run.)
I'm not sure. I'm about 80% sure that I don't care about .pth files
except setuptool's, but others may care. shrug
...
Well, you haven't really explained your use case enough for me to
understand even what these programs are or what people are doing with
them. You just listed some requirements.
Not for lack of trying.
My goal is to have self contained working directories. In addition,
I want to have chained self-contained working directories, which is
why the excellent put scripts and eggs in the same place solution
doesn't work for me (more below).
You could generate a sitecustomize.py in a scripts+eggs directory, and
have it do:
import site
site.addsitedir(this directory)
site.addsitedir(another directory)
etc. This would basically chain the directories in a way that supports
.pth files. Then just easy_install scripts and eggs to the single
target directory, including the target directory in --site-dirs so
easy_install knows it can use .pth files.
Presto - you're done. Running a script would now suffice to set its
environment. Note that anything it references via .pth files will be
*subordinate* to the same things in site-packages, but since you're
presumably controlling what Python install is used for this, and in any
case the scripts have require() used to bump up anything the script
directly requires, this shouldn't be a big issue.
Note that in this setup, sitecustomize.py is the only configuration file
you need, and you don't need any upgrades to setuptools as it exists
today to make this work. Just slap sitecustomize in the same directory
with the scripts and eggs, and all is well.
Why must eggs be in the same directory? Since sitecustomize adds additional
directories, couldn't the eggs be there?
You can, if you wish,
modify it to do other sys.path munging according to your needs.
Yup.
Unfortunately, due to differences in path processing on Windows and
'nix, the above
Re: [Distutils] DWIM installation with setuptools
Phillip J. Eby wrote: At 12:07 PM 2/11/2006 -0500, Jim Fulton wrote: I think you're on the right track, at least from a functionality point of view, although I don't want to require a special PYTHONPATH setting to use a generated wrapper script. Just two quick comments: the PYTHONPATH implementation is now in SVN. I decided to go with it on the reasoning that the new site.py hack is actually less fragile than the old one, because the new one allows any vendor-specific site.py patches to take effect, whereas the old one didn't. This allows full DWIM for people who expect PYTHONPATH to basically work as it always has without setuptools, and now allows installation of any setuptools-based package without any RTFMing whatsoever. :) Second, now that the PYTHONPATH support works, this provides a stable base for your desired environment to be built on. We only need to have the option of building a PYTHONPATH into a generated script (and perhaps also generate a script to run the interpreter with a set PYTHONPATH) in order to have everything work. This seems more modular to me than trying to do everything in the wrappers. I think I agree. So, the only things left to figure out at this point are: 1. how the PYTHONPATH wrappers should deal with an existing PYTHONPATH setting IMO, they should honor it, putting it ahead of the bits they want to add. 2. how to tell easy_install whether to use PYTHONPATH wrapping and which kind #2 includes the question of whether scripts should be able to affect the wrapping via options specified by the setup script. In other words, if I have a security-sensitive script, should I be able to specify in my setup that my script requires a baked-in and frozen PYTHONPATH? Good point. Explicit control. I like it! (Mmmm, baked and frozen...) Regarding #1, my first thought is that the most backward-compatible thing to do would be to extend the current PYTHONPATH with the one that was in effect when the script was installed. Thus, if you set a PYTHONPATH with the intent of overriding something, your wish will still be granted. (The 'site' module eliminates duplicate paths, so sys.path isn't elongated in the case of duplication.) You can still break the script by setting PYTHONPATH to something that creates a version conflict, but you could do that anyway. This is close, but too implicit, IMO. I'd rather have an easy_install option (command-line and config file) to specify the path that the wrappers should set. The main question, then, is whether this path-freezing should be done by default, or whether it should be an option. My inclination is to make it a user-controlled option, at least at first. Yup. IMO, this option shouldn't be boolean, but, rather, should specify the paths to add explicitly. Also, this technique has some hurdles to overcome for traditional distutils scripts (i.e., ones specified with setup(scripts=[...])), because they don't get .exe wrappers on Windows, and it's not even guaranteed that those scripts are Python code. OTOH, if PYTHONPATH freezing becomes popular, I suppose it's just another incentive for people to move up to entry-point scripts. ;) IMO, the scope should be limited to entry-point scripts. Thanks for improving this! Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Script-specific .pth files?
Here's a nutty idea I thought of when thinking about some of the recent back and forth. I realized, when thinking about plugins versus scripts is that a key use case is getting a script installed so that it will run. A script comes from an egg, which depends on other eggs, and so on. (Of course, this gets more exciting for complex script, like, say, Zope. :) It occurs to me that, when a script is installed, easy_install could generate a .pth file that named exactly those eggs the script needed. A wrapper script could be generated that used this .pth file. There wouldn't be any need for run-time analysis. The script wouldn't need to manipulate PYTHONPATH to find the needed eggs, as all of the needed eggs would be captured in the script-specific .pth file. Rather than determining the working set at run time, the working set would be determined at install time. I see a number of advantages to this approach: - No complicated run-time path gymnastics - Faster start-up, as there is no run-time searching for eggs or analysis of which eggs to use - More determinstic behavior. An installed script will always use the same eggs. The human who installed the script can inspect the .pth file to see exactly what's being used. They can even modify it if the feel so bold. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API for finding plugins
Phillip J. Eby wrote: I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements: * It should not cause anything to be imported before it actually needs to be used by the application * It should be able to deal gracefully with multiple installed versions of a project in designated plugin directories, falling back to older versions if a newer version's requirements can't be met * It should not actually add anything to sys.path until all plugins have been analyzed. The proposed API would be a 'find_plugins()' method added to the WorkingSet class, as follows: == find_plugins(plugin_env, full_env=None) Scan `plugin_env` and identify which distributions could be added to this working set without version conflicts or missing requirements. Example usage:: distributions, errors = working_set.find_plugins( Environment(plugin_dirlist) ) map(working_set.add, distributions) # add plugins+libs to sys.path print Couldn't load, errors# display errors The `plugin_env` should be an ``Environment`` instance that contains only distributions that are in the project's plugin directory or directories. The `full_env`, if supplied, should be an ``Environment`` instance that contains all usable distributions, *including* those listed in `plugin_env`. If `full_env` is not supplied, one is created automatically from the ``WorkingSet`` this method is called on, which will typically mean that every directory on ``sys.path`` will be scanned. This method returns a 2-tuple: (`distributions`, `error_info`), where `distributions` is a list of the distributions found in `plugin_env` that were loadable, along with any other distributions that are needed to resolve their dependencies. `error_info` is a dictionary mapping unloadable plugin distributions to an exception instance describing the error that occurred. Usually this will be a ``DistributionNotFound`` or ``VersionConflict`` instance. Most applications will use this method mainly on the master ``working_set`` instance in ``pkg_resources``, and then immediately add the returned distributions to the working set so that they are available on sys.path. This will make it possible to find any entry points, and allow any other metadata tracking and hooks to be activated. The resolution algorithm used by ``find_plugins()`` is as follows. First, the project names of the distributions present in `plugin_env` are sorted. Then, each project's eggs are tried in descending version order (i.e., newest version first). An attempt is made to resolve that egg's dependencies. If the attempt is successful, the egg and its dependencies are added to the output list and to a temporary copy of the working set. The process continues with the next project, and no older eggs for that project are tried. If the resolution attempt fails, however, the error is added to the error dictionary and the next older version of the plugin is tried, until a working version is found. Note that this algorithm gives precedence to satisfying the dependencies of alphabetically prior project names in case of version conflicts. If two projects named AaronsPlugin and ZekesPlugin both need different versions of TomsLibrary, then AaronsPlugin will win and ZekesPlugin will be disabled due to version conflict. == My question at this point is, is this algorithm sane? Depends on what you mean by sane. :) There seems to be an assumption that things get into the plugin directory outside of the application's control. Is that a good idea? Perhaps a better model would be for users of the application to install plugins one by one. The application can advise them of sucess or failure, let them know about conflicts and possible remedies and let the user decide what to do. I think this would be a better model. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API for finding plugins
Ian Bicking wrote: ... * At work we're using an environmental variable (ACTIVE_SITE) which changes sys.path (in sitecustomize) and the distutils installation options. This is like basing it on $HOME, but a bit more flexible (since $HOME has a lot of meanings). Note, fwiw: Zope 2 uses a similar approach. ZOPE_HOME points to a Zope software installation. INSTANCE_HOME points to a Zope site. Both are typically used to affect the Python path and to find other information such as configuration and data files. Are there other ways we can identify the user's intended working set? I don't think environmental variables are easy to work with on Windows, and it's a little opaque from a GUI user's perspective. $HOME isn't granular enough. Multiple scripts can work, but I've found it challenging to manage when the binaries in $PATH work, but potentially with bad side-effects (the discipline of keeping working sets separated isn't enforced, or even suggested by making it easier to do the right thing). Multiple scripts seems the most workable and transparent from the point of view of a GUI user, and not particularly bad from the perspective of a Posix command-line user either. I don't really follow what you mean by multiple scripts. [The more I think about it, the more I think site-packages in its entirety is pointless, distracting, and dangerous. But I don't think a PEP proposing its removal would be well received at this time ;) ] +1 on both counts. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] API for finding plugins
Phillip J. Eby wrote: At 07:15 AM 2/9/2006 -0500, Jim Fulton wrote: Phillip J. Eby wrote: I recently started work on adding egg support to Chandler ( http://chandler.osafoundation.org/ ), and ran into some interesting issues with respect to plugin discovery. Specifically, it's not easy to do it well with the APIs that pkg_resources currently offers. I suspect that others who've worked on plugin loading for application environments like Zope and Trac have probably run into similar issues. I'm proposing, therefore, to add a new API to pkg_resources to make plugin-finding easier. Among the requirements: I don't fully understand the goal here. From later discussion, I think you envision a model where people drop eggs into some directory and an application should be able to analyze this directory to determine which ones to use, meaning which to include in some working set. In addition to automatically determining the working set, the application should be able to find the entry points in that working set. Does that capture what you want to do? Yes, similar to the Zope 2 Basket product or Trac's plugin facility. OK, well I'll share the following intuition. I think it's important to distinguish, as I think you have, between application construction and pluggins. (I think Zope 2 blurred these too much.) I think these activities are performed by different people with different concerns. I think the people who install plugins are idiots, in the best sense of the word. :) Meaning people who are not technically savy or not technically interested. Therefore, I think plugins should be as simple as possible, providing well constrained functionality. As mentioned elsewhere, I think, pluggin installation should be an explicit operation under the control of the application. (This is not what Zope 2 does, btw.) It also relates to non-entrypoint metadata, like the resource system that Ian and I have been discussing. The goal there is to allow the located plugin eggs to offer translations, localizations, skins, etc. for other eggs to use. I expect this will be relevant to Zope also, so I'd appreciate your input there as well. I've been away from Zope 3 a little too long to know whether the ideas we're discussing can be made to work for it as well. I've read the thread a couple of times and am not entirely clear what your use cases are. I think it's all a bit too abstract for me. I suspect that Zope's component architecture is close to what you might want for a plugin registry. It is serving our needs in this area very well. A package advertises the faclities it provides through xml configuration files (ZCML). People installing a package elect whether to use the package configuration, or provide their own. Often, they do both, since ZCML supports overriding. This system is far from perfect and continues to evolve. We do have a lot of valuable experience in this area though. This sounds like a good topic for an Open Space discussion at PyCon. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working environment
-- easy_install will pick it up if invoked from there. But that doesn't work with setup.py develop, or setup.py install, or some other scenarios. I don't follow this. It seems to work for us, at least for setup.py develop. The main lamosity is depending on the current working directory. Personally, I'd like (at least optionally) the generated easy_install script to include information about the environment that was present when it was installed. That is, I'd be happy to see setup.cfg consuled when easy_install is installed and then ignored thereafter. If you wanted to change the environment definition, then simply update setup.cfg amd reinstall easy_install. Or, better yet, when installing easy_install, have the generated script remember the full path of the setup.cfg file used and reread that file when run, regardless of where easy_install is run from. This brings me to the topic of configuration. Today, I write wrapper scripts by hand, I may have some application like Zope, or ZEO or our test runner that is implemented by a an entry point in a module. Then there's a wrapper script that imports the module and calls the entry point. The wrapper script is written (manually or with some custom installation script) to include the path to be used and configuration data, which may be the location of a configuration file. I really like the fact that easy_install will generate wrapper scripts for me, but I really need more control over how these scripts are generated to include *both* path and configuration information. ... Another option is a completely new python interpreter bound to the environment. Basically the virtual-python.py option (http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python). In this model using env/bin/python indicate the proper environment, and you'd have local installs of *everything* including easy_install. This fixes so many problems without crazy hacks that it strongly appeals to me, especially if we can make it somewhat lighter. I get this /usr/lib/python2.4.zip on my path, that doesn't usually exist (does it ever get created by default); if we could create that somehow on demand and use such a big-bundle-zip, that seems lighter and faster and nicer. If we just put .pyc files in it, and those .pyc files refer back to the actual module source (in /usr/lib/python2.4/), then tracebacks should also still work, right? No actual symlinks either, so it should work on Windows. I'm not entirely sure where I'm going with this, though. I think that something much simpler can be made to work. I think a little more control over how scripts get generated would go a long way. (There is still the question of the interactive interpreter...) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working environment
Ian Bicking wrote: Jim Fulton wrote: ... lib/python2.4/ is for packages. Minor note: this needs to be flexible. I'd be more inclined to go with something shallower and simpler, like just lib, Why? Top-level packages aren't portable, since .pyc files aren't portable. Eggs are portable, since they contain the Python version. I have no idea what you are saying or how it relates to whether or not packages go in lib/python2.4 or lib. ... This brings me to the topic of configuration. Today, I write wrapper scripts by hand, I may have some application like Zope, or ZEO or our test runner that is implemented by a an entry point in a module. Then there's a wrapper script that imports the module and calls the entry point. The wrapper script is written (manually or with some custom installation script) to include the path to be used and configuration data, which may be the location of a configuration file. I really like the fact that easy_install will generate wrapper scripts for me, but I really need more control over how these scripts are generated to include *both* path and configuration information. I'm not sure what to think of this. I don't think of it as a script. It's like a specific invocation of the script. A shell script. Maybe we can improve on shell scripts, but I think it's a different idea than the script alone. What it are you talking about? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Script-specific .pth files (or baking paths into scripts)
I seem to remember a threat from from Phillip to make a proposal for installing script-specific ,pth files with scripts, but I don't think I every saw anything. Have I missed anything? If so, what? :) If not, I'll probably write something myself and share it. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Ian Bicking wrote: Jim Fulton wrote: I seem to remember a threat from from Phillip to make a proposal for installing script-specific ,pth files with scripts, but I don't think I every saw anything. Have I missed anything? If so, what? :) If not, I'll probably write something myself and share it. workingenv (http://svn.colorstudy.com/home/ianb/workingenv) doesn't use .pth files, but does put in script-specific paths. As I understand it, it is really putting in environment-specific paths. I think I want script-specific paths. As I note in my previous email (setuptools: hardcoding the path in a script and site.py), I think this would probably be cleanest to do through a new installation option, that could also be placed into distutils.cfg. Right now what workingenv does is very specific to workingenv, and doesn't work off any public setuptools API. I have an intuition that we're thinking about this incorrectly. We seem to be going through a lot of effort to fake out something, athough I'm not sure what. I have a growing suspiction that the traditional model of a large namespace of modules that is used by many applications serves simple scripting very well but doessn't work well for applications. I think this is an area where more prototyping is warrented, at least for my needs. I'll let y'all know what, if anything I come up with. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script-specific .pth files (or baking paths into scripts)
Ian Bicking wrote:
Jim Fulton wrote:
Ian Bicking wrote:
Jim Fulton wrote:
I seem to remember a threat from from Phillip to make a proposal
for installing script-specific ,pth files with scripts, but I don't
think I every saw anything. Have I missed anything? If so, what? :)
If not, I'll probably write something myself and share it.
workingenv (http://svn.colorstudy.com/home/ianb/workingenv) doesn't
use .pth files, but does put in script-specific paths.
As I understand it, it is really putting in environment-specific paths.
I think I want script-specific paths.
I don't think I understand the distinction? Specifically, this is what
gets put in:
#!/usr/bin/python2.4 -S
import sys, os
join, dirname = os.path.join, os.path.dirname
lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' %
tuple(sys.version_info[:2]))
sys.path.insert(0, lib_dir)
import site
# EASY-INSTALL-SCRIPT: 'Cheetah==1.0','cheetah'
__requires__ = 'Cheetah==1.0'
import pkg_resources
pkg_resources.run_script('Cheetah==1.0', 'cheetah')
I.e., it's hardcoding ../lib/python2.4 into the path. The relative
directory is right for workingenv (it makes it movable), but an absolute
location might be what you want. That's not very different, really.
Right. I understood this from the links you gave. Every program (script)
installed
into the environment has the same path. If two programs use the same package,
then upgrading the package for one program updates them all. I want to avoid
that.
...
I have an intuition that we're thinking about this incorrectly.
We seem to be going through a lot of effort to fake out something,
athough I'm not sure what.
Primarily I'm trying to fake out site.py,
That's one of the things we're trying to fake out. :)
...
I also want to fake out distutils slightly, mostly to provide my own
distutils.cfg, which is again a good idea that is far harder to use than
it should be.
Yup, that's another example.
And then I fake out setuptools, which I'd really rather not do, in part
because my faking out keeps breaking.
Faking isn't fun.
Oh, and faking out of #!, which if it wasn't so obnoxiously useless
would be sufficient to handle the path stuff on its own.
:)
I suspect this is a case of too much automation in the wrong place.
We are tiptoeing around trying to trick setuptools/distutils/site
into doing the right thing automatically.
I have a growing suspiction that
the traditional model of a large namespace of modules
that is used by many applications serves simple scripting very well
but doessn't work well for applications.
Absolutely, that's what I feel as well. I want to opt-in to modules,
not get some huge cloud. At this point, setuptools/easy_install
actually makes that cloud expand much more quickly with multi-version
installations, and that's caused real problems for me.
Yup.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Script processing plugins?
Phillip J. Eby wrote: Okay, so after today's discussions I'm still not really clear on how path freezing (baking, whatever) should work for standalone setuptools. Would it solve the problem for your use cases if I provided an option like --script-wrappers=plugin1,plugin2,..., where the plugin names are entry points that would get a chance to modify the contents of generated scripts? That way, you guys could experiment freely with your script-wrapping strategies without having to monkeypatch. And any tools that you build which wrap easy_install can simply add the necessary setting(s) to an appropriate .cfg file. That might be nice at some point. For now, I think it best if I do some prototyping sans monkeypatching. Hopefully, I'll get back to you soon. I started the discussion today mainly to get caught up, as I haven't had time to work on this for a while. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] What is the reason for requiring that an explicitly set egg-install installation is in the path?
If I explicitly specify an installation directory, for example, by specifying: [install] install_lib = ~/py-lib install_scripts = ~/bin in setup.cfg, setuptools gives me an error is the specified lib dir isn't in the Python path. Why is this? Is this for my protection? If so, I don't need a path nanny. :) I can understand this, to some degree, if --prefix is used, but if I specifically set an install location, setuptools should trust that I know what the heck I'm doing. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What is the reason for requiring that an explicitly set egg-install installation is in the path?
Phillip J. Eby wrote: At 10:08 AM 5/18/2006 -0400, Jim Fulton wrote: If I explicitly specify an installation directory, for example, by specifying: [install] install_lib = ~/py-lib install_scripts = ~/bin in setup.cfg, setuptools gives me an error is the specified lib dir isn't in the Python path. Why is this? Is this for my protection? If so, I don't need a path nanny. :) Maybe not, but everybody else does. ;) Says you. I can understand this, to some degree, if --prefix is used, but if I specifically set an install location, setuptools should trust that I know what the heck I'm doing. EasyInstall is telling you that as things stand, it can't accomplish what you asked it to -- specifically to install to that directory in such a way that the package is importable. If you use -m (aka --multi-version), then it will not care because getting stuff on the runtime path is then *your* responsibility, not setuptools'. The issue here is that when you install eggs without -m, the target directory must support .pth files. To do that, it must be on PYTHONPATH (and use a site.py hack) or it must be a site directory. You can assert that a directory is a site directory using --site-dirs, and EasyInstall will then believe you. In short, use -m or --site-dirs, and in either case EasyInstall will permit you to be responsible for the results, without nannying. But if you don't use either of those options, then it will want proof of your sanity (i.e. a correct PYTHONPATH) when installing to a non-site directory. :) How about an i_do_not_need_a_path_nanny in the config file? ;) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Why isn't a link given higher precedence than an egg
In an egg directory, I have: zc.buildout-0.1-py2.3.egg zc.buildout.egg-link and the version the link is pointing to is 0.1. If en environment is asked to pick the best match, it picks zc.buildout-0.1-py2.3.egg. Given that a link is the result of setup.py develop, I would expect it to get preference bacause it reflects current developemnt. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Bug in easy_install?
If I give easy_install
- an ordinary spec (e.g. demo)
- an -f option that points to a local directory containing a bunch of
eggs and that is not on sys,path,
- ask it to install to some directory (-d), and
- use the -m option
it doesn't copy eggs found to the install directory even though
the source directory isn't on sys.path. The logic for deciding whether
to copy seems to be:
# Installation is also needed if file in tmpdir or is not an
egg
install_needed = install_needed or os.path.dirname(download)
== tmpdir
install_needed = install_needed or not download.endswith
('.egg')
where install_needed starts out as false in this case because the
spec is not a url. The second part has no effect because the
value if the download variable ends in .egg.
The --always-copy documentation hints that distributions will be
copied unless they are found in a directory sys.path, but I see no
check.
Why are the eggs not copied?
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED]Python
Powered!
CTO (540) 361-1714
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Requirent specifiers specified? :)
When specifying a dependencies and when asking for a package with
easy_install, you can specify one or more specifiers. It's unclear what
the rules are for combining specifiers.
Imagine that I have a collection of eggs like:
/home/jim/tmp/dist:
used 92 available 41345796
-rw-rw-r-- 1 jim jim 671 Jun 19 17:43 demoneeded-1.0-py2.4.egg
-rw-rw-r-- 1 jim jim 672 Jun 19 17:46 demoneeded-1.1-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.2-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.3-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.4-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.5-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.6-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.7-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.8-py2.4.egg
-rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.9-py2.4.egg
Now, here are some examples of using the pkg_resources api to fetch a
required distribution:
import pkg_resources
e = pkg_resources.Environment(['tmp/dist'])
ws = pkg_resources.WorkingSet()
ws.resolve([pkg_resources.Requirement.parse('demoneeded ==1.1, ==1.4')], e)
[demoneeded 1.4 (/home/jim/tmp/dist/demoneeded-1.4-py2.4.egg)]
Here, the specifiers were or-ed. OK.
ws.resolve([pkg_resources.Requirement.parse('demoneeded 1.1, 1.6')], e)
[demoneeded 1.5 (/home/jim/tmp/dist/demoneeded-1.5-py2.4.egg)]
Here they were and-ed. This makes sense, from a dwimy point of view. :)
If they were or-ed, I'd expect to get 1.9.
ws.resolve([pkg_resources.Requirement.parse('demoneeded 1.1, 1.6,
==1.8')], e)
[demoneeded 1.8 (/home/jim/tmp/dist/demoneeded-1.8-py2.4.egg)]
Hm, here the ==1.8 seems to have been or-ed with the result of anding 1.1
and 1.6.
ws.resolve([pkg_resources.Requirement.parse('demoneeded 1.1, 1.6')], e)
[demoneeded 1.9 (/home/jim/tmp/dist/demoneeded-1.9-py2.4.egg)]
ws.resolve([pkg_resources.Requirement.parse('demoneeded !=1.8, ==1.8')], e)
[demoneeded 1.9 (/home/jim/tmp/dist/demoneeded-1.9-py2.4.egg)]
I really don't know what's going on with these. :)
Are the rules for combining specifiers specified anywhere?
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
Phillip J. Eby wrote: At 04:52 PM 6/21/2006 -0400, Jim Fulton wrote: ... I was going to say, of course, and give you a link, but I'm surprised to discover there's nothing I can find. Which is weird because I *know* I wrote something. I thought you did too. However I suspect it was originally on the PythonEggs wiki page, and it didn't get transferred to the PkgResources or setuptools pages before being deleted. Ah, yes, that's in fact the case. Here's the missing paragraph, grabbed from the wiki history: Requirement strings basically consist of a distribution name, an optional list of options (more on this in a moment), and a comma-separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of or = and or = conditions, or exactly matches a == or != condition. OK, this is something. I'm sure what ascending version order is. In ascending version order, what is the ordering of: 1.2, 1.2, ==1.2, =1.2, =1.2, and !=1.2 ? -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Optimizing easy-install upgrade
Suppose I have the directory: /home/jim/tmp/dist: used 92 available 41345796 -rw-rw-r-- 1 jim jim 671 Jun 19 17:43 demoneeded-1.0-py2.4.egg -rw-rw-r-- 1 jim jim 672 Jun 19 17:46 demoneeded-1.1-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.2-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.3-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.4-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.5-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.6-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.7-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.8-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.9-py2.4.egg and then run easy install telling it to install something I already have: [EMAIL PROTECTED] ~]$ /usr/local/python/2.4/bin/easy_install -d tmp/dist -mxU demoneeded==1.1 Searching for demoneeded==1.1 Reading http://www.python.org/pypi/demoneeded/ Couldn't find index page for 'demoneeded' (maybe misspelled?) Scanning index of all packages (this may take a while) Reading http://www.python.org/pypi/ Best match: demoneeded 1.1 Processing demoneeded-1.1-py2.4.egg Using /home/jim/tmp/dist/demoneeded-1.1-py2.4.egg Because this distribution was installed --multi-version, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require(demoneeded) # latest installed version pkg_resources.require(demoneeded==1.1) # this exact version pkg_resources.require(demoneeded=1.1) # this version or higher Note also that the installation directory must be on sys.path at runtime for this to work. (e.g. by being the application's script directory, by being on PYTHONPATH, or by being added to sys.path by your code.) Processing dependencies for demoneeded==1.1 Note that it looked at pypi even though it already had the only acceptable version. I'm not sure why it did this. Is this just a case of a missing optimization? Or is there some other reason for checking the index in a situation like this? I have some scripts that invoke easy_setup and I'd like to try to do some of this logic myself. Given a requirement, I'd like to get the specifiers and decide myself whether to invoke easy_install. I have 2 problems: - I don't want to parse the requirement myself, but, rather, use Requirement.parse. If I use Requirement.parse, I can use the specs attribute to get the specifiers, however, this attribute isn't documented. Should I assume that it is private? Or is it safe to use. - I'm still a bit foggy on how to interpret the specifiers. But we're discussing that in another thread. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 21, 2006, at 6:21 PM, Phillip J. Eby wrote: At 05:36 PM 6/21/2006 -0400, Jim Fulton wrote: In ascending version order, what is the ordering of: 1.2, 1.2, ==1.2, =1.2, =1.2, and !=1.2 ? It's actually unspecified, because it makes no sense to have more than one condition for the same version. What could the assortment of values you just listed possibly *mean*? I don't know. If it's illegal, then say so. The software` certainly accepts it now. I think it would be helpful to call this out in your specification. I also think it might be good to expand the explanation a bit, perhaps with some examples. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 21, 2006, at 5:11 PM, Phillip J. Eby wrote: ... Requirement strings basically consist of a distribution name, an optional list of options (more on this in a moment), and a comma- separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of or = and or = conditions, or exactly matches a == or != condition. I suppose that given: 1.2, 1.3 there is an implicit -infinity and infinity so that, for example, 1.9 falls between the pair 1,3 and infinity. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Optimizing easy-install upgrade
On Jun 21, 2006, at 6:27 PM, Phillip J. Eby wrote: At 05:37 PM 6/21/2006 -0400, Jim Fulton wrote: Suppose I have the directory: /home/jim/tmp/dist: used 92 available 41345796 -rw-rw-r-- 1 jim jim 671 Jun 19 17:43 demoneeded-1.0-py2.4.egg -rw-rw-r-- 1 jim jim 672 Jun 19 17:46 demoneeded-1.1-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.2-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.3-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.4-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.5-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.6-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.7-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.8-py2.4.egg -rw-rw-r-- 1 jim jim 673 Jun 19 17:46 demoneeded-1.9-py2.4.egg and then run easy install telling it to install something I already have: [EMAIL PROTECTED] ~]$ /usr/local/python/2.4/bin/easy_install -d tmp/dist -mxU demoneeded==1.1 The -U option means always search PyPI. I have some scripts that invoke easy_setup and I'd like to try to do some of this logic myself. Given a requirement, I'd like to get the specifiers and decide myself whether to invoke easy_install. I have 2 problems: - I don't want to parse the requirement myself, but, rather, use Requirement.parse. If I use Requirement.parse, I can use the specs attribute to get the specifiers, however, this attribute isn't documented. Should I assume that it is private? Or is it safe to use. Why do that when you can just ask the Requirement whether it matches a particular version? The __contains__ method of Requirement objects accepts a version string, distribution object, or parsed version number you can use. (For that matter, you can query an Environment or WorkingSet for the distributions whose versions you want to check.) Here's my use case: I want to get the most recent distribution that satisfies my requirement. If my requirement sets an upper bound, and I already have the distribution at the upper bound, then I don't want to have to search the index. For us, it will be very common to specify specific distribution versions. I don't want to have to search an index if I already have the specific version. I understand that the -U option isn't designed to meet this use case. That's fine. I want to be able to introspect a requirement to determine wether it set an upper bound. __contains__ doesn't let me do that. With the rules that you've explained for evaluating a set of specifiers, I can do that if I can get at the specifiers. What I want is either a public API for getting the specifiers, or an API that lets me retrieve the upper bound, if there is one. (Obviously, it could return None if no upper bound exists.) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Puzzling easy_install behavior
I'd like to share an experiment that I find puzzling and see if anyone could share some insight. Perhaps I'm doing something incorrectly. So, I have an links directory: [EMAIL PROTECTED] tmp]$ ls links demo-1.0-py2.4.egg demoneeded-1.0-py2.4.egg index.html [EMAIL PROTECTED] tmp]$ cat links/index.html htmlbody a href=demo-1.0-py2.4.eggdemoneeded-1.0-py2.4.egg/a a href=demoneeded-1.0-py2.4.eggdemoneeded-1.0-py2.4.egg/a /body/html and an empty eggs directory: [EMAIL PROTECTED] tmp]$ ls eggs Note that demo depends on demoneeded, without any qualifiers. I run easy_setup (0.6b3): [EMAIL PROTECTED] tmp]$ /usr/local/python/2.4/bin/easy_install -q \ -f file:///home/jim/tmp/links/index.html -m -deggs demo and get the eggs I expect. [EMAIL PROTECTED] tmp]$ ls eggs demo-1.0-py2.4.egg demoneeded-1.0-py2.4.egg I update the links with new versions of demo and demoneeded: [EMAIL PROTECTED] tmp]$ ls links demo-1.0-py2.4.egg demoneeded-1.0-py2.4.egg index.html demo-1.1-py2.4.egg demoneeded-1.1-py2.4.egg [EMAIL PROTECTED] tmp]$ cat links/index.html htmlbody a href=demo-1.0-py2.4.eggdemoneeded-1.0-py2.4.egg/a a href=demoneeded-1.0-py2.4.eggdemoneeded-1.0-py2.4.egg/a a href=demo-1.1-py2.4.eggdemoneeded-1.1-py2.4.egg/a a href=demoneeded-1.1-py2.4.eggdemoneeded-1.1-py2.4.egg/a /body/html I haven't changed the unqualified dependence on demoneeded. Now I rerun easy_install: [EMAIL PROTECTED] tmp]$ /usr/local/python/2.4/bin/easy_install -q \ -f file:///home/jim/tmp/links/index.html -m -deggs demo and I get a new demo egg even though I didn't elect to upgrade: [EMAIL PROTECTED] tmp]$ ls eggs demo-1.0-py2.4.egg demo-1.1-py2.4.egg demoneeded-1.0-py2.4.egg I find this a bit mysterious. I thought easy_install wouldn't search the find links and index if existing distributions meet a requirement unless --upgrade was used. Is special consideration given to file URLs? (If so, this makes test writing a bit more challenging.) If I supply the upgrade option, I get: [EMAIL PROTECTED] tmp]$ /usr/local/python/2.4/bin/easy_install -q \ -f file:///home/jim/tmp/links/index.html -m -U - deggs demo Couldn't find index page for 'demo' (maybe misspelled?) Scanning index of all packages (this may take a while) [EMAIL PROTECTED] tmp]$ ls eggs demo-1.0-py2.4.egg demo-1.1-py2.4.egg demoneeded-1.0-py2.4.egg So when I said upgrade, the index was checked, but when I installed the distributions initially, it wasn't. Why? Why is upgrade handled differently from an initial install. Also, it's a bit surprising that I didn't get the available upgrade for demoneeded. Is that intended? Are there plans for a recursive upgrade option? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Puzzling easy_install behavior
On Jun 22, 2006, at 11:38 AM, Phillip J. Eby wrote: nest is supposed to be part of setuptools 0.7, but 0.6 final isn't out yet. I don't know what nest is. Is it described anywhere? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Optimizing easy-install upgrade
On Jun 22, 2006, at 11:48 AM, Phillip J. Eby wrote: At 07:25 AM 6/22/2006 -0400, Jim Fulton wrote: I can get at the specifiers. What I want is either a public API for getting the specifiers, or an API that lets me retrieve the upper bound, if there is one. (Obviously, it could return None if no upper bound exists.) Use the specs attribute, which is a list of (op,ver) pairs in ascending version order (and an undefined order among pairs that have equivalent versions). The 'ver' is an *unparsed* version number, so you should parse it in order to do comparisons. The specs attribute is currently undocumented, but I'll document it. It has to be the way it is, because it's needed in order to produce a consistent string representation of a particular requirement. Cool. Thanks. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 22, 2006, at 12:49 PM, Phillip J. Eby wrote: At 12:26 PM 6/22/2006 -0400, Jim Fulton wrote: On Jun 21, 2006, at 5:11 PM, Phillip J. Eby wrote: Requirement strings basically consist of a distribution name, an optional list of options (more on this in a moment), and a comma- separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of or = and or = conditions, or exactly matches a == or != condition. I don't think this is right. :) ... When scanning left to right, 1.9 matches !=1.1, so it should match and, since it is the highest version, it should be returned. Either your description of the algorithm is incorrect or I'm misunderstanding it. You're missing the exactly matches part. The relevant context is: Until the package's version ... exactly matches a == or != condition. Perhaps making that a == or != condition's version would have been clearer, as that's what I meant by that phrase. Anyway, 1.9 does not exactly match the 1.1 in !=1.1. Um, OK. So I guess the idea is that we scan these things trying to make a decision. The decision is either match or not match. Is that how I was supposed to read the above quote? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 22, 2006, at 1:42 PM, Phillip J. Eby wrote: At 01:05 PM 6/22/2006 -0400, Jim Fulton wrote: On Jun 22, 2006, at 12:49 PM, Phillip J. Eby wrote: At 12:26 PM 6/22/2006 -0400, Jim Fulton wrote: On Jun 21, 2006, at 5:11 PM, Phillip J. Eby wrote: Requirement strings basically consist of a distribution name, an optional list of options (more on this in a moment), and a comma- separated list of zero or more version conditions. Version conditions basically specify ranges of valid versions, using comparison operators. The version conditions you supply are sorted into ascending version order, and then scanned left to right until the package's version falls between a pair of or = and or = conditions, or exactly matches a == or != condition. I don't think this is right. :) ... When scanning left to right, 1.9 matches !=1.1, so it should match and, since it is the highest version, it should be returned. Either your description of the algorithm is incorrect or I'm misunderstanding it. You're missing the exactly matches part. The relevant context is: Until the package's version ... exactly matches a == or != condition. Perhaps making that a == or != condition's version would have been clearer, as that's what I meant by that phrase. Anyway, 1.9 does not exactly match the 1.1 in !=1.1. Um, OK. So I guess the idea is that we scan these things trying to make a decision. The decision is either match or not match. Is that how I was supposed to read the above quote? Um, no. :) The specification is: 1. The version conditions you supply are sorted into ascending version order 2. and then scanned left to right (from low to high version number) 3. until the package's version 3a. falls between a pair of or = and or = conditions 3b. or exactly matches a == or != condition In both #3a and #3b, we are saying that the package's version is compared to the condition's version. If the version is *between* the version of a or = condition on the left, and the version of a or = condition on the right, then it is accepted. If the version exactly matches the version of a == or != condition, then it is either accepted (==) or rejected (!=). If neither #3a nor #3b happens, then we continue scanning left to right (per #2). This is the clearest thing I have seen so far and matches what I was trying to say. I suspect that you will next come back and ask about this: 1.0, !=1.2, 2.0 at which point I will remind you that #3a says nothing about the pairs being adjacent. :) I'm not sure what you are trying to say. I'm really trying to make sense of this. I assume you are alluding to what I assume is the case that that there is an implicit -infinity and infinity, so the above example becomes: -infinity, 1.0, !=1.2, 2.0, infinity Which isn't all that different from: 1.0, 2.0, 5.0, 6.0 so how is a case like this handled? How should fals between a pair of be interpreted? I would tend to expect innermost, otherwise: 1.0, 3.0, 5.0, 7.0 is ambiguous too. I suggest that a more careful specification is needed. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 22, 2006, at 1:56 PM, Jim Fulton wrote: I assume you are alluding to what I assume is the case that that there is an implicit -infinity and infinity, so the above example becomes: No, that can't be right. If that were so, then ==1.2 would be meaningless So I have no idea what you were trying to say. The more I find out about the specification specification, the less I understand it. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 22, 2006, at 3:00 PM, Phillip J. Eby wrote:
At 02:20 PM 6/22/2006 -0400, Jim Fulton wrote:
On Jun 22, 2006, at 1:56 PM, Jim Fulton wrote:
I assume you are alluding to what I assume is the case that that
there is an implicit
-infinity and infinity, so the above example becomes:
No, that can't be right. If that were so, then
==1.2 would be meaningless
So I have no idea what you were trying to say.
The infinities are implied only if they are needed to complete a
pairing with an actual condition.
The more I find out about the specification specification, the
less I understand it.
Exactly, so stop trying to understand it, and you'll see how
obvious it is. :)
Ah, if only it was.
The implementation scans from left to right until it's *sure* that
the version is either accepted or rejected. Each condition is a
point, a bound, or a point and a bound. If the version matches the
point part of condition, it's an exact match and you are *sure*
of an accept, unless it's a !=, in which case you're *sure* it's
a reject.
If the version falls below an upper bound ( or =), you are also
*sure* that it's accepted. If it is above an upper bound, it is
*tentatively* rejected.
So given:1, 3, 5, 7
You are sure that 4 is accepted?
Given: 2, 5
Is 3 accepted or rejected?
If the version falls below a lower bound ( or =), you are *sure*
that it's rejected. If it is above a lower bound, it is
*tentatively* accepted.
So given:1, 3, 5, 7
You are sure that 2 is rejected?
If you reach the end of the conditions without being sure of
anything, then your most recent tentative acceptance or rejection
is used.
A simple state machine is used to implement this:
state_machine = {
# =
'' : '--T',
'=': 'T-T',
'' : 'F+F',
'=': 'T+F',
'==': 'T..',
'!=': 'F++',
}
cmp() is used to determine whether the version is =, , or than
the condition's version, and the appropriate row and column is
pulled from the above table. T means sure accept, F means
sure reject, + means tentative accept, and - means
tentative reject. (. means don't care.) The state machine
simply compares versions until its sure or there are no more to
compare.
I don't understand the meaning of the values in the dictionary above.
Do the character positions reflect states somehow?
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED]Python
Powered!
CTO (540) 361-1714
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 22, 2006, at 4:08 PM, Jim Fulton wrote:
...
A simple state machine is used to implement this:
state_machine = {
# =
'' : '--T',
'=': 'T-T',
'' : 'F+F',
'=': 'T+F',
'==': 'T..',
'!=': 'F++',
}
cmp() is used to determine whether the version is =, , or than
the condition's version, and the appropriate row and column is
pulled from the above table. T means sure accept, F means
sure reject, + means tentative accept, and - means
tentative reject. (. means don't care.) The state machine
simply compares versions until its sure or there are no more to
compare.
I don't understand the meaning of the values in the dictionary
above.
Do the character positions reflect states somehow?
It's a truth table: the rows are condition operators, and the
columns are cmp() results. It is simply a transcription of the
rules about points and bounds that I spelled out verbally, reduced
to a table lookup on the condition and the comparison results.
OK, than makes sense. So, with the requirement: 1, 3, 5, 7
So let's see, with 4, we get +-F and we reject it. OK, that makes
sense.
The state machine helps a lot. My question is now answered.
I think that the fact that you need to understand a non-trivial
algorithm
with a state machine to understand how non-trivial specifications are
interpreted is a problem. Maybe it's enough to tell people don't
use complex specifications, but maybe it would be better to use a
simpler system.
I also think that the complete algorithm, including the state machine
needs to be documented clearly. Your original paragraph really
isn't adequate.
Maybe, you should document simple cases and refer to the full
complex model for non-trivial cases.
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED]Python
Powered!
CTO (540) 361-1714
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
I think you are making a mistake to assume that everyone has the same notion of obvious that you do. For example, someone could reasonably expect that ==1.0 matches 1.1.1. (I'm not advocating this interpretation, but simply pointing out that obvious is not universal.) In addition, people sometimes make typos. A system that figures out what they really meant rather than complain when they enter something that doesn't make sense isn't doing them any favors. In any case, I expect that having people build tools on top of setuptools is a use case you anticipated. For people to do that, they sometimes need precise specifications of behavior. Jim On Jun 22, 2006, at 4:36 PM, Phillip J. Eby wrote: At 04:08 PM 6/22/2006 -0400, Jim Fulton wrote: So given:1, 3, 5, 7 You are sure that 4 is accepted? Scanning left to right, 1 is a lower bound and 4 is above it, so that's a tentative accept. Next we see 3, and it is an upper bound, and we fall below it, so it is a sure accept at that point, and we stop scanning. Huh? 4 is below 3? Hm, somehow I got confused with 2. It should say, 3 is an upper bound, 4 is above it, tentative reject. 5 is a lower bound, 4 is below it, it's a sure reject. I think I copied 2 in from another example while editing. Given: 2, 5 Is 3 accepted or rejected? 2, upper bound, 3 is above, tentative reject. 5, upper bound, 3 is below it, sure accept. (Intuitively -- at least for me -- 5 is paired with -infinity here.) OK. I think that many people would find this non-obvious. You keep missing the big picture. The algorithm is designed to interpret a *meaningful* list of versions *specified by a human*. Humans are usually not excessively redundant, and will thus tend to express a version requirement in the form of ranges with exceptions. Humans tend to think that it is obvious that if they ask for a version 5, they do not need to also specify that the version be more than negative infinity, or even that it be greater than zero. It is, after all, a *version* number. :) Similarly, if I ask for a version 10, I assume that you will understand I want it to be less than negative infinity. :) If it weren't for the fact that '.' and '-' are often used in version numbers, I might have used range expressions using '-' or '...', but these are also harder to read for the most common case where you want '=someversion'. Ah, so in your explanation above, f the version falls below a lower bound only applies to a scanning position. OK, that clarifies this case. If you look back at my previous descriptions, the concept of scanning is repeated many times, including the simple 1-paragraph description. It's all scanning, just like a human would do when reading a series of version requirements. I think that the fact that you need to understand a non-trivial algorithm with a state machine to understand how non-trivial specifications are interpreted is a problem. It's only a problem if you try to reduce it to an algorithm. If you simply try to *understand* a version requirement, it's really quite straightforward. Think of what the human who wrote the requirement is saying: Um, let's see, it should be less than version 3, because that version's got a new API, and it should be more than version 1.2, because there were some bugfixes, and oh yeah, don't use version 1.6 because it was just plain hosed. Oh, and 1.1.3 is also good. The algorithm is designed to correctly interpret a person who is simply recording their wishes in this manner -- describing a series of acceptable or unacceptable ranges, and mentioning specific versions that are exceptions to the ranges. Maybe it's enough to tell people don't use complex specifications, but maybe it would be better to use a simpler system. People don't normally express themselves redundantly -- especially programmers. The target users of specifications are these humans, not the programmers trying to interpret the specification for specifications. I did not anticipate that particular user and use case. ;) -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Requirent specifiers specified? :)
On Jun 22, 2006, at 6:51 PM, Phillip J. Eby wrote: At 04:54 PM 6/22/2006 -0400, Jim Fulton wrote: In any case, I expect that having people build tools on top of setuptools is a use case you anticipated. For people to do that, they sometimes need precise specifications of behavior. Of course -- but I didn't anticipate that they would need anything more than the ability to test whether a version matches a requirement. I think I've explained, and I thought you understood why that wasn't adequate for me. ... Also, at some point, one does what one can with the available manpower. Bob Ippolito was the only person besides me who was actively involved in the design process at the time -- and he was kind of like, whatever, man on the subject. :) It would've been nice to have had your participation then. Now, we'll pretty much have to live with what we've got. 1. You should know that no system is perfect. You can't anticipate all requirements up front. Surely, you want people to use this system beyond those who invented it. It's also not practical for all of us to get involved in every design at it's inception. 2. I have tried not to ask you to implement my use cases. There are a number of reasons for this, including: - I don't know fully what my use cases are. After all, use cases change with experience. - I don't want to disrupt setuptools development. I don't think setuptools should try to meet everyone's needs. I do think that people should be able to meet their own needs on top of setuptools. I think setuptools has stood up very well in this regard. I really haven't asked for much. I believe the only changes you've made for me so far have been bug fixes. I don't intend to ask for much. The only non-bug-fix change I've asked for so far has been to declare public a pre-exiting API so that I can implement what I want myself. Of course, I've also asked that some semantics be documented. I needed this documentation for what I was doing. I also believe that documenting things well sheds important light on them. You learn things about decisions to make by explaining these decisions to other. I think setuptools is pretty cool and I have sung it's praises in many forums, admittedly, even before I know what I was talking about. ;) I don't regret that. I'm building a buildout system on top of setup tools that will help it meet use cases that it doesn't meet now. I appreciate your advice and support during this process process and look forward to more of the same in the future. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Is there any harm in clearing sys.path_importer_cache from time to time?
I have a script based on setuptools that invokes easy install to check for and download newer distributions. I allow users to specify whether or not they want unzipped downloads. As a result, easy_install will sometimes replace a zip file with a directory or the other way around. This leads to problems in the invoking program because the sys.path_importer_cache is stale. Does anyone know if it is safe to clear the cache from time to time? If so, I'd be inclined to clear it any time I invoke easy_install. If it isn't safe, I'll probably just forego the feature of letting users force downloads to be unzipped, which I may do anyway to be safe. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is there any harm in clearing sys.path_importer_cache from time to time?
On Jun 23, 2006, at 11:53 AM, Phillip J. Eby wrote: At 11:37 AM 6/23/2006 -0400, Jim Fulton wrote: I have a script based on setuptools that invokes easy install to check for and download newer distributions. I allow users to specify whether or not they want unzipped downloads. As a result, easy_install will sometimes replace a zip file with a directory or the other way around. This leads to problems in the invoking program because the sys.path_importer_cache is stale. Does anyone know if it is safe to clear the cache from time to time? If so, I'd be inclined to clear it any time I invoke easy_install. Hm. easy_install should be doing this itself... oh wait, it's clearing the *zip directory cache*, which fixes the problem of overwriting a zip with a zip. You're having a problem when overwriting a zip with a directory. Now I understand why sometimes I still get reports of behavior that resembles this problem, but then they aren't reproducible. Actually, I happened to be having a problem overriding a directory with a zip. This was in a test in which I demonstrate being able to unzip and went back to not unzipping in a later example. After the egg is reinstalled unzipped, the cache entry has None, left over from finding the directory before. This was in a doctest which invoked my buildout script as a subprocess, which then invoked easy_install as a subprocess. When the buildout script started, the egg in question was a directory. It then ran easy_install, which replaced the directory with a zip file. Then the buildout script tried to load the distribution for the installed file, which failed. I haven't been able to manually find the right combination to make easy_install override a distribution manually, so I'm a bit puzzled. If you wish me to pursue that I will. I'll fix easy_install to also clear the path importer cache for a path entry when it clears the zip directory cache for that entry. OK. Note that that won't help me, because the cache I'm affected by is in the parent process. (And to answer your general question, it's fine to clear the path_importer_cache, per PEP 302's advice to do so whenever you add new import hooks. It just slows down the next import search a bit.) Cool, cause that solves my immediate problem. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Specification for package indexes?
What form must an index take to be usable with setuptools? Is there anything documented how such a beast should be organized? What should it's pages should look like? Is there any special pattern setuptools (easy_install?) looks for to find pages with download links? Or does it search any link given? Also, I've noticed that if you gibe a location with find-links that has links to distributions for the thing you're looking for, it you can specify pretty much anything as an index. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for package indexes?
On Jun 23, 2006, at 4:30 PM, Phillip J. Eby wrote: At 04:10 PM 6/23/2006 -0400, Jim Fulton wrote: What form must an index take to be usable with setuptools? Is there anything documented how such a beast should be organized? What should it's pages should look like? Is there any special pattern setuptools (easy_install?) looks for to find pages with download links? Or does it search any link given? Also, I've noticed that if you gibe a location with find-links that has links to distributions for the thing you're looking for, it you can specify pretty much anything as an index. That's correct. If all your packages can just be linked from a single page, that's more than enough to satisfy. However, if you want to give easy_install an alternate --index-url to use in place of PyPI, it must meet the four simple requirements described here: http://mail.python.org/pipermail/catalog-sig/2005-June/000654.html These are the only conditions that easy_install has for the organization of an index in the PyPI sense. Note that they can all be accomodated via static HTML pages. That's a lot of screen scraping. :) It would be good to capture this as part of the documentation IMO I'm considering adding XML-RPC support to easy_install in 0.7, though. PyPI now has a nice XML-RPC API that is more responsive than the web UI, and it supports case-insensitive partial match searches, making it suitable for easy_install to query when a typed- in name doesn't exactly match the spelling of a PyPI entry. I think that would be much better. I think it would also be helpful to have an option (-I) to disable index search. There will be cases where people don't want to put distributions in PyPI and won't want to have to implement an index server. The find-links mechanism seems to work fine most of the time. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] setup.py develop ignores --no-deps option
AFAICT, with 0.6b3, setup.py develop ignores the --no-deps option. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] README(.txt)?
If I don't have a README or README.txt file next to my setup.py file when I create an edd, I'll get the warning: warning: manifest_maker: standard file not found: should have one of README, README.txt I don't like warnings. So I create a REAME,txt file and then generate an egg and, guess what? It's not included. Bug or feature? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Best practices for creating eggs?
Has anyone written up any best practices for creating eggs? How do people handle documentation? Our packages tend to have documentation files included as doctests, but It's not clear how folks are expected to get to them, especially if the eggs are zip files. -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Weird PyPI Behavior
If I do something like: python2.4 setup.py register bdist_egg upload as recommended in the setuptools documentation, I end up with 2 releases in PyPi. One has all my descriptive information and the other has the egg. One or more will or won't be hidden according some rule that I haven't been able to divine. If the release with the meta data (e.g. description, author, etc.) is not hidden, then easy_install can't seem to find the eggs. At least this is true today. I thought it wasn't true yesterday, but I don't trust my recollection. If I hide the release with the descriptive information, then easy_install can find the eggs, but most of my meta data isn't shown. Has anyone else seen this behavior, or am I just insane. For now, I'm not uploading eggs to PyPI, which is a shame, because setuptools sure made it convenient. :( Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] README(.txt)?
On Jun 28, 2006, at 4:56 PM, Phillip J. Eby wrote: At 04:37 PM 6/28/2006 -0400, Jim Fulton wrote: If I don't have a README or README.txt file next to my setup.py file when I create an edd, I'll get the warning: warning: manifest_maker: standard file not found: should have one of README, README.txt I don't like warnings. So I create a REAME,txt file and then generate an egg and, guess what? It's not included. Bug or feature? The warning comes from the distutils, in a method that I call to create a source manifest. I don't know of any way to suppress it, although I'll look again. In normal distutils usage, you would only see it when doing an sdist, as it wants a README for the source distribution. Is the right thing to do to suppress the warning? Or to include the README in the egg? I'm not clear on what the README is for. If it contains installation instructions for a source distribution, then obviously, it shouldn't be included. FWIW, this is most annoying to me when doing setup.py develop. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Weird PyPI Behavior
On Jun 28, 2006, at 5:30 PM, Phillip J. Eby wrote: At 05:16 PM 6/28/2006 -0400, Jim Fulton wrote: Interesting. I thought that --tag-svn-revision was a common/ recommended practice. It *is* -- but not for releases being sent to PyPI. a dev-r release is a development snapshot rather than an official release. I followed the advice to have a setup.cfg like: [egg_info] tag_build = .dev tag_svn_revision = 1 But I noticed that the release with the meta data didn't have the subversion information or the .dev tag, which is consistent with register not being aware of the extra information. Is this a bug? :) Probably. :) A workaround would be to either put egg_info before register, or to move the register after the bdist_egg. I'll add this to the bug list nonetheless. Interesting. That's good to know. If it works, I'm happy. I suggest noting this in the docs. If it isn't, then I recommend that you either stop recommending that people use setup.cfg this way or stop recommending that they upload their files to PyPI. Do you really need to upload every development snapshot to PyPI? If I want people to be able to download it, I have to upload it somewhere. I have two choices, upload it to pypi, or create my own download area. I would do one or the other, not both. It seems confusing, to me and to consumers to have two download areas, my own and pypi's. I'd rather use PyPi's because it's integration with setup.py is so convenient. OTOH, if it is considered bad form to upload dev releases to PyPi, I can just write helper scripts to upload somewhere else. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Weird PyPI Behavior
On Jun 28, 2006, at 6:09 PM, Phillip J. Eby wrote: At 05:42 PM 6/28/2006 -0400, Jim Fulton wrote: If I want people to be able to download it, I have to upload it somewhere. Not necessarily. If you have a pure-Python package, and your target audience has Subversion, you can provide a URL on PyPI that always obtains the most recent possible development snapshot. For example, easy_install setuptools==dev will check out the setuptools trunk and install an egg from it, properly tagged as a development release with the subversion revision. easy_install setuptools==dev06 will check out the head of the 0.6 maintenance branch. The docs are rather unclear how to to set this up. In any case, it isn't my intent to have people track subversion Of course, if you want to provide *built* daily snapshots, then this doesn't help you. Or to do daily builds. I have two choices, upload it to pypi, or create my own download area. I would do one or the other, not both. It seems confusing, to me and to consumers to have two download areas, my own and pypi's. I'd rather use PyPi's because it's integration with setup.py is so convenient. OTOH, if it is considered bad form to upload dev releases to PyPi, I can just write helper scripts to upload somewhere else. I don't know if there's a consensus that it's bad form. I do think that it would quickly become annoying to readers of Planet Python or even just the PyPI RSS feed if every Zope project was churning out snapshots of every SVN checkin, every day. :) I didn't realize that this information was being broadcast. I imagine I am unintentionally annoying people with all of my experiments. :( It isn't my intention to provide daily builds. I just happen wanted to make quick and dirty releases while stuff is new. These are like alpha release, but I'm leveraging the revision tagging to automate generation of release numbers. As it is, there are lots of projects that churn their PyPI releases more often than I'd prefer to see going by, but ah well. Moderation in all things is best, I suppose. :) I won't say you *should* do one thing or the other, but I will note that the rotate command of setuptools is meant to help clean up directories containing automated nightly builds or svn snapshots. So, if you have servers whose job it is to do continuous builds, they can do stuff like bdist_egg -b /output-dir rotate -k5 -m.egg - d /output-dir to put the built egg in /output-dir and keep the 5 newest eggs. Several projects can safely share the same output directory, and the output directory doesn't need to be served by anything fancy; see e.g. http://peak.telecommunity.com/snapshots/ for an example of such a directory. That's not what I'm up to. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] How does one register a framework classifier?
How does one register a new Framework classifier? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Weird PyPI Behavior
On Jun 28, 2006, at 11:28 PM, Phillip J. Eby wrote ... It isn't my intention to provide daily builds. I just happen wanted to make quick and dirty releases while stuff is new. These are like alpha release, but I'm leveraging the revision tagging to automate generation of release numbers. Hm. Well, different strokes for different folks. I prefer to bump the alpha or beta number, since a short number is easier for humans to refer to; it's annoying to have to ask if someone has 0.6a9dev- r19684, versus something like 0.6a9snap3. The latter can be done with tag_build='snap3' in your setup.cfg, modulo the register issue we've already discussed. But of course your practices are up to you; I just mention this as an idea to make communications about your projects a little easier, at the cost of editing setup.cfg (or overriding --tag-build on the command line) when you do a release. I'm happy to get input. 1. If I'm going to edit setup.cfg, I might as well edit setup.py 2. A reason I want to automate this is a feat that I'll forget to edit setup.py or setup.cfg and overrite existing releases (or fail to do an update without realizing it because the existing release doesn't get overwritten. Another issue is that I see us moving toward lots of fairly fine-grained packages and I want to keep the ceremony pretty low. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How does one register a framework classifier?
On Jun 29, 2006, at 6:52 AM, Kevin Dangoor wrote: On Jun 28, 2006, at 7:30 PM, Jim Fulton wrote: How does one register a new Framework classifier? FWIW, I've been using keywords a lot, because I can add arbitrary ones whenever I wish and still do searches. TurboGears has a package listing (the CogBin) which is built from Cheeseshop queries. Sure, but I assume that TurboGears also uses classifiers since 4 out of the 5 Framework classifiers are for TurboGears. Well, actually, 3 out of the 5. One is for TruboGears, which, I suppose, if for a related framework. ;) If we think that classifiers shouldn't be used for Frameworks, then lets get rid of the existing Framework classifiers. Otherwise, there needs to be a way to add new ones as new frameworks emerge or old frameworks make more use of PyPI. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] Specification for package indexes?
On Jul 7, 2006, at 4:20 PM, Phillip J. Eby wrote: At 02:52 PM 7/7/2006 -0400, Jim Fulton wrote: ... Perhaps someone should propose an API and we'll see. :) I thought I already did. :) Here it is again: baseURL/ should return a page containing href links to projects baseURL/projectname should return a page containing href links to version pages baseURL/projectname/version should return a page with download links (ideally with MD5 info) Links are found via href= attributes URLs' trailing path components are used to identify distributions. Hm. I hadn't seen this before. Perhaps I'm missing some messages from this thread. By download links, do you mean links to distributions? Or to links to pages containing links to distributions. Can the links to projects, links to version pages, or download links point off site? Can any of these pages contain other links? This is a sufficient API to allow querying packages for downloading purposes, as long as all download links are found in the index's pages. Additional information is only needed to allow following external links to *other index pages*. so, for example: http://www.python.org/pypi/ZODB3/3.6.0 Has a link to http://www.zope.org/Products/ZODB3.6. Is this a download link? Or an off-site index link. I'm having a little trouble following the jargon. setuptools has a notion of an index. That notion is not at all well defined. It's mostly operationally defined in terms of what PyPI did when it was written. Right, not well defined. :) I'm not criticizing. What it does was great as a prototype, but it would be good move beyond this. Currently, the index has linkes that are followed to find package links elsewhere. This seems reasonably useful. I dunno. I'm not sure I care. What I do care about is that the index API should be well defined so that we can implement alternate indexes and alternate tools to read indexes. I'm not looking to satisfy use cases beyond what we have now. Sure. I'm just saying we only need something beyond href= links if they are intended to be followed by tools looking for package links. The reason this is necessary, is that it's not sufficient to just follow links that point outside the package index; PyPI has links on its pages that go to other parts of python.org, so there needs to be something that distinguishes links that might help find downloads. Links that *are* downloads are detected via URL content. Right. That's why I think the hrefs we care about should be marked with class attributes or some such. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Mystery solved
On Jul 10, 2006, at 5:50 PM, Phillip J. Eby wrote: ... 2. The messages: Couldn't find index page for 'jimtest' (maybe misspelled?) Scanning index of all packages (this may take a while) should be info messages, not warning messages. If they remain warnings, and you fix the first problem, it will be impossible to avoid the warnings without creating a PyPI project or creating an index server, and I don't think it was your intent to require either of these. I don't think a warning should be issues for correct use of software. Here's the problem. Reducing everything to info messages means there's effectively no control over output detail. I generally use 'warn()' for things that *may* reflect an error in input parameters. So, my take on the above is that although the Scanning message could become an info(), the previous one shouldn't. This means that one always has to use an index. In which case, what is the point of find-links? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Mystery solved
On Jul 11, 2006, at 11:25 AM, Phillip J. Eby wrote: At 03:54 AM 7/11/2006 -0400, Jim Fulton wrote: On Jul 10, 2006, at 6:22 PM, Phillip J. Eby wrote: Here's the problem. Reducing everything to info messages means there's effectively no control over output detail. I generally use 'warn()' for things that *may* reflect an error in input parameters. So, my take on the above is that although the Scanning message could become an info(), the previous one shouldn't. This means that one always has to use an index. In which case, what is the point of find-links? The point of --find-links is to provide links to unindexed packages. But if you use an unindexed package, you'll get a warning. Which means it *may* be an error. See above. If it was an error, it would be an error. The point of a warning is to inform you that something *may* be an error. IMO, you should not get a warning for correct use of software. Users should try to make warnings go away. Repeating it doesn't make it so. I'm not convinced that this particular warning (that you may have misspelled the package name because it's not in the index you're using) is in any way harmful. It is definitely so. That is definitely my opinion. :) OK that's an interesting point wrt possible misspellings. If you can find the package via the find links, but not via the index, that seems to me to be a pretty good indication that this is not a misspelling. This is the case I'm worried about. If the package can't be found anywhere, then I agree that a warning is warranted. If you give people warnings that they shouldn't make go away, That wasn't clear. If people are using the software correctly, but choosing to find distributions via find-links rather than an index, and they get warnings, then they will always get warnings and tend to ignore them. Huh? They can put the package in the index, use a different index, not use -U, specify an exact download URL (either directly or via -- find-links), etc. There are a huge number of ways *not* to encounter that particular warning. I have to use -U to get newer versions of distributions, even if I happen to store distributions in a directory that is not a valid index. In this case, I use find-links and -U together, and I'll get a warning unless I put distributions in an index. Either you can't have valid unindexed software, or setuptools shouldn't generate a warning if software isn't in the index. This is back to argument by assertion. Please explain to me why this warning is actually bad, rather than simply asserting that it's so. I assert and take as a premise that when users are using software correctly, including not misspelling anything, they should not get a warning. If you can't buy that, then we have an unreconcilable difference. The specific case, which I'll repeat from above, as clearly as I can, is this: - A user chooses not to store their software in an index. - The user places distributions on a web server somewhere. This is just a directory, it is not a valid index. - The user points at their server using find-links - The user has an installation and they want to check for newer versions. - The distributions that they are looking for newer versions of can be found on the server that they name via find-links. In this case, they will get a warning that the distribution they are looking for couldn't be found on the index. They didn't misspell anything, as setuptools should be able to deduce from the fact that their distribution was found on the link server. I don't think that they should get a warning. As far as I'm concerned, this means that distributions must always be stored on index servers and the find-links is just an attractive nuisance. I really find the distinction between indexes and find-links rather puzzling. --find-links is used to allow you to point easy_install to a project's non-indexed home page or download page to find links, or to provide other easy_install-processable links, without needing an index. But they are unusable without getting warnings whenever you want to check for updates. Personally, I'd like to find a way to merge these two concepts into one by choosing a definition of an index that admits a directory full of distributions. Feel free to try to come up with one. However, --find-links allows *multiple* links to be specified, and it is also the basis for the dependency_links argument to setup(). --find-links is also a primitive upon which the index facility is built, since index pages are treated more-or-less like --find-links URLs that are automatically generated. I don't need to, you already did At a minimum, merging the concepts would mean allowing multiple index URLs, or else eliminating the idea of an index, Yup. Sound good to me. and treating all --find-links URLs as though
Re: [Distutils] Mystery solved
On Jul 11, 2006, at 2:07 PM, Phillip J. Eby wrote: At 11:50 AM 7/11/2006 -0400, Jim Fulton wrote: OK that's an interesting point wrt possible misspellings. If you can find the package via the find links, but not via the index, that seems to me to be a pretty good indication that this is not a misspelling. This is the case I'm worried about. If the package can't be found anywhere, then I agree that a warning is warranted. The interesting question there is, should the fallback scan still take place in the absence of the warning? If it *does* take place, then the reason for the scan (and delay) is unexplained. If it does *not* take place, then there is an undesirable change in semantics. Currently, if you have a package called Bob's Incredible Package, this will be treated by easy_install as being spelled Bob-s- Incredible-Package, and it will require a top-level index scan to find the right URL. It is also possible to have --find-links pages containing obsolete versions, while PyPI contains the latest version, so removing the scan doesn't seem to be a reasonable option. So, I will simply change the message to an info message stating that the index page couldn't be found (rather than a warning suggesting misspelling), *if* easy_install has previously seen at least one valid distribution file or link for the applicable project name. Great! The specific case, which I'll repeat from above, as clearly as I can, is this: - A user chooses not to store their software in an index. - The user places distributions on a web server somewhere. This is just a directory, it is not a valid index. - The user points at their server using find-links - The user has an installation and they want to check for newer versions. - The distributions that they are looking for newer versions of can be found on the server that they name via find-links. In this case, they will get a warning that the distribution they are looking for couldn't be found on the index. Okay, this scenario is fixed by changing to an info message as described above. Yup. Cool. If you did that, however, it brings in the question of which of the --find-links URLs should be checked for a /projectname/ subdirectory. All of them? Just the first one that finds a result? None of them, if some other criterion is met? I would stop when a result is found. Even so, this means O(N x M) web hits, where N is the number of packages and M is the number of --find-links (including dependency links supplied by eggs installed so far). I don't think it's reasonable to hit so many non-existent URLs on non-index servers, and is impolite to the servers' operators. (For example, if they receive a daily report of all 404 errors from their web servers, as I do. This is pretty common on Red Hat boxes using logwatch, for example.) It's particularly unfair since using e.g. http:// peak.telecommunity.com/snapshots/ as a --find-links while installing, say TurboGears, would cause a whole host of index hits to subdirectories of that URL, even though none of them can or will be found. The fallout from this approach is far worse than any screen scraping issues we've had. Isn't this the approach that's followed now? Aren't all of the find- links searched as well as the index? I suppose you're referring to the search for /projectname, which potentially doubles the number of requests. What is the use case for spreading distributions over multiple servers? Do people really want to do that? I can see providing multiple places to look, because different distributions might be on different servers, but I don't see why distributions for a single project should be spread over multiple servers. Platform-specific distributions may be provided by contributors to a project, rather than by the project's author; see, for example, Bob Ippolito's pages for distributing Mac OS X builds of popular Python packages. For this reason, you may have certain pages that you always want included in your --find-links, to be checked in addition to the normal indexes. OK Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Passing build_ext arguments to easy_install?
I need to install a package that depends on an external non-python library. I need to be able to specify the locations of the library's include and library files. Normally, when invoking setup.py myself, I'd pass these using the --include-dirs and --library-dirs options. Is there any way to pass this information to easy_install? Note that the package in question doesn't use setuptools, but I want an egg as a result. The only way I know how to do that is with easy_install. One hack I can think of is to use --editible and --build-directory to download the package, write a setup.cfg into the resulting directory and then run easy_install on the result. I would hope though that there is a better way. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Passing build_ext arguments to easy_install?
On Aug 8, 2006, at 11:58 AM, Phillip J. Eby wrote: At 10:43 AM 8/8/2006 -0400, Jim Fulton wrote: One hack I can think of is to use --editible and --build-directory to download the package, write a setup.cfg into the resulting directory and then run easy_install on the result. I would hope though that there is a better way. :) Not at the moment, no. However, you could subclass easy_install and override the 'run_setup()' method to edit the setup.cfg first. The setuptools.command.setopt module includes some nice routines for editing setup.cfg files, so that you won't even overwrite settings you don't care about. OK, I'll look at that when I get a chance. Thanks. Would you be interested in some sort of patch for 0.7 to make it possible to pass this information to easy_install? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Passing build_ext arguments to easy_install?
On Aug 8, 2006, at 12:15 PM, Phillip J. Eby wrote: At 12:02 PM 8/8/2006 -0400, Jim Fulton wrote: On Aug 8, 2006, at 11:58 AM, Phillip J. Eby wrote: At 10:43 AM 8/8/2006 -0400, Jim Fulton wrote: One hack I can think of is to use --editible and --build- directory to download the package, write a setup.cfg into the resulting directory and then run easy_install on the result. I would hope though that there is a better way. :) Not at the moment, no. However, you could subclass easy_install and override the 'run_setup()' method to edit the setup.cfg first. The setuptools.command.setopt module includes some nice routines for editing setup.cfg files, so that you won't even overwrite settings you don't care about. OK, I'll look at that when I get a chance. Thanks. Would you be interested in some sort of patch for 0.7 to make it possible to pass this information to easy_install? Do you mean via the command-line interface? Yeah. Although, ultimately, I want to escape the command-line interface. I'd still much rather use a Python API. I'll need to understand the internals a lot better. I expect looking at the things you suggested will help me do that. Generally, the farther I go and the more I learn of setuptools, the less I actually reply on easy-install. One thing I originally thought about doing was trying to have some way for the child setup to inherit settings from the easy_install command; that is, to copy default options from the parent process to the subprocess (not really a subprocess), perhaps by editing the target's setup.cfg. But maybe for what you want, all you need is something like: easy_install --build-commands=build_ext --blah fah wah foo whatever The big question there of course is whether those commands should also be applied to dependencies. Probably not. For something like include and library dirs, it's probably not a problem, I'm just wondering about the general applicability. Yup. All good points. It's not obvious how to expose this. I think what's needed is a way to say more about how a particular package should be handled. This kind of detail doesn't lend itself to command-line arguments, which is why zc.buildout is command-line based. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Want to use PackageIndex (and other APIs)
I'd like to start using the PackageIndex class. For now, I want to use: - The constructor (of course) - prescan and - __getitem__. I imagine I'll use other methods in the long term. I'd be happy to document these as I go. Is there any problem with considering these public? As I mentioned the other day, my plan to to make more use of setuptools and less use of easy_install in zc.buildout, until I eventually don't use easy_install at all. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] pkg_resources.declare_namespace vs pkgutil.extend_path
We currently have two ways of implementing namespace packages, pkg_resources.declare_namespace vs pkgutil.extend_path Do we need 2? What does pkg_resources.declare_namespace do that pkgutil.extend_path does, other than provide zip file support? Can we consolidate them somehow? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] zip_safe flag not set; analyzing archive contents... should be info, not a warning
IMO, the message zip_safe flag not set; analyzing archive contents... should be an info message, not a warning message, at lease when processing non-setuptools-based source distribution. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Problems with cli.exe
I've found a bug with cli.exe and a missing feature. The bug is that arguments containing embedded spaces aren't handled correctly. For example, suppose you have foo-script.py and a copy of cli.exe named foo.exe. If you call foo.exe with a single argument, like this: foo a b then foo-script.py will be called with 2 arguments, a, and b, rather than a single argument, a b. The missing feature is that options on the #! line aren't honored. It would be nice if, for example, a script beginning like: #!\Python24\python.exe -O would run Python in optimized mode. I'd be happy to take a crack at fixing the bug and adding the feature. Where is the source? I assume that this also applies to gui.exe. BTW, Shouldn't the setuptools project be moved out of sandbox and be made a proper svn project with it's own trunk, branches, and tags? Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Problems with cli.exe
Phillip, I'v updated launcher.c, cli.exe, and gui.exe on the jim-fix- setuptools-cli branch. I've also added a doctest for cli.exe. The update: - Causes script arguments to be quoted with correct handling of embedded double quotes. This allows passing arguments with embedded spaces and double quotes to scripts. - Added support for passing a single argument (possibly specifying multiple options) to the Python executable on the #! line. - Fixed a bug in handling trailing spaces on the #! line. Let me know if you'd like me to merge this to the 0.6 branch and trunk. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Problems with cli.exe
On Aug 22, 2006, at 3:21 PM, Phillip J. Eby wrote: Let me know if you'd like me to merge this to the 0.6 branch and trunk. Yes, please. I'll take care of updating the release notes for 0.6c2. Done. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Improving setuptools bootstrapping
On Sep 1, 2006, at 7:35 PM, Phillip J. Eby wrote: ... Right now, ez_setup.py is used for two purposes. First, you can include it in a source package in order to be able to use setuptools. Second, it's also used as a way of installing setuptools in a standalone setting. ... So, perhaps we should move to doing something like this: when you run a setup.py that's using ez_setup(), and it appears necessary to download/install setuptools, a message is displayed, something like: This would reduce ez_setup.py to hardly any code at all. Pros: * Net-less installs are less painful * Users don't get surprised by setup.py install trying to access the net * No version skew or missing version issues * Users can install system packages (RPM, .exe, etc.) of setuptools Cons: * Manual step for new setuptools adopters (but who's left?) * Might break setuptools-wrapping programs (like zc.buildout and Enstaller) if they rely on ez_setup * Developers might not want to switch to this new approach, if they feel the manual step is unwieldy Does anybody have any thoughts on this, one way or the other? +1 However, this only addresses the first use case above. I still like ez_setup for the second use case. In particular, the zc.buildout bootstrap script lets ez_setup do most of it's work. It downloads a fresh copy of ez_setup and uses it to download setuptools and zc.buildout. Part of what makes this work is the fact that ez_setup has a well-known URL, so my bootstrap script, http://svn.zope.org/zc.buildout/trunk/bootstrap/bootstrap.py? view=auto can be pretty stupid and minimal. Of course, if ez_setup went away, I'd cope. :) Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [setuptools] best way to create offline install bundles?
On Sep 8, 2006, at 9:14 AM, Kevin Dangoor wrote: Is this something zc.buildout can help with? Probably. zc.buildout isn't aimed squarely at this use case. It might work, either as is, or expanded, or some of it's parts might help. Also keep in mind that it is fairly young and rapidly evolving. zc.buildout can certainly assemble a complete installation and it's offline mode should support installation of a shipped buildout that contains all of the eggs needed. In the near future, I'll be adding a missing piece that allows you to nail down all versions used: http://svn.zope.org/zc.buildout/trunk/specifications/repeatable.txt? view=auto Jim I'm not certain what the best way is to automate this process... Thanks, Kevin -- Kevin Dangoor TurboGears / Zesty News email: [EMAIL PROTECTED] company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [setuptools] best way to create offline install bundles?
On Sep 12, 2006, at 7:37 AM, Jim Fulton wrote: On Sep 8, 2006, at 9:14 AM, Kevin Dangoor wrote: Is this something zc.buildout can help with? Probably. zc.buildout isn't aimed squarely at this use case. It might work, either as is, or expanded, or some of it's parts might help. I'll note that we'll need something like this for future Zope releases, which I expect to be eggs based. I would probably leverage zc.buildout for this, so I suppose that this is likely to be a future zc,buildout use case. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] HOwto download only egg's with setuptools/EasyInstall
On Sep 13, 2006, at 7:20 AM, Kevin Dangoor wrote: ... I just want to gather up all of the original .egg files and put them into a tarball. Everything you'd need would be in there, and there could even be a trivial wrapper script that runs ez_setup.py with all of the right options to install those packages. This can give the user all of the easy_install options for installing, rather than being stuck manually dropping the untarred directory into the right place or adding it to their PYTHONPATH.. Note that you probably want more than this. In particular, you probably want to assemble a bunch of source distributions so that your tar ball is useful on a number of distributions, at least for a source tarball. FWIW, I plan, in the next few days to publish a proposal for providing this functionality in zc.buildout. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] should setuptools be an implicit dependency of every egg?
On Sep 19, 2006, at 5:43 AM, Martijn Faassen wrote: Hi there, Normally setuptools is installed system-wide. When you install an egg using easy_install, the egg itself can count on setuptools being available, and use for instance its resource management API and namespace package marking facility. It therefore is unnecessary to users to mark setuptools as a dependency to their eggs in setup.py. Since it's unnecessary, it'll just not be done. This works fine in the case of system-wide installation of setuptools, but when installing eggs locally, such as in Buildout, this can lead to problems. Buildout makes sure that only eggs that are dependencies are importable when starting up a script. An egg that is depending on setuptools implicitly will therefore not have access to setuptools facilities. Note that the scripts generated by zc.buildout don't use pkg_resources to determine the eggs to be loaded at run time. The needed eggs are computed at install time and their locations are written into the script. If a script doesn't need setuptools (as determined from egg dependencies), then setuptools is not included in the script path. Unfortunately, many eggs that do import pkg_resurces or setuptools don't bother to list setuptools as a dependency. Telling everybody who makes eggs that use setuptools to please take Buildout into account and include setuptools as an explicit dependency in setup.py seems like a hopeless exercise, as things will just plain work for most egg users and developers, at least until the possible eventuality of Buildout reaching critical mass in the community. I would put this differently. I would ask people to be accurate about their dependencies rather than implicitly assuming that setuptools is around. What would work for Buildout, and as far as I can see does no harm to everybody else, is to make setuptools an implicit dependency of all eggs. It won't affect anyone else (but for a hopefully tiny performance impact) as setuptools is definitely available on the system when installing the egg through easy_install anyway. setuptools is avaialble to buildout, because buildout uses it. The small downsides of making setuptools an implicit dependency of every egg is that: - setuptools will be included in the path of every script even when not needed. This probably has a tiny performance penalty. - setuptools will be downloaded an installed in some cases where it isn't needed. zc.buildout can install scripts for versions of Python other than that used to run zc.buildout. For example, you could run zc.buildout with Python2.4 and asl it to install a Python 2.3 script. We actually do this on occasion when we use buildout to install applications that don't support Python 2.4. These applications generally don't use setuptools, but I might still have to install setuptools just because we consider every egg, implicitly to depending on it. These downsides are, in practice, pretty minor. My main objection is that I don't like encouraging people to be sloppy about dependencies. The counter argument, of course, is that the standard library is an implicit dependency and we are behaving as if setuptools is part of the standard library. So, would this be something that could be added to setuptools? Nothing would have to change in setuptools. If we agree that setuptools is an implicit requirement of every egg, even if the egg never imports anything from the setuptools egg, then I'll modify zc.buildout to include setuptools in the path of every script. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] should setuptools be an implicit dependency of every egg?
On Sep 19, 2006, at 8:11 AM, Jim Fulton wrote: ... Note that the scripts generated by zc.buildout don't use pkg_resources to determine the eggs to be loaded at run time. It's also worth noting that the eggs documentation, http:// peak.telecommunity.com/DevCenter/PythonEggs, explicitly advertises that setuptools is not needed to use eggs: If you have a pure-Python .egg file that doesn't use any in- package data files, and you don't mind manually placing it on sys.path or PYTHONPATH, you can use the egg without installing setuptools. I'm going to start with the following compromise in zc.buildout. If an egg uses namespace packages, then I'll treat setuptools as a dependency of the egg even if the egg doesn't list it as a dependency. This will address the common case of eggs using setuptools soley for namespace-package support. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] sftp support
Last week I released a plugin for zc.buildout that adds sftp support: http://www.python.org/pypi/zc.buildoutsftp This is useful when you need to use a private distribution server. I also released a version of buildout that takes advantage of it. This plugin uses the very nice ssh library, paramiko. This plugin should be useful for in other setuptools applications such as easy_install. There isn't really anything buildout specific about it. In fact, it's really more of a setuptools plugin than a buildout plugin. It's implemented as a urllib2 plugin that returns data in a form tailored specifically to setuptools. To use it, just import and call zc.buildoutsftp.buildoutsftp.install(). This installs the sftp handler into urllib2. Then, in your setuptools-based application, you can use sftp urls to name an index server or find links. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Draft of new setuptools installation instructions
Jim Fulton wrote: ... This will probably break buildout's bootstrapping script, which uses ez_setup. I was too hasty in making this remark. The buildout bootstrap script, http://svn.zope.org/zc.buildout/trunk/bootstrap/bootstrap.py?view=markup Uses the use_setup function from ez_setup.py and, as this will still be supported, there is reason to hope that this won't be broken. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] YAGNI extras and tests_require
I'd like to call Ya aint gonna need it on the extras feature of setuptools. As far as I can tell, extras are just a way to avoid fine-grained packages. Is this benefit worth the complexity? I don't think so. It violates There's Only One Way To Do It and increases the complexity of setuptools. Setuptools is wonderful but it is complex. I think it would be helpful to make it simpler and I really don't see a need for extras. I think a similar argument could be made against the tests_require feature. (In the presence of the extras feature, it's puzzling that this isn't handled as an extra.) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] YAGNI extras and tests_require
Matt Good wrote: Oops, I forgot to copy the list on my first respose. On Sun, 2006-10-22 at 14:45 -0400, Jim Fulton wrote: Matt Good wrote: On Sun, 2006-10-22 at 13:52 -0400, Jim Fulton wrote: I'd like to call Ya aint gonna need it on the extras feature of setuptools. As far as I can tell, extras are just a way to avoid fine-grained packages. Is this benefit worth the complexity? I don't think so. It violates There's Only One Way To Do It and increases the complexity of setuptools. Setuptools is wonderful but it is complex. I think it would be helpful to make it simpler and I really don't see a need for extras. Well, I've found them to be useful with entry points in Trac plugins. For example the SpamFilter plugin provides several spam filtering methods, not all of which will be needed by every user. There's an IP blacklist filter that requires DNSPython. By declaring the entry point for the blacklist module with an extra for DNSPython the module will only load if that requirement is available. Why not just distribute the IP blacklist filter as a separate package? I don't think it's uncommon for packages to have optional features that require additional dependencies, and extras provide an easy way to declare those dependencies. Sure, sometimes these features could be broken out into a separate package, but in the cases where I've used extras making additional packages would be the more complex solution. How more complex? You would simply have had more packages? Is that really more complex that extras? Yes. An extra is simply a line or two in setup.py. A separate package means another branch in SVN, Not necessarily. You could manage multiple related packages as a single SVN project. I do. more releases to make, upload to the Cheeseshop, True. more files for users to download, If the downloads are automated, does this make a difference? and unneeded duplication between the original and new package. Why duplication? Likely, one of the packages would simply depend on the other. This seems like an awful lot of additional work just to justify removing a feature from setuptools. I think the extra work is just making the extra releases, at least assuming automated downloads. Is it advantagous enough to violate TWOOWTDI? How does it violate that? There's *always* more than one way, but there should be one clear right way, and I think extras *is* that way. There don't have to be multiple ways. I also don't see extras as being clearly superior, but we disagree. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SetupTools: Running EasyInstall From Within A Script?
Alexander Michael wrote: I am attempting to put together a simple package installation replicator to make it easier to pull updates from a local repository (as an alternative setup to a shared NFS directory). The idea is to put a requirements list and the necessary eggs in a local shared directory and then have a script on each computer do a quick check to see if any updates are needed and install them if there is an update. Is it possible to ask setuptools to install a package from within a Python script without resorting to os.system or friends? I tried browsing the source, but it was difficult to follow the call chain from the easy_install script to the actual method being called. Technically yes, but not practically. You might look at the easy_install module in zc.buildout, which tries to provide a programmatic interface, although even it resorts to os.spawn at times. See: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/easy_install.txt?view=markup Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Organizing for buildout questions
David Pratt wrote: Hi. I am trying to get my head around using buildout with both private and public repositories. So my questions relate to a way to organize my svn in a better way for eggification and construction of a simple sandbox to keep my svn checkouts together in a consistent and portable way. The goal is to checkout sources, modifying and checking in my changes as I go, and having the buildouts in their own folders reflect the changing code and dependencies. My first couple of questions are whether buildout can use a folder as source for a (development) egg - something that will update, Yes. (I wonder if I understand your question.) secondly whether an egg can be created with ssh to a private repository like: svn+ssh://[EMAIL PROTECTED]/usr/home/myaccount/svn/path /to/my/project I have been using ssh for private code sources. Yes. You check out the projects you want to work on and then configure buildout to create develop eggs from them. The rule of thumb is, use develop eggs for checkouts. It doesn't matter if the checkouts are public or private. Note that you can also have private egg repositories using sftp. See: http://www.python.org/pypi/zc.buildoutsftp In a recent buildout I have been studying, I have seen this: find-links = http://download.zope.org/distribution/ I am also looking at a simple sandbox structure that could also make some sense ... development --+-- private-src (private checked out sources) | +-- public-src (external checked out sources) | +-- buildout (a container for my active buildouts) My source containers would just hold folders of checked out code I am currently working on. I plan on checking out my buildouts from svn to a buildout container so that I could have a few or more different buildouts going at a time (each being an app or just testing some things in their own environment) I am curious about folders as sources since sometimes I just want to try some code without a lot of hassle or committing it to a repository right away. I think it would be good to try it in a buildout. With the structure of my development folder I could use a relative link from a source folder as well. Note that I strongly prefer to keep buildouts self contained. Typically, for each package I work on, I make the svn project for that package a buildout. The buildout.cfg therefore usually has: develop = . I use eggs for everything else needed to work on the package. If, for some reason, I want to work on another package at the same time, I'll check that out into the working directory for the first project(or add an external) and add it to the list of develop eggs. On the repository side, I have been looking at zope's repository more critically which is structured like this: container--branch--src--namespace--package Right, where container == project or when more nesting needed: container--branch--src--namespace--second namespace--package setup.py is always at the branch with __init__ for eggs within each namespace package. It seems a good model for eggification since many packages are setup for eggs in a way that is fairly transparent. This of course is a public repository with packages being registered in PyPI. The main downside of course, is all the nesting, which can get tedious. I wish I could think of a way to have namespace packages without creating all of the annoying subfolders. I may start omitting the src directory in some projects. I also think that it was a mistake to create the zc.recipe namespace packages. Rather than zc.recipe.foo, I should probably have done zc.foorecipe. I may switch to this in the future to reduce the nesting. For a private repository, I would like to checkout the code as eggs equally as well. Why not follow the same pattern as for public projects? I don't see why a project's structure should depend on whether it is public or private. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout and zc.recipe.egg:custom problems
Jim Fulton wrote: Martijn Faassen wrote: Hi there, I'm trying to induce zc.buildout's zc.recipe.egg:custom recipe to build a library (lxml) against C libraries (libxml2, libxslt) installed in a custom place, i.e. as parts, such as parts/libxml2/include and parts/libxml2/lib. I thought I had that working before and even blogged about it, but it turns out that something's not entirely right yet. Right now I'm struggling to actually see an 'include-dirs' directive to have any effect on what's passed along to the C-compiler (namely the -I) option. It doesn't. After some research I think I've figured out why. I've installed lxml as a development egg, and somehow it always gets built the default way. The custom egg's install code is reached, but only when it's too late and lxml is already built. Therefore any special options such as include-dirs and library-dirs are never passed, as lxml has already been created. I figure lxml gets created automatically and not through the custom recipe as I have it listed as a develop egg in the [buildout] section. What to do about this? Unfortunately, you've hit a limitation in buildout and the custom recipe. This limitation will, eventually, be removed. When creating develop eggs, you have no control over build options. When creating custom eggs, you have no way to use directories as source, and besides, you really want a develop egg. I think what we want is a develop recipe that lets you specify various options when creating develop eggs. I'll look at writing this over the next few days. I still haven't finished this, but while working on it I realized that I should have suggested a work around. You can create a setup.cfg file in the directory containing your library setup.py file. This ConfigParser-style configuration file should have a build_ext section containing the options you want: [build_ext] include-dirs = /foo/include This is actually what the develop recipe will do behind the scenes. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] develop egg recipe?
On Dec 2, 2006, at 8:50 AM, Jim Fulton wrote: ... I plan to write a develop recipe, and am hoping to get to that this weekend. I've just released a new version of zc.recipe.egg that provides a develop recipe: http://www.python.org/ pypi?:action=displayname=zc.recipe.eggversion=1.0.0b3#controlling- develop-egg-generation Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout minitutorial at PyCon
Jim Fulton wrote: The other mini tutorial is on the Zope Component Architecture. It is aimed a Python programmers in general. You needed be or intend to be a Zope developer to benefit from it. I tried to type needn't. Sorry. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] HTTP authentication support
On Dec 27, 2006, at 11:25 PM, David Smith wrote: I needed support in easy_install for installing from HTTP digest and basic authenticated sites so I wrote the included patch. It currently supports specifying username and password information at the command line (however misguided that may be) and prompting the user interactively. It should also be easy to integrate with graphical interfaces. Please consider it for inclusion in the next release. I don't know if it is an option for you to use zc.buildout. If it is, then zc.buildout has an extension that lets you use sftp: http://www.python.org/pypi/zc.buildoutsftp This has the advantage that, if you use an SSH agent, you don't have to enter credentials when you download a transaction. And, of course, it is more secure than digest or basic authentication. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] tests_require and test_suite in egg-info
It appears that the values of the tests_require and test_suite keyword arguments isn't captured in egg-info. Am I missing something? Why would this information be omitted? I would like to use it when generating test runners in zc.buildout. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] tests_require and test_suite in egg-info
On Jan 4, 2007, at 4:37 PM, Phillip J. Eby wrote: At 04:14 PM 1/4/2007 -0500, Jim Fulton wrote: It appears that the values of the tests_require and test_suite keyword arguments isn't captured in egg-info. Am I missing something? Why would this information be omitted? Because the only existing tool that uses that information (the test command) can't be run without the setup.py, which isn't installed with the egg. uh huh. In any case, many projects treat tests as development tools only, rather than installing them with the project's main code. Many projects like to run tests even of packages they use, especially if there is a problem. I would like to use it when generating test runners in zc.buildout. Well, you could add an egg_info plugin writer to do it, or you could add a custom setuptools command like create_test_runners and run setup.py on it. There are probably also other ways I haven't thought of. :) I can't do any of this if all I have is an egg. I want to be able to be able to run the tests that come with eggs. I suggest that it would be better for setuptools to record this information by default. Then again, I think that these are duplicative of other features. For example, tests_require could just be handled by an entry in extras_require for a tests extra and test suites could be handled as entry points. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] RFC: Standard Declaration of tests in eggs
Here is a rough draft proposal for declaring tests in eggs: Introduction Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests. Proposal This proposal aims to be extremely simple. It has 2 parts: 1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites. 2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner. Thoughts? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
David Fraser wrote: Jim Fulton wrote: Here is a rough draft proposal for declaring tests in eggs: Introduction Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests. Proposal This proposal aims to be extremely simple. It has 2 parts: 1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites. How would this work if for example, you're using an alternative testing framework (like py.test) for your test? I'm not familiar with py.test. I guess it's not based on unittest. Why? Couldn't it at least have a unittest wrapper, like the one I wrote for doctest? Even though I use doctest almost exclusively, I view unittest as a common API that various testing frameworks can and should play with. I certainly think there should be some common API like that and see unittest as the incumbant. It would be nice to be able to bootstrap it :-) What do you mean by that? 2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner. Great, so at least the testing framework could be declared as a dependency My assumption (and my use case) is that you'd run whatever test runner you want and tell it to run tests from some given eggs. So the test runner would already be loaded. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
Phillip J. Eby wrote: At 01:11 PM 1/5/2007 -0500, Jim Fulton wrote: Phillip J. Eby wrote: ... Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me. Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project. It's probably a couple of lines. Import a loader and call it. That doesn't seem like a big deal to me. Nice and explicit too. And it's enormously repetitive if you have a lot of projects. Besides, I expect most projects will have a single test suite (function) that could just be named directly. You expect wrongly. :) I can only speak based on my own experience. Projects like Zope and ZODB certainly want some sort of tool to find and assemble tests. Eggs are allowing and encouraging us to move away from such large projects. As we break up Zope into individual packages, I'm finding that the individual packages have single test suites. 'nose' and 'py.test' are popular *precisely because* they do NOT require this. Setuptools changed to emulate them by adding ScanningLoader, even though it originally did the test suite function thing you're proposing. See, I originally wrote all my test suites the way you're proposing, but I changed because this is one of those convention beats configuration situations. It's easier to just conform to a policy and have a tool that applies the policy. Different projects may have different policies, so providing an option for the loader, allows them to follow it. (And it made my life easier, too, as I began doing more small projects instead of a few large ones.) And, if you want your approach to be widely adopted, it would be best to allow projects to follow their own policies without requiring them to add any code, even if it's a one-liner. They'll be adding code one way or another. Either they add it to their test modules, or they add it to setup.py. The advantage of the former is that it provides more flexibility without requiring any more work. It is also more explicit. So, *technically*, there may be no reason to do this, but from a usability, friendliness, compatiblity, marketing, adoption, etc. format I don't think the standard will be successful without allowing this out. People often have heavy psychological investment in existing tools, and it is hundreds of times easier to rationalize adding a line of configuration to setup() than it is to rationalize adding two or three lines of code. One is merely a packaging change, the other is a programming change. We disagree. I don't have any problem with having tools that can be used to find and assemble tests into a test suite. I find the unittest loader framework to be baroque. I'd rather let people use whatever framework they want to do this and to control this from Python. I withdraw the proposal. I wouldn't object to the proposal you want, if you want to write it. I don't have the time or interest to write it myself. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC: Standard Declaration of tests in eggs
Phillip J. Eby wrote: At 10:14 AM 1/22/2007 -0500, Jim Fulton wrote: I'd rather let people use whatever framework they want to do this and to control this from Python. The approach I suggested certainly allows that to happen, I know. That's why I wouldn't object to it. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building an egg from Subversion using zc.buildout
Ian Bicking wrote: Martin Aspeli wrote: http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall I found myself searching for this very thing, and couldn't find a way to do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL in 'find-links' didn't work either. I'm not sure I understand zc.recipe.egg and setuptools well enough to be able to guess what may be wrong. Since zc.buildout uses setuptools though, if there is a standard setuptools way of doing this, I'd hope it should be possible to do it in zc.buildout? Is there a way to do find-links in buildout? I think so, maybe named something like find-links. Yup. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building an egg from Subversion using zc.buildout
Martin Aspeli wrote: Ian Bicking wrote: Is there a way to do find-links in buildout? I think so, maybe named something like find-links. Yep. Then you'd point at a page that has the link with #egg= IIRC you have to point find-links at a real HTML page. Right. But I don't have such a page. Basically, I want my buildout to track e.g.: http://svn.plone.org/svn/plone/plone.portlets/trunk This having a setup.py and I hope other necessary info to check it out and build it. What would I need in terms of find-links for this to work? I think you need to create such a page. I sometimes put on in svn (maybe actually in the buildout), and point it there. Annoying part is you have to commit it to test. buildout, kind of like install_requires, I think needs you to give it something it can actually fulfill. So you'd need Foo==dev,1.1 or somesuch. You mean in buildout's list of eggs-to-install? Basically, my buildout.cfg now has: find-links = http://download.zope.org/distribution/ http://effbot.org/downloads eggs = elementtree python-yadis python-openid python-urljr I believe I can do foo==dev or whatever in that list of eggs as well. But I'm not sure how that relates to svn link above. There seems to need to be a page you can point to with an #dev anchor that specified the project name. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building an egg from Subversion using zc.buildout
Phillip J. Eby wrote: At 04:25 PM 1/28/2007 -0600, Ian Bicking wrote: Is there a way to do find-links in buildout? I think so, maybe named something like find-links. Then you'd point at a page that has the link with #egg= IIRC you have to point find-links at a real HTML page. Not for easy_install, you don't. If buildout doesn't support it, that's a bug or limitation of buildout. buildout does support find-links (and index, for that matter). Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building an egg from Subversion using zc.buildout
Phillip J. Eby wrote: At 10:01 PM 1/28/2007 +, Martin Aspeli wrote: I found myself searching for this very thing, and couldn't find a way to do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL in 'find-links' didn't work either. Hm. easy_install accepts svn: URLs in --find-links as long as they have the #egg part, so that sounds like a problem with zc.buildout. buildout uses setuptools.package_index.PackageIndex to find and download distributions. Does that support svn urls? Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] zc.buildout likes to download eggs...
Martin Aspeli wrote: ... which is great, but say I have 10 buildouts all using all the eggs from zope and plone (either explicitly or as dependencies). I'd rather not replicate this in every buildout. I think there are two concerns here: - bandwidth: My ISP gets funny when I go over 20Gb/mo - disk space: A lot of information could be duplicated Are there any facilities to share eggs among packages (I have a feeling Jim implied there isn't yet)? Yes If not, what would such a facility look like? See below. For the bandwidth part, perhaps it could be some kind of local repository of eggs (but how to stop zc.buildout being over-zealous and getting eggs again thinking something may be newer?). By default, buildout tries to get the newest distributions that meet your requirements. This is not a bandwidth issue, but given how slow PyPI is, it is a time issue. (Note that if all of your requirements have upper bounds and you have distributions that match the upper bounds, then no network access is performed). Buildout has an offline mode that prevents checks for newer versions, but also prevents getting new distributions to meet new requirements. I've decided to add a fast mode that skips the check for newer versions if existing local distributions satisfy requirements. For the disk space part, I'm guessing we'd need a shared eggs directory. I imagine I could specify eggs-directory as an absolute or relative path, e.g. ../shared-eggs. I don't know whether one buildout would clear out eggs from another one and so on in this scenario, though. You can arrange for multiple buildouts to share a common eggs directory. This is a major win both for space and speed since we can usually satisfy requirements from a shared eggs directory. The easiest way to do this is through the user-default configuration file. Buildout normally reads default configuration options from .buildout/default.cfg in your home directory (aka ~/.buildout/default.cfg). Typically people specify the buildout eggs-directory option in this file. For example, my ~/.buildout/default.cfg file looks like this: [buildout] eggs-directory = /home/jim/.buildout/eggs I had to manually create /home/jim/.buildout/eggs when I set this up. All of my buildouts look for eggs in and download eggs to this shared directory. (Note that this directory isn't used for develop eggs or eggs that are build with custom build options. It is only used for eggs that are known to be constant for a given version and thus sharable accross projects.) Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building an egg from Subversion using zc.buildout
On Jan 29, 2007, at 5:14 PM, Martin Aspeli wrote: Jim Fulton wrote: So, from your example, you now have a kss.core-0.1dev egg. It isn't a real release. It reflects the state of your subversion repository at the time you happened to run the buildout. My guess is that if you run the buildout again, the egg won't be recreated. You won't be tracking subversion changes as you wished. Further, if you use a shared eggs directory, any other buildouts will likely use that same egg, which has a more or less random snapshot of the software. I suggest that this is a bad thing. IMO, this is what you really want: - You want something to check out your project and make it a develop egg. - When you run buildout later, you want something to do an svn up in the checkout and to rerun setup (in case there are extensions). I think this something should probably be a recipe (or a feature of some existing recipe). I wonder what easy_install does in a case like this. For example, I wonder if easy_install would download and rebuild the egg every time you ran it. Of course, with easy_install, you probably would only run it once, but, again, then you wouldn't be tracking subversion. I'd need to test this, but I *think* that it considers the svn revision as part of the version number so that when the svn revision changes, I wonder if the egg it generated has the version # in it. if buildout goes online to check for a new revision, it'd re-download and re-build. That sounds like what I want, anyway. :) Well, it would be an improvement, but it still might not be what you want. After all, it will check out the entire project everytime, when it would be quicker to just do an svn up. In any case, you should experiment with both easy_install and buildout. I'm fairly sure that buildout won't pick up new versions -- but I could be wrong. easy_install won't pick up new versions unless you use -U, but I wonder if it will even then. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building an egg from Subversion using zc.buildout
On Jan 29, 2007, at 5:26 PM, Phillip J. Eby wrote: At 05:07 PM 1/29/2007 -0500, Jim Fulton wrote: On Jan 29, 2007, at 4:52 PM, Martin Aspeli wrote: Jim Fulton wrote: So do you get a develop egg and egg link or a regular egg for each of these packages? If the later, as I fear, I think you have a problem. I think the latter (I *just* deleted my test instance and I don't want to do it all again now), but why is it a problem? So, from your example, you now have a kss.core-0.1dev egg. Nope - it'll be 0.1dev-rFOO where FOO is the repository revision in question. Ah, good. I hope that's what he got from buildout. :) I wonder what easy_install does in a case like this. For example, I wonder if easy_install would download and rebuild the egg every time you ran it. Only if it thought the tagged version was a better match for what you were looking for. How would it determine that? Would it try to find the current svn revision #? Of course, with easy_install, you probably would only run it once, but, again, then you wouldn't be tracking subversion. Development snapshots of setuptools can be checked out with setuptools==dev, and that will always download a new version because the installed version is always a version like 0.6c2dev- r51290 or something. This doesn't match ==dev, thereby forcing easy_install to download a fresh copy. I wonder if buildout behaves this way. Buildout will ask a PackageIndex for the best available version using the obtain method. I wonder what a package index will return in a case like this. I guess I expect he'll get a version of ==dev which, as you say, will be considered newer and cause buildout to get a new distribution. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Building an egg from Subversion using zc.buildout
On Jan 29, 2007, at 5:37 PM, Martin Aspeli wrote:
Phillip J. Eby wrote:
Development snapshots of setuptools can be checked out with
setuptools==dev, and that will always download a new version
because the
installed version is always a version like 0.6c2dev-r51290 or
something. This doesn't match ==dev, thereby forcing easy_install to
download a fresh copy.
Could you explain how the find-link would work with an ==dev version
specification?
Again, this may be a buildout issue, but I have:
find-links =
...
http://svn.plone.org/svn/archetypes/archetypes.kss/
trunk#egg=archetypes.kss-0.1dev
...
eggs =
...
archetypes.kss==dev
...
This gives me an error:
Traceback (most recent call last):
File ./bin/buildout, line 12, in ?
zc.buildout.buildout.main()
File
/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/
buildout.py,
line 1056, in main
File
/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/
buildout.py,
line 309, in install
File
/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/
test/bot/eggs/z2c.recipe.zope2instance-0.1dev_r12104-py2.4.egg/z2c/
recipe/zope2instance/__init__.py,
line 36, in install
requirements, ws = self.egg.working_set()
File
/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/
test/bot/eggs/zc.recipe.egg-1.0.0b4-py2.4.egg/zc/recipe/egg/egg.py,
line 79, in working_set
path=[options['develop-eggs-directory']]
File
/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/
test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py,
line 355, in install
File
/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/
test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py,
line 294, in _get_dist
ValueError: (Couldn't find, Requirement.parse
('archetypes.kss==dev'))
I have a feeling this means my find-link is wrong, but I couldn't
find a
permutation that works...
I think you don't need/want the the ==dev in the requirement.
That is:
eggs = archetypes.kss
Jim
--
Jim Fulton mailto:[EMAIL PROTECTED]Python
Powered!
CTO (540) 361-1714
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Optional C extensions in packages
On Feb 1, 2007, at 1:03 PM, Phillip J. Eby wrote: At 07:11 AM 2/1/2007 -0500, Jim Fulton wrote: I like the goal, since most of our extensions are in this category, however, I have a feeling that this use case might be better served by packaging the extensions as separate distributions. True, but without automation, it's more work for the package author to do the right thing. Yes. It also makes it easier for the consumer to be explicit about what they want. Although maybe there are better ways. If your extension is in a package, you have to make it a namespace package. You have to create another setup.py and directory structure, and manage another set of releases. This is a lot more work than just marking the extension as optional. Yes. See below... Once concern I have is that I think I'd want an egg with the extensions and the egg without the extensions to have different names, and perhaps different requirements. Of course, the names would be different because an egg with a C extension would be platform dependent and an egg without would be platform independent. What if both a platform-independent egg and a platform-dependent egg were available? Which would take precedence? I think they have the same precedence now. I think I'd be +1 if platform-dependent eggs were prefered over platform-independent ones. ...of the same version? Yes That's easy enough to do by changing the Distribution.hashcmp property to put the 'platform' part earlier in the comparison. If they're different versions, however, it's a whole 'nother kettle of fish. Yup. Of course, that points up that the ambiguity remains, on some level. So I retract my precedence idea, The solution I was thinking of, however, has the additional benefit of working right now. If you're generating eggs for a package with optional extensions, you would simply *not* generate a cross- platform egg, but instead just ship an sdist tarball, plus platform- specific eggs for the platforms you care about. That way, anybody on an unsupported platform will end up building a local egg from the source, and either producing the extensions or skipping them locally. (And if you don't have access to a compiler, it seems unlikely that you will be then *distributing* the eggs you produced.) Notice that this setup is a natural side-effect of the way most people would produce and upload packages to the Cheeseshop anyway - you would have to make an intentional effort to suppress extension- building and upload a cross-platform egg; otherwise you will just end up with a source distribution and a platform-specific egg, which then works as intended. I'm still worried about the ambiguous case when there are both platform-dependent and platform-independent eggs installed. I think you were proposing an easy_install option. This helps when someone installs a distribution directly, but doesn't help when a distribution is installed as a dependency. It also doesn't help with controlling selection of eggs after installation. And I think it doesn't make it easy to change one's mind. For example, one might install an egg with extensions and then install one without extensions to debug a problem using the Python debugger. Would the option let them do that? Is it possible to control this as part of the requirement specification? Perhaps this could be some kind of standard extra? I'd strongly prefer to be able to control this via the requirements mechanism. I'd like to be able to say that I want or don't want extensions as part of a requirement string. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Optional C extensions in packages
Phillip J. Eby wrote: At 05:32 PM 2/1/2007 -0500, Jim Fulton wrote: I'm still worried about the ambiguous case when there are both platform-dependent and platform-independent eggs installed. How would this happen? At least in a couple of ways. 1. As I mentioned in my previous note, when a package has optional extensions, one will often want to disable the extensions for debugging purposes. It is easier debugging Python code than C code, especially in combination with other Python code. In the past, this was typically done by removing .so (or .pyd) files. This can still be done with eggs, but I thik it will be attractive to do this by selecting diffeent eggs. 2. Consider the following scenario: Someone has a mac without a development environment installed. They install some eggs and get versions without extensions. Later, they install the development tools that came on the CD with their mac. How do they reinstall the eggs with extensions? If they install in multi-version mode, won't they have a mix of eggs with and without extensions? I think you're trying to solve a broader problem than the one I'm trying to solve, which is that I'd like to make it possible for people who don't have working compilers (i.e. mostly Windows, with some Mac users and some people in virtual hosting environments) to install packages that contain C extensions. I'm trying to avoid a problem I think you may create. As soon as there can be two eggs that satisfy the same requirements but with different semantics, I think there is a problem. I understand that in the use case you are thinking of, this would normally not happen, but it still can happen and I suggest will happen. In that scenario, you're going to *always* want to use this option to suppress optional extensions, because there isn't any way for you to build them. But, you would presumably still want to know about packages that *require* their extensions to be built. I think you were proposing an easy_install option. This helps when someone installs a distribution directly, but doesn't help when a distribution is installed as a dependency. This would be an option to suppress compiling *all* optional C extensions, period. So it would apply to dependencies as well. Yeah, that makes sense. It also doesn't help with controlling selection of eggs after installation. And I think it doesn't make it easy to change one's mind. For example, one might install an egg with extensions and then install one without extensions to debug a problem using the Python debugger. Would the option let them do that? The idea was that it would be a build-time option. Will it be possible to reinstall eggs with the same versions but with different choices wrt optional extensions? I guess even if it isn't supported by easy_install, I could make it work with buildout. Is it possible to control this as part of the requirement specification? Perhaps this could be some kind of standard extra? I'd strongly prefer to be able to control this via the requirements mechanism. I'd like to be able to say that I want or don't want extensions as part of a requirement string. Yeah, I see the benefit of that, certainly. The problem is that we're trying to solve different problems. I just want to make it *possible* to suppress building extensions during easy_install. I want that too, certainly. I'll give some more thought to what you're asking for. Super. I have an inkling of an idea, but the problems have to do with things like having to actually check the egg's contents to see if it meets requirements, and there are problems regarding the need to clean up the build/ directory if you change what features you build something with. You see, setuptools has an undocumented 'feature' mechanism (which is still used by some PEAK projects) to control the inclusion of various packages, extensions, etc. The main reason this is undocumented is because it turns out that it's fragile to specify what features to use or not use on the command line alone, due to some distutils' commands just taking whatever's in the build/ directory as gospel. Anyway, that feature mechanism could probably be tied in to the requirements system, as long as there was a way to wipe the build/ directory whenever the features changed between runs of setup.py, and there was a way to list the features in the .egg-info, and pkg_resources was changed to query a distribution's features info when validating a requirement that includes extras. I'm a little concerned that this will incur additional disk access under various circumstances, unless there is some way to statically distinguish between extras that denote features and ones that indicate additional requirements. Of course, matching a requirement against a distribution when the requirement doesn't list any extras, will not incur
