Re: [Distutils] PyPI template improvements
Two from me: * Make the textareas not do a hard wrap in the package info edit screen. Right now you can't edit (or even save without editing) the metadata for packages with long lines without breaking them, which includes breaking long URLs. * Kind of relatedly, make long descriptions wrap in some fashion. Right now if you have a plain-text long_description for your project without newlines it just gets really wide. I think that'd be fixable with just CSS. On Wed, Jun 16, 2010 at 4:14 PM, Simon de Vlieger si...@ikanobori.jpwrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hey all, the recent activity on this mailinglist has kickstarted my contributing sense. As long as the mirroring debate is still ongoing I will focus my efforts somewhere else. Namely: the HTML/Javascript/CSS. In this regard I have a few questions before I really dig into these templates: - - Is there a list of improvements, maybe a nice TODO of points which people want to see improved? - - How are design changes handled, is there a committee to run them through? People who decide on what gets in and what not? (I'll outline some of my first thoughts lower in this mail) - - What are the supported browser versions by PyPI, I reckon it's IE6/7/8+, Fx 2+, Opera 9+ Safari 4+? The changes I have on my personal 'todo list' are: - - Add labels to all forms. - - Make tables consistent width (see for example the table in the top of the Browse packages page and compare with the table when you actually select one of the classifiers). - - Restyle the metadata display on package pages and move it up in the page. - - Have downloads readily available on the right side of the screen (at least the latest release). - - Look sternly at the top right floating account information page. - - Look at the your details page where the form does not align with the right floating profile box. - - Make one consistent styling for all forms. Include help texts in all forms. There are more things I want to do, but this is the start. I have already cloned Tarek's PyPI clone on Bitbucket and I'll add my changes there. Is there anything you guys (and the users) would really like to see improved? Regards, Simon de Vlieger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQIcBAEBAgAGBQJMGT6kAAoJEBBSHP7i+JXfHmsQALbm5rqRpLdjTDWNHLQHqoVE D4rDht/oaOFwPBfWp2FMRIhqLa9yA2AQfEAWSBFLqNMr2FGwpRPAXGgd38VOKOf0 pTORXM7cw92yjQXFz4J4xvcN7IcmyYEfftbwlITStnAEc5Q/5sl93nxyyigyQr7D y62EfCRGQP/OTfU7PYoj1KS9Qwi4ep6saD1cdL+tM7AgPiGwJMU2f29vElMVvT1G of+3gZjEi/UpsiYP/qD3JW3iFoOKv3KJbMmdHfMaPCa/vyfGRBoM4eGf5XvV+3oy U6EJ3YvvvTTS/w+TR36USxonuspYWKxoku1hUDaRsDgUC1fdW1UWVtocvnGzr1JW j06KN19ypzVK6UFoRTTsbA6K3h/pppgO1MIH9iAQgkvNY/irZiACO//x6bLT7ZFr 3PRt2ZaJzutKpUT0dGf0HjqcXdF6tPGnaQK1kkBGIgJrtXgqHqmc6Ee4S3Kd/boD tS9ZVqtIB2npJR4e0ZA6iqqGCNvpJlQEiUXSIpmTBRIgvpIjGcA/DQ+rQSIY8twS KPMGdnLm9u0HJdmlsRvJvH6/FBYIhoWDYWulil8ZpZnjHsyLAfTQjNeWzQDylEKF O750a9QgRIbeEFIOZK1kkaccy0dL3oX4MqJBfLlaJUlBJ0pYq3jTXf9+kc6g+uGc hutNfaPX84pe/oxL/z8k =hMyY -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI template improvements
On Thu, Jun 17, 2010 at 1:15 PM, Barry Warsaw ba...@python.org wrote: On Jun 17, 2010, at 12:56 PM, Ian Bicking wrote: * Kind of relatedly, make long descriptions wrap in some fashion. Right now if you have a plain-text long_description for your project without newlines it just gets really wide. I think that'd be fixable with just CSS. How about supporting reST in descriptions? It is supported, but if something doesn't render cleanly as reST it is treated as plain text and wrapped in an unwrapping pre. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip vs easy_install vs distutils2
On Thu, Jun 3, 2010 at 8:38 AM, Barry Warsaw ba...@python.org wrote: So at the end, the end user would chose an installer that is compatible with these archive, and know how to install them. In other words, have ez_setup for example, run once for all at the Python level, and be THE installer. Or run a pip_setup or whatever. How would OS vendors get into the game? I could imagine that Ubuntu would want to make an opinionated choice for our users, and maybe even set things up so that packages are installed from our archives (as .deb packages) by default. This would make things very easy for the majority of users, though of course we'd have to allow experts to customize it to grab from the Cheeseshop or use a different installer. I could see how a deb might make sense for an unqualified installation, i.e., one where no specific version or location is indicated. *Probably* a specific version would be okay, but the way deb archives work it seems like the archive would usually be unable to satisfy the request. It would be interesting if you could also hook into a deb generation script, and install ad hoc packages as debs. This really isn't a system choice, but a where-the-package-is-installed choice. If installing in /usr/lib/* then using the system package makes sense. If installing anywhere else it doesn't make sense (home directory, virtualenv environment, something ad hoc using install options). I wonder if it would work best to control this through some distutils.cfg-like file (distutils.cfg is terrible though), that would be looked up based on the installation location. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] At least one package management tool for 2.7
On Wed, Mar 24, 2010 at 7:27 AM, Olemis Lang ole...@gmail.com wrote: My experience is that only `install_requires` is needed (unless you want to create app bundles AFAICR) , but in practice I've noticed that *some* easy_installable packages are not pip-able (though I had no time to figure out why :-/ ) Usually this is because Setuptools is poking at objects to do its work, while pip tries to work mostly with subprocesses. Though to complicate things a bit, pip makes sure the Setuptools monkeypatches to distutils are applied, so that it's always as though the setup.py says from setuptools import setup. easy_install *also* does this. But then easy_install starts calling methods and whatnot, while pip just does: setup.py install --single-version-externally-managed --no-deps --record some_tmp_file The --no-deps keeps Setuptools from resolving dependencies (because it does so using easy_install), and --single-version-externally-managed keeps Setuptools doing egg directories. And --record keeps track of installed files, which are later moved around to facilitate uninstall. But some distributions pay extra attention to those options, or do other tricky things in their setup.py, and as a result this causes failures. Because easy_install is just calling internal methods, it kind of sidelines those tricks (also people don't tend to give these options to easy_install, and some don't even exist, so some code paths just aren't exercised with typical easy_install usage). Oh, the other reason is the link searching mechanism is slightly different between the two installers. Not particularly intentionally, just incidentally. Also because pip doesn't install anything but source packages, some packages are installable via easy_install but not pip -- usually this is an oversight on the part of the person doing the packaging, but they just never noticed. -- Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [venv] question about future pip capabilities
On Thu, Dec 10, 2009 at 11:18 AM, Darren Dale dsdal...@gmail.com wrote: Questions concerning pip: * Is there a roadmap or timeline concerning installing from windows exe or msi files? I didn't see anything at bitbucket. There isn't a timeline, as we don't have anyone actively contributing with respect to Windows. * Is there a roadmap or timeline concerning extras? I didn't see anything at bitbucket. There isn't really; extras would be nice, but there hasn't been much demand for them. pip requirement files serve a similar purpose though in a somewhat different way. * Could anyone please expand on the comment that pip is incompatible with some packages that customize distutils or setuptools in their setup.py files, and that it maybe doesn't work on windows? Maybe the Windows statement is exaggerated. Unfortunately we don't have a good Windows testing setup, and I don't even know what that would look like. Some buildbot (or buildbot-like-thing, e.g., Hudson) running on a Windows machine would be really helpful. pip itself relies only on the command line and some expected behavior of setup.py files, it doesn't really care how this is implemented. To install a package it does: python -c 'import setuptools; __file__=setup.py; execfile(__file__)' install --single-version-externally-managed --record=/tmp/XXX The first part installs the setuptools monkeypatching for packages that just use distutils. So you at least have to make something compatible with that monkeypatching (i.e., it can't rely on setuptools not being imported). Then it uses those two command-line switches (and I'm pretty sure that's it; you can see a complete record of what pip does with pip install -vv). Those are provided by Setuptools (or Distribute) and not by Distutils, but pip doesn't care how they are implemented so long as you accept those options. Also to be compatible with pip you have to create a proper Package.egg-info/ directory (to record that the package is installed) and use that --record option to write a list of all the files that are written to disk, so they can be removed when the package is removed. Questions for pip and Distribute: * How closely coupled are these projects? They are not closely coupled at all. * I have a bit of time after hours, how can I help? If you are a Windows developer, Windows help for pip would be great. Installing Windows binaries would be the biggest help; I suspect this means unzipping the .exe or .msi and moving the files into place manually (since there is no setup.py), and maybe even generating that .egg-info/ file. I suspect that if you install a Windows installer directly that pip (and Setuptools/Distribute/pkg_resources) might not be able to see that the package is installed. But I've never tried. It would be nice if these installation techniques were compatible with each other, which means the installer creating .egg-info files (or shipping with EGG-INFO, similar to how Windows binary eggs work). In theory Windows binary eggs could be installed similarly, but I don't think there's as much enthusiasm for that among Windows users, and .exe or .msi packages should be mostly equivalent (except that lack of metadata). But I don't consider myself a good judge of Window users' preferences, so maybe I'm wrong. A shorter time commitment is to download pip and run the tests on Windows and report problems, as it's unfortunately quite possible we've introduced Windows regressions. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [venv] question about future pip capabilities
On Thu, Dec 10, 2009 at 1:40 PM, Carl Meyer c...@meyerloewen.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ian Bicking wrote: A shorter time commitment is to download pip and run the tests on Windows and report problems, as it's unfortunately quite possible we've introduced Windows regressions. I did a bit of work on getting pip tests running under Windows last spring, but didn't get very far. There were many test failures, most of them seemingly superficial: posix-specific assumptions about file locations and path separators in the tests themselves, not necessarily broken functionality in pip (though it's hard to say for sure with broken tests). Fixing those issues isn't hard, just tedious. I don't know what deeper issues or regressions might be uncovered once the tests run. Probably some of those issues could also be fixed in ScriptTest, instead of making the pip tests tediously platform abstracted (e.g., normalizing \ vs /). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] 70 packages in total use setuptools' extras feature
On Thu, Dec 3, 2009 at 1:05 PM, P.J. Eby p...@telecommunity.com wrote: At 10:21 AM 12/3/2009 -0800, Sridhar Ratnakumar wrote: Do you have anything particular in mind? What I did was a ``grep ^.*[a-zA-Z]\[`` in the requires.txt of (almost) all the packages in PyPI. Do note that this won't tell you about end users' use of extras. The main use case described in the documentation for extras is allowing users to install optional support for things. That means you're more likely to see them in buildout configurations (which won't be found on PyPI), and manual installation steps (which aren't recorded anywhere). Incidentally, I never got around to implementing extras in pip, and no one has asked about it so far. (Though I guess pip generally encourages people handle transitive dependencies explicitly; i.e., if you want zope.component with zcml, then install zope.component and zope.component.zcml.) -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] 70 packages in total use setuptools' extras feature
On Thu, Dec 3, 2009 at 3:21 PM, P.J. Eby p...@telecommunity.com wrote: At 09:53 PM 12/3/2009 +0100, Attila Oláh wrote: Wouldn't it require zope.component to be a namespace package? Extras allw you to add extra files (subpackages) withot making the container package a namespace-package, it seems to me. No, they don't. Extras just cause a project to pull in additional dependencies, not additional *files*. The basic advantage, I guess, is the indirection; you can require zope.component[zcml], and the maintainers of zope.component determine what is necessary to provide that functionality. And from the changelog it seems they have updated that to depend on different things over time. Though I suspect that was actually a kind of misfeature of extras... some library X requires zope.component[zcml], but in fact only requires a subset of that functionality, and some framework F requires library X, which draws in zope.component[zcml], which in turn requires oodles of other things, and there is pushback from F to zope.component (maybe involving library X maintainers, or not) to reduce the dependencies to what is more strictly required. All of which requires lots of tedious communication and negotiation, which I don't think is really that helpful for anyone, and is why I prefer a more manual assembling of sets of libraries, and am not a fan of Setuptools runtime checking. My general feeling is that these dependencies should only be loosely trusted; only actual testing can confirm that an update is correct for a developer's use case. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345 - 3 new fields
On Tue, Dec 1, 2009 at 9:11 AM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Tue, Dec 1, 2009 at 4:19 AM, Ian Bicking i...@colorstudy.com wrote: [..] If something more like that was added (more like Trunk-URL), that would be genuinely useful. Barring that, I'd recommend removing the Repository-URL field from the recommendation until we have a clearer use case. Right now, the better use case would be a field giving the checkout command, e.g.,: Repository-Checkout: hg clone http://bitbucket.org/ianb/webob But I don't think that is particularly useful, only that it is more useful than Repository-URL. Here's a proposal then, that seems to synthetize what people have been saying: Let's drop Repository-Browse-URL and keep a single Repository-URL field, which is a free URL that can take any URL form. e.g. a browsable url, or a git/hg url etc. I prefer Repository-Browse-URL, as it is more explicit in its use: it's a link that someone using a browser can usefully click through to. I expect it will be displayed as such on PyPI. So this link is good: http://github.com/cloudkick/libcloud And this link is bad: git://github.com/cloudkick/libcloud.git A similar distinction exists for code.google.com projects. Now for Change-Log vs Change-Log-URL, I think the first one is better, because that's what people are already doing in their packages (when they add their changelog at the end of their long_description option), and it's not hard for PyPI to store it and display it, besides Description. This seems fine to me. Is ReST allowed? Could one potentially just do: `Changes http://myproject.com/changes.html`_ ? And then essentially the changelog would be a single link? I'm not sure if that's a good idea. Would it be too vague to say that if the change log is a single URL that PyPI should link directly through to the change log instead of displaying the link? (The exact UI for PyPI hasn't been proposed, but if it's something like a tab with changes, that tab could actually link offsite) -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345 - 3 new fields
On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: As suggested in Catalog-SIG by some people, I would like to propose the addition of three more fields for 1.2: Repository-URL A string containing the URL for the package's repository. Example: http://svn.python.org/projects/python/trunk/ I like this in concept, but it's a bit vague as just a URL. What kind of repository is it? Sniffing is possible, but doesn't work that well. For pip I require something like hg+http://... (with exceptions for schemes that are self-describing, like svn: and git:). It's also unclear whether it should point to the trunk, a branch, or a tag (related to the release). As a URL that's only even possible for svn, as git/hg/bzr all point to the repository and URLs can't point to revisions, tags, etc. (pip has some extensions to the URL format to accommodate these, but they aren't formally defined at this time). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 345 - 3 new fields
On Mon, Nov 30, 2009 at 4:30 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Mon, Nov 30, 2009 at 8:17 PM, Ian Bicking i...@colorstudy.com wrote: On Sun, Nov 22, 2009 at 4:52 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: As suggested in Catalog-SIG by some people, I would like to propose the addition of three more fields for 1.2: Repository-URL A string containing the URL for the package's repository. Example: http://svn.python.org/projects/python/trunk/ I like this in concept, but it's a bit vague as just a URL. What kind of repository is it? Sniffing is possible, but doesn't work that well. For pip I require something like hg+http://... (with exceptions for schemes that are self-describing, like svn: and git:). I didn't think that Pip would use this field to do some automatic process, I think the use case was to simply display that url at PyPI. OTHO, Repository-Browse-URL already plays that role, so it would be interesting to provide in Repository-URL, urls that could be processed by installers. Do you have a particular use case in mind ? I don't really. There is an active use case for links like: http://repo-location/MyPackage/trunk#egg=MyPackage-dev which are used to get the dev version. These links should go to a svn index (which both easy_install and pip understand specifically) or to some automatically-generated tarball (which I believe most of the other VCSs generate through their popular frontends, e.g., http://bitbucket.org/ianb/webob/get/tip.zip). If something more like that was added (more like Trunk-URL), that would be genuinely useful. Barring that, I'd recommend removing the Repository-URL field from the recommendation until we have a clearer use case. Right now, the better use case would be a field giving the checkout command, e.g.,: Repository-Checkout: hg clone http://bitbucket.org/ianb/webob But I don't think that is particularly useful, only that it is more useful than Repository-URL. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Sun, Nov 8, 2009 at 10:28 PM, Robert Kern robert.k...@gmail.com wrote: David Cournapeau wrote: I don't understand what's there to buy. Several people reported distutils errors without any backtrace, though a fair shared of those were caused by our own extensions. distutils specifically swallows exceptions and formats them for users by default. After all, it is trying to behave like a regular command line program that interacts with users, not developers. This is easily overridable for developers who are trying debug problems by setting the environment variable DISTUTILS_DEBUG=1. This will make distutils just give the traceback. In the tools I've written, I generally give the traceback if the verbosity is turned up, and in a case like this (an unexpected exception -- for distutils that's any exception except a few that distutils defines) I include use -v for more. I think distutils (or Distribute) could do the same. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] People want CPAN :-)
On Mon, Nov 9, 2009 at 1:09 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: David Lyon david.l...@preisshare.net writes: On Sun, 8 Nov 2009 23:28:42 -0600, Ian Bicking i...@colorstudy.com wrote: In the tools I've written, I generally give the traceback if the verbosity is turned up, and in a case like this (an unexpected exception -- for distutils that's any exception except a few that distutils defines) I include use -v for more. I think distutils (or Distribute) could do the same. Hi Ian, I think PIP is quite an accomplishment. But don't you think that its a big ask to refactor distutils/distribute to redo their error messages for package building? I've just had a read through the code for ‘pip’; AFAICT, the “redo the error messages for package building” essentially amounts to using the ‘logging’ module. Is that a big ask? pip doesn't use the logging module, it uses its own logger, which is intended more for managing the output of a command-line program and not just post-mortem debugging. I don't think changing distutils to improve error output would be hard at all. It looks like there's a line in distutils.core that catches these exceptions (and doesn't look like it actually catches all exceptions?), and that can just be fixed. Another topic that has come up: I do agree subclassing makes it really hard to have multiple lines of development (specifically something like Setuptools or Distribute, along with ad hoc development in setup.py files). But I think that can be resolved. Perhaps, for instance, Distribute can have implementations of commands like build, that can easily be customized or extended without subclassing (e.g., by pre-build or post-build functions). I'd really be shocked if a rewrite of distutils was necessary, or even necessary to simplify things. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] small distribute improvement
On Fri, Oct 16, 2009 at 6:20 AM, Neal Becker ndbeck...@gmail.com wrote: I think it is confusing that easy_install blah doesn't do anything, and doesn't provide any clue, if it was to do an update. You need to say easy_install -U blah There are a few choices here. One might be that easy_install blah could notify you that an update is available. pip works somewhat similarly (it won't do anything in this case), but in the tip it will say Requirement already satisfied (use --upgrade to upgrade): blah. It doesn't actually check if an upgrade is available though. It actually keeps track of why something is installed, so it can tell the difference between something installed from the command line and installed as a requirement of some other package (where you definitely want to quickly pass over satisfied requirements). On the other hand, I find it useful sometimes to test things manually knowing the behavior is the same on the command line as with install_requires. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [RFC] Recentering the static metadata need : PKG-INFO
On Fri, Oct 16, 2009 at 8:54 AM, Tarek Ziadé ziade.ta...@gmail.com wrote: Marc-André Lemburg gave me a key points about the static metadata discussions we have (wether its PEP 390 or around it) He said that the important thing was to have the context-dependant markers in PKG-INFO, and that having in described in setup.cfg or in setup.py by any way is not the important thing. And he is right ! Excellent, this resolves my own concern about the discussion as well. Putting this in PKG-INFO is something I can concretely make use of, regardless of how it is generated. PKG-INFO is already somewhat flawed as a format for holding the data, in particularly maintaining indentation for Description. I think adding a general ...; condition places another syntactic constraint, where no field can have ; in it. Ideally I'd like to see both cases resolved. And certainly I don't want to end up in a place where weird bugs emerge if I put ; in a field (especially since many are free text). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 390 - new format from setup.cfg
On Mon, Oct 12, 2009 at 11:34 AM, Tarek Ziadé ziade.ta...@gmail.com wrote:
On Mon, Oct 12, 2009 at 6:10 PM, Ian Bicking i...@colorstudy.com wrote:
If you don't have tuples or , , etc, it seems like something like
Python version 2.6 or higher is hard to express. You'd have to
enumerate 2.6, 2.7, and speculate on 2.8 and 2.9.
python_version not in '2.3,2.4,2.5'
(it's not optimal, but enough I guess, until PEP 386 is accepted)
Given the small number of values available, string comparison works
fine in this case. Is there a situation when the data would be
dependent on a package version other than Python's?
One use case for the current setup.cfg is for extensions, like
generating Sphynx documentation. Those extensions can sometimes take
quite a lot of options, so they are best when partitioned into their
own section. I'm also not sure whether [metadata] is intended to have
extensible variables, or a fixed set of variables.
The variables are fixed so it works with a vanilla python and doesn't
require anything else
to run. Having extensible variables would break that idea and would just
make people move code they use to have in setup.py in another place imho.
It should at least be allowed to have other values with a warning,
similar to keyword arguments to setup() that are unknown. This lets
the format be forward compatible with new variables. E.g., if in
Python 2.8 there's a new variable, people can just dump it in.
How would you define/provide features here ?
I'm not sure. With Setuptools the equivalent is extras, like:
setup(name='foo',
extras_require={'xmlrpc': ['lxml']})
Then if you do easy_install foo[xmlrpc] (or somehow require
foo[xmlrpc]) it will also install/require lxml.
[...]
...
So, since the result involves multiple sections it wouldn't naturally
map to what you are proposing.
That could be done I think as long as the extras are not calculated by
third-party code.
Let's say, we define an 'extras' variable, which value can be provided
when interpreting setup.cfg.
If could be described like this :
[metadata:'xmlrpc' in extras]
requires = lxml
The part that is unclear to me is how to list the extras a setup.cfg file has.
There needs to be some enumeration of extras, yes. And there really
needs to be more metadata than Setuptools allows for -- extras should
have at least a readable description. So wherever they are defined,
there should be room for that documentation as well.
And given other variables (ones that perhaps distribute doesn't even
know about) how are they combined?
Right, this needs clarification. In any case, I think using multiline
is a bad idea
because it'll break RCF 232 compatibility for the long_description field.
Oh... long_description -- I forgot about that. How will that be dealt
with? I now typically do something like this in setup.py:
long_desc = open(os.path.join(os.path.dirname(__file__), 'docs',
'index.txt')).read()
long_desc = long_desc.split('..contents::', 1)[1].strip()
setup(..., long_description=long_desc)
Keeping the long description in setup.py (or in setup.cfg) is not
something I want to do. It would be even worse in setup.cfg than it
is in setup.py. Also ConfigParser eats leading whitespace (I believe
that's also a problem with the PKG-INFO format), which breaks ReST.
(I assume the register command does something other than uploading
PKG-INFO to get long_description in place?)
(Come to think of it, I'm not actually sure what we're accomplishing
with this declarative metadata; setup.py --name is too minimal, but
some new setup.py metadata command that dumps everything isn't hard
to imagine; what are the *real* advantages of this new proposal? If I
have to use a build process to build my setup.cfg, then absolutely
nothing will be accomplished. And is the complexity all just because
sometimes people need to use version-specific requirements?)
--
Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Alternate static metadata PEP submission...
On Tue, Oct 13, 2009 at 6:36 PM, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: To me it would seem a little early to start a PEP like this, there's been virtually no discussion about this particular proposal nor any proof of concept code. And given the scope of wanting to change the entire way build-steps are declared some code that can give experience and show the good and bad points seems important to me. Could an example API be encapsulated in something like this in setup.py? from test_this_pep import setup_cfg setup(other args, **setup_cfg()) Then packages could be converted to test it out, without breaking the package. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 390 - new format from setup.cfg
On Mon, Oct 12, 2009 at 4:45 AM, Tarek Ziadé ziade.ta...@gmail.com wrote:
On Mon, Oct 12, 2009 at 2:29 AM, Ian Bicking i...@colorstudy.com wrote:
The grammar in Context-dependant sections indicates possible EXPR
values. Because the in operator is supported, I would assume that
tuples should also be allowed.
in here is restricted to string. It was added so we could write things like:
'linux' in sys_platform (where sys_platform can be linux2)
If you don't have tuples or , , etc, it seems like something like
Python version 2.6 or higher is hard to express. You'd have to
enumerate 2.6, 2.7, and speculate on 2.8 and 2.9.
I'll add a note on that.
One aspect of the current setup.cfg is that it supports multiple
sections, for different setup.py commands or components. This gives a
kind of namespacing. I assume, but it isn't that specified, that any
section (not just metadata) will be parsed the same way?
I guess yes, even if I don't see a use case yet for that.
One use case for the current setup.cfg is for extensions, like
generating Sphynx documentation. Those extensions can sometimes take
quite a lot of options, so they are best when partitioned into their
own section. I'm also not sure whether [metadata] is intended to have
extensible variables, or a fixed set of variables.
Presumably setup.py will just contain an empty call to setup()? (I
assume at least setup.py will be allowed for, for backward
compatibility, even if it is not required.)
No because we might need to define extra options in setup()
that are not part of the metadata, like what is required the for the
sdist command
(package_data, scripts, package, etc)
OK, so setup.cfg is for generating PKG-INFO, but installing a package
still involves running setup.py and some maybe-declarative code in
there.
I believe this does not include the concept of extra requirements.
Possibly it could as a side-effect of some variable available to the
language. Like:
[metadata:'xmlrpc' in features]
requires = lxml
Sets and the operator could be useful for this.
How would you define/provide features here ?
I'm not sure. With Setuptools the equivalent is extras, like:
setup(name='foo',
extras_require={'xmlrpc': ['lxml']})
Then if you do easy_install foo[xmlrpc] (or somehow require
foo[xmlrpc]) it will also install/require lxml.
If I was to say why this is a problem, I think it's because the
default is that there are no features for a package. So someone
naively wants your package and installs it, but they don't get all the
features they thought the package provided (and which the package
actually *does* provide, but just doesn't have the prerequisite
libraries for -- since the package ships with the xmlrpc code, it's
just not working xmlrpc code). There's also not a good way of seeing
what extras are provided, or what their purpose is. So library
authors avoid the issue entirely and don't factor out requirements
into specific extras/features. If there was a set of default extras
maybe it would be more workable. I.e., easy_install foo installs
foo with all its default extras, and easy_install foo[] installs foo
without any extras (or you put the specific extras you want in []).
Anyway, the way extra requirements are serialized to disk is requires.txt, with:
default_req1
default_req2
[extra]
extra_req1
...
So, since the result involves multiple sections it wouldn't naturally
map to what you are proposing.
The way variables are handled is unclear. Presumably all variables
are cumulative. But given:
requires = Foo
requires = Bar
What is the value of requires? A list, a value with newlines?
Presumably you could also do:
requires = Foo
Bar
Anyway, we're diverging from INI semantics at this point, so it needs
to be specified how it works.
Right, this needs to be defined. I would be in favor of the latter, to
stay ConfigParser
compatible.
You still have to define how options are combined from multiple
sections, and what the resulting value is from the API. That is, if
you have:
[metadata]
requires = Foo
Bar
[metadata:python_version == '2.4' or python_version== '2.3' or
python_version=='2.2']
requires = BackwardCompat
Then what is the ultimate value of requires? Is it
Foo\nBar\nBackwardCompat or [Foo, Bar, BackwardCompat], or
[Requirement.parse(Foo), Requirement.parse('Bar),
Requirement.parse(BackwardCompat)].
And given other variables (ones that perhaps distribute doesn't even
know about) how are they combined?
Is there a way to eliminate values? Right now, for instance, if you
have [build] home = /foo in a distutils.cfg, there's no way to unset
that. I'd like to see this functionality included. Perhaps to delete
a specific value, but the simpler case of deleting a variable is
really all I want.
Do you have a syntax in mind for this ?
Well, the way I added more meta-operations in Paste Deploy (which uses
ConfigParser
Re: [Distutils] Eggs vs bdist_wininst (Was: Distutils and Distribute roadmap (and some words on Virtualenv, Pip))
On Sun, Oct 11, 2009 at 3:39 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: So why is there a need for an egg binary format on Windows? I think the only extra specific feature bdist_egg provides is the ability to use the self-contained egg (zipped or not) directly in sys.path without having to install it in a site-packages like directory. That's how zc.buildout works for example to isolate an execution environment : it collects eggs in a directory called 'eggs', then creates scripts with a modified sys.path that lists the paths of each egg located in 'eggs'. That's a feature of the installed (multi-version) egg, not the distribution format. That feature can be kept (or discussed, or whatever) separately from the discussion of bdist_egg as a distribution format. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 390 - new format from setup.cfg
The grammar in Context-dependant sections indicates possible EXPR values. Because the in operator is supported, I would assume that tuples should also be allowed. One aspect of the current setup.cfg is that it supports multiple sections, for different setup.py commands or components. This gives a kind of namespacing. I assume, but it isn't that specified, that any section (not just metadata) will be parsed the same way? Presumably setup.py will just contain an empty call to setup()? (I assume at least setup.py will be allowed for, for backward compatibility, even if it is not required.) I believe this does not include the concept of extra requirements. Possibly it could as a side-effect of some variable available to the language. Like: [metadata:'xmlrpc' in features] requires = lxml Sets and the operator could be useful for this. The way variables are handled is unclear. Presumably all variables are cumulative. But given: requires = Foo requires = Bar What is the value of requires? A list, a value with newlines? Presumably you could also do: requires = Foo Bar Anyway, we're diverging from INI semantics at this point, so it needs to be specified how it works. Is there a way to eliminate values? Right now, for instance, if you have [build] home = /foo in a distutils.cfg, there's no way to unset that. I'd like to see this functionality included. Perhaps to delete a specific value, but the simpler case of deleting a variable is really all I want. On Sun, Oct 11, 2009 at 5:17 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: Hey this is the PEP for setup.cfg, as requested : http://www.python.org/dev/peps/pep-0390 Please comment, Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与 ___ Distutils-SIG maillist - distutils-...@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] the virtualenv-distribute mess
On Fri, Oct 9, 2009 at 5:54 AM, Chris Withers ch...@simplistix.co.uk wrote: kiorky wrote: Hi Lennart, If i read 'virtualenv-distribute 1.3.4.2 on pypi' I can do some googling or even do some Pypi searching for 'virtualenv-distribute'. Thus, the first link found may be [1]. On this link, the second sentence is: The fork was started by Philip Jenvey at http://bitbucket.org/pjenvey/virtualenv-distribute/ and this version by Florian Schulze lives at http://bitbucket.org/fschulze/virtualenv-distribute/ Add http://bitbucket.org/kiorky/virtualenv-distribute/ to the list and this, ladies and gentlemen, is why distributed source control sucks. I think one (pjenvey) was an experiment, and another is actually a released virtualenv-distribute package (updating the name in setup.py, etc). And the third, I dunno. Anyway -- I'm reluctant to switch virtualenv to distribute right now, as I'm not confident it is ready for people to use distribute without even realizing it. Because if someone just upgrades virtualenv and starts using it, they'll get whatever virtualenv is distributed with. I'd be happy to include a --distribute option if someone wants to repackage one of these forks to include both setuptools and distribute support. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)
Probably all these discussions are better on distutils-sig (just copying python-dev to note the movement of the discussion) On Fri, Oct 9, 2009 at 11:49 AM, Michael Foord fuzzy...@voidspace.org.uk wrote: Outside of binaries on Windows, I'm still unsure if installing eggs serves a useful purpose. I'm not sure if eggs are any better than wininst binaries either...? Many Windows users would be quite happy if the standard mechanism for installing non-source distributions on Windows was via the wininst binaries. I wonder if it is going to be possible to make this compatible with the upcoming distutils package management 'stuff' (querying for installed packages, uninstallation etc) since installation/uninstallation goes through the Windows system package management feature. I guess it would be eminently possible but require some reasonably high level Windows-fu to do. As far as pip works, it unpacks a package and runs python setup.py install (and some options that aren't that interesting, but are provided specifically by setuptools). Well, it's slightly more complicated, but more to the point it doesn't install in-process or dictate how setup.py works, except that it takes some specific options. Running a Windows installer in the same way would be fine, in that sense. Alternately pip could unpack the wininst zip file and install it directly; I'm not sure if that would be better or worse? If wininst uses the central package manager of the OS then certain features (like virtualenv, PYTHONHOME, --prefix, etc) would not be possible. For Distribute (or Setuptools or by association pip) to see that a package is installed, it must have the appropriate metadata. For Setuptools (and Distribute 0.6) this is a directory or file, on sys.path, Package.egg-info (or in Package-X.Y.egg/EGG-INFO). If a file, it should be a PKG-INFO file, if a directory it should contain a PKG-INFO file. So however the package gets installed, if that metadata is installed then it can be queried. I don't think querying the Windows system package management would be necessary or desirable. Nobody is trying that with deb/rpm either. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] tracking requested vs dependency installs in PEP 376 metadata
On Thu, Oct 8, 2009 at 11:39 AM, Carl Meyer c...@meyerloewen.net wrote: I propose adding a metadata file REQUIRED within the .egg-info directory. The presence of this file indicates that the user specifically required this distribution. The absence of the file indicates that the distribution was installed as a dependency. The contents of the file are not used. I can imagine adding a little information, basically a log of when and why and who installed the package. For instance: agent: pip 0.5 install-date: 2009-10-08T13:44:00UTC installed-for-user: False installed-for-package: OtherPackage==0.3 Potentially a package could have multiple records, because multiple installers may in some sense touch a package (e.g., if you install YetAnotherPackage that requires the same library as OtherPackage). You could use INI-style and maybe label each record with the date, like [2009-10-08T13:44:00UTC]. This information seems fairly easy to generate. Updating it after installation would be nice, but also means already-installed packages can be written to, which is not as nice IMHO. Being unable to write to this file should be a non-fatal error for an installer. Either way, a package could become REQUIRED (or user-requested) at any time after it is installed. E.g.: easy_install Markdown easy_install ElementTree # which is required by MarkDown Now ElementTree should not be considered orphaned if MarkDown is removed. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] dev versions
So after creating, say, version 0.3.1, I always mark a package as 0.3.2dev. But this is annoying, you might never create a version 0.3.2 (e.g., 0.4 might be the next level). So, it would be better to use something like 0.3.1~dev. What is considered best practice for this? Ideally something that works with both Setuptools and the upcoming Distribute version spec. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] dev versions
On Thu, Oct 8, 2009 at 5:06 PM, Jim Fulton j...@zope.com wrote: On Thu, Oct 8, 2009 at 5:41 PM, Ian Bicking i...@colorstudy.com wrote: So after creating, say, version 0.3.1, I always mark a package as 0.3.2dev. But this is annoying, you might never create a version 0.3.2 (e.g., 0.4 might be the next level). So, it would be better to use something like 0.3.1~dev. What is considered best practice for this? Ideally something that works with both Setuptools and the upcoming Distribute version spec. I like using a version of 0 on my project trunks. I set the release version on release tags. I really wish there was a special version (or a version pattern) that indicated that something is a development version *only* and can't be released. I don't think best practices have been established. It would be nice if there was a sense of branches. E.g., if I fork a project, say setuptools-0.6c3, I could make setuptools-ianb-0.6c3 and someone could install, say, setuptools==ianb, getting whatever was the newest version of my branch. But ianb-0.6c3 wouldn't be comparable to any other version except versions on that branch. Though once installed it would satisfy a generic setuptools requirement. This could be used for a dev branch as well, which would satisfy a requirement but not be considered part of the same version series as the stable releases. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] something else to throw into the static metadata mix
2009/9/30 Sridhar Ratnakumar sridh...@activestate.com On Wed, 30 Sep 2009 12:26:15 -0700, cgal...@mail.utexas.edu wrote: I agree with Tarek that the system dependency handling is not something that should be dealt with in the python package metadata. The best thing that can be hoped for is that packages which rely on a specific version of a system dependency will throw a descriptive error indicating this particular problem. The MySQL_python project is one example of a project that is already doing this, since they throw an ImportError with a descriptive title in this case. The other option could be to enable support for static linking so as to make the egg a reasonably self-sufficient binary (eg: lxml's --static flag to setup.py) Just to throw more in the mix -- this option is totally ad hoc in lxml, which means it's hard to handle in an installer like easy_install or pip. That is, there's no good way to do easy_install --static lxml (it's more possible in pip, but still quite awkward). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bootstrapping virtualenv?
On Tue, Aug 11, 2009 at 5:06 AM, Chris Withers ch...@simplistix.co.ukwrote: Am I right in thinking the correct way to get a virtualenv without installing virtualenv into your site-packages is: - download a source distro of virtualenv - unpack it and copy virtualenv.py somewhere - use that virtualenv.py with whatever python you want to create your virtualenvs? You don't actually have to install virtualenv, just grab virtualenv.py (e.g., directly from svn) and run it. (Ditto pip) -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools: can I specify different install_requires for different python versions?
On Thu, Jul 30, 2009 at 2:28 PM, Kevin Teagueke...@bud.ca wrote: ... It would be nice if the python standard library was packaged up into a set of distributions in some sensible way. These distributions would still be pulled into main Python release so that they are installed the same as they always have, but the [python-install]/lib/ python/ directory would contain .egg-info files, so that it would be possible to tell that the 'multiprocessing' distro is supplied by the standard lib, and what version of the distro is included in a particular Python release. This is done for ElementTree (and maybe a couple other things?), but I agree, it would be great for every new addition to the standard library. It would be possible to create a script that did this for existing Python installations. Necessary in a sense, because 2.6.0 installations can't be changed (along with every other old installation), and dealing with older versions is the entire point of such an exercise. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Cache PYTHONPATH? (Re: make unzipped eggs be the default)
On Tue, Jul 28, 2009 at 8:02 PM, Greg Ewinggreg.ew...@canterbury.ac.nz wrote: P.J. Eby wrote: So the optimum performance tradeoff depends on how many imports you have *and* how many eggs you have on sys.path. If you have lots of eggs and few imports, unzipped ones will probably be faster. If you have lots of eggs and *lots* of imports, zipped ones will probably be faster. I'm wondering whether something could be gained by cacheing the results of sys.path lookups somehow between interpreter invocations. Most of the time the contents of the directories on one's PYTHONPATH don't change, so doing all this statting and directory reading every time an interpreter starts up seems rather suboptimal. I can see how this could go quite wrong, but maybe if installers touch some file in the library directory anytime a package is installed/reinstalled/removed/etc, then it would be fast to check if the cache was correct. Though the optimization seems like its working around something that maybe shouldn't be a problem. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Cache PYTHONPATH? (Re: make unzipped eggs be the default)
On Tue, Jul 28, 2009 at 9:40 PM, P.J. Ebyp...@telecommunity.com wrote: At 09:22 PM 7/28/2009 -0500, Ian Bicking wrote: I can see how this could go quite wrong, but maybe if installers touch some file in the library directory anytime a package is installed/reinstalled/removed/etc, You mean, like, the mtime of the directory itself? ;-) Do directory mtimes get recursively updated? I don't think they do. So if you have a layout: site-packages/ zope/ interface/ __init__.py And you update the package and update __init__.py, the mtime of site-packages doesn't change, does it? I'm saying if there was a file in site-packages/last_updated that gets touched everytime an installer does anything in site-packages, then you could cache (between processes) the lookups. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] version scheme: a case for dropping .devNNN and .postNNN
On Thu, Jun 11, 2009 at 10:53 AM, P.J. Ebyp...@telecommunity.com wrote: At 03:16 PM 6/11/2009 +0100, Paul Moore wrote: 2009/6/11 P.J. Eby p...@telecommunity.com: PyPI uploads aren't a suitable basis for analyzing dev use cases, since the whole point of having a dev tag is for *non-released* versions. (E.g., in-progress development via SVN.) If it's non-released, I've yet to see a clear explanation of why the PEP is relevant. Who is going to use an API from the PEP to parse your version number, and why? Dev tags are so that while you're doing development, your locally-installed versions can be distinguished from one another. Distinguished by what? What code (that you didn't write yourself, purely for internal use) needs to parse your dev tag? Distinguished by setuptools for processing version requirements of scripts, or require() statements in code, and installation requirements of newly-installed code. For example, if I'm working on two projects that are distributed via SVN and one depends on the other, if I update one, it may require an update of the other; the failure of the .dev version requirement in the first one will inform me of the need to svn up the second project and rerun setup.py develop on it. This is a routine circumstance in at least my development cycle; I would expect that it's the case in other open source development workflows as well as proprietary ones. Agreed, I do this all the time. Pylons dev versions also regularly rely on other packages with a dev version, and people regularly use these non-released versions, with dependencies detected and installed via dependency_links. -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Adding entry points into Distutils ?
On Wed, May 6, 2009 at 1:32 PM, Tres Seaver tsea...@palladion.com wrote:
I'd be tempted to call this an edge-case. You should be able to expose
the internal detail you'd need via a module scope alias for the
particular case. That seems easier than providing a whole new notion.
I'm actually a big fan of the ':', because it makes explicit the
difference between the import and the named thing, even for
module-scoped names.
Yeah, I also like it primarily for clarity. Also you can provide better
error messages when it fails, matching up the error against the intention.
In some contexts I also extend this syntax to do something like:
module, path = name.split(':', 1)
obj = eval(path, load_module(module).__dict__)
I don't propose that we do eval for plugins, but it's a nice side-effect of
the syntax that it's possible to add in other contexts. Also I don't think
there's any strong precedence for purely dot notation, loading objects from
strings is something that's always done ad hoc, and the only widely used
library I know of that people use for this is Setuptools (indirectly through
entry points).
--
Ian Bicking | http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools, namespace packages, --single-version-externally-managed
Any ideas on this? Phillip?
On Fri, May 1, 2009 at 5:07 PM, Ian Bicking i...@colorstudy.com wrote:
So, a bit of a problem came up with pip and namespace packages. Here's my
understanding of what's going on:
When you install a namespace package with pip, it uses install
--single-version-externally-managed, and generally the namespace directory
is empty and there's a *.nspkg.pth file that has this:
import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('NAMESPACE',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m =
not ie and sys.modules.setdefault('zope',new.module('zope')); mp = (m or [])
and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
So the lack of an __init__.py file doesn't really matter, because it's
created right there, and has its __path__ added to. But there's a problem
when there's another namespace package elsewhere on the path, that wasn't
installed with pip (or Setuptools) and uses pkgutils.extend_path(__path__,
__name__). This doesn't get imported because of that .pth file, and the
.pth file doesn't itself use extend_path, so the path isn't searched. This
is currently happening with Zope packages installed with plain distutils,
then another package installed with the zope namespace elsewhere with pip.
(When using easy_install, I think pkg_resource.declare_namespace comes into
play somewhere, and this seems to handle this case, but I'm not sure why the
installation is different with pip.)
So... what should pip be doing differently to make this work?
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] RFC : PEP 376 - egg.info
On Mon, May 4, 2009 at 11:31 AM, Tarek Ziadé ziade.ta...@gmail.com wrote: Ok then, we will have to provide extra documentation to make people understand that the '.egg-info' directory has absolutely nothing to do with egg-the-format but is rather a metadata container. 'egg-info' was introduced with adding the whole 'egg' thing in Python in mind at some point I believe. And it seems that the .egg directory/zip file for projects setuptools provides will not make it into Python and is still very controversial ( http://www.reddit.com/r/Python/comments/8he79/am_i_alone_in_feeling_like_python_got_a_whole_lot/ ) So removing the 'egg' part of 'egg-info' seemed natural to me at this point. Egg-the-format is what we are recreating in distutils, isn't it? Obviously some people are unhappy with some things related to packaging, but I don't think egg-the-format is something people actually mind (if they know what it is). Of course few people really know what the format is, or are able to distinguish it from other parts of the Setuptools stack, but that doesn't change just because you rename the extension. Eggs don't even carry any particular naming attachment to Setuptools. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Setuptools, namespace packages, --single-version-externally-managed
So, a bit of a problem came up with pip and namespace packages. Here's my
understanding of what's going on:
When you install a namespace package with pip, it uses install
--single-version-externally-managed, and generally the namespace directory
is empty and there's a *.nspkg.pth file that has this:
import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('NAMESPACE',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m =
not ie and sys.modules.setdefault('zope',new.module('zope')); mp = (m or [])
and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
So the lack of an __init__.py file doesn't really matter, because it's
created right there, and has its __path__ added to. But there's a problem
when there's another namespace package elsewhere on the path, that wasn't
installed with pip (or Setuptools) and uses pkgutils.extend_path(__path__,
__name__). This doesn't get imported because of that .pth file, and the
.pth file doesn't itself use extend_path, so the path isn't searched. This
is currently happening with Zope packages installed with plain distutils,
then another package installed with the zope namespace elsewhere with pip.
(When using easy_install, I think pkg_resource.declare_namespace comes into
play somewhere, and this seems to handle this case, but I'm not sure why the
installation is different with pip.)
So... what should pip be doing differently to make this work?
--
Ian Bicking | http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Questionnaire: Why do you use setuptools?
On Sat, Apr 18, 2009 at 4:09 AM, Lennart Regebro rege...@gmail.com wrote: Setuptools non-support for Python 3 is currently a serious hindrance towards Python 3 aceptance. I'm trying to figure out what to do as a next step in the Python 3 support for setuptools. And I have encountered some obstacles. The first one is that setuptools requires itself for installing and running tests. That makes it hard to install it under Python 3. There are various solutions to this, but the next obstacle I encounter in choosing the right solution is that the code is hard to understand, and it makes me want to just rip it out and start over, or in even more frustrated moments, avoid the problems by not using setuptools at all. But the third obstacle for that is that I don't actually know what features of setuptools people use. I personally use setuptools for these reasons: 1. When I create projects with paster, it uses setuptools. Of course if you create a project template that uses distutils, then that's what you'll get... just happens no one does that with paster templates. 2. Setuptools makes it possible to specify requirements, which is then used by buildout. In pip at least it does this with *.egg-info/install_requires.txt, but in easy_install and I'm guessing buildout it's using the pkg_resources.Distribution object. 3. Namespace packages require pkg_resources? There's a way of doing it with pkgutils, but in some way that I don't understand, pkg_resources does it better. 4. The test command. What are the other major reasons people use setuptools? Entry points are the big one for me. There is some other metadata that I use from time to time, but I'm sure I could work around it. Of course there's the specific things pip uses as noted in a previous email. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Help making setuptools install more like plain distutils one
Use --single-version-externally-managed (also pip using this flag by default, so if you install via pip you'll get this behavior). On Mon, Apr 20, 2009 at 12:28 PM, Christian Hudon chr...@apstat.com wrote: Hello, We have a setup where a central /usr/local is copied to all our machines. The packages installed in said central /usr/local are managed via stow. (Basically, each package is installed in a separate directory under /usr/local/stow, and invoking the stow command creates symlinks to make the package appear under /usr/local.) This works very well for a wide range of packages: autoconf packages, CRAN packages for R, etc. This includes plain distutils python packages (install via python setup.py install --prefix /usr/local/stow/some-package-name, then run stow). The only exception is setuptools-based python packages. Is there a way to ask setuptools to do an install that looks more like a standard distutils install? I don't care if I lose some the advanced setuptools features. Basically, I need an install that's done via just copying new files. Problems like setuptoosl checking that the destination directory is in the PYTHONPATH I can work around (although if there's a switch to disable that check, I'd be happy to learn about it). The main problem is the file that's edited on each install to add a new line for each package install via setuptools. Is there a way with setuptools of getting just a directory tree (or a tarball, etc.) that either setuptools or myself can just copy somewhere to have an installed python module? Thanks in advance for any help, Christian ___ Distutils-SIG maillist - distutils-...@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 376 - install/uninstall script in Distutils ?
OK, just thinking through a little what it would mean to have an installation tool in Python core... On Thu, Apr 16, 2009 at 5:08 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: But then I don't think Python should have a built-in installer or package manager. There are excellent tools already available (Buildout, pip, dpkg, RPM), it would be better if we guided people to these tools and let them pick the right one for their installation use case. I wouldn't put zc.buildout in the same level than pip or easy_install. and I guess what we would have in DIstutils would be quite similar to what easy_install or pip offers. I think there are questions about scope. zc.buildout does more than pip, and pip does more than easy_install. I think easy_install has some important usability problems, otherwise I wouldn't have written pip. pip on the other hand has features that extend its scope in ways that might make it hard to include in the standard library. For instance the version control support, requirement files, bundles, and some miscellaneous functionality like zipping. Some of that could be separated out, though the version control support is more difficult. Also it really mostly makes sense in the context of virtualenv. I'm strongly considering having an interactive warning if you try to install something with pip into the global site packages directory. PYTHONUSERBASE is an okay solution (not great, but okay), so I don't think this is contingent on something like virtualenv being standard... but it would help. I'm not sure how I'd pursue the virtualenv concept in Python core, as it's more a question of the concept of virtualenv than the implementation itself, but if there was general interest I could try. Anyway, it seems odd to me to include a tool that hasn't been written yet, when there are tools that are being used and developed. OTOH, the only tool that is stable enough (not just bug wise, but stable as in isn't-changing) is easy_install. But while pip uses setuptools, it doesn't use easy_install at all, so including easy_install would really only make pip development harder. That said, there might be parts of pip or easy_install which would be useful. I'm not sure what those would be though. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What's missing from easy_install
For people interested in uninstall, you might want to check out this branch of pip: http://bitbucket.org/carljm/pip-uninstall/overview/ It only uninstalls things pip installed (because pip is keeping a record of installed files, which is used for the uninstallation). On Tue, Apr 7, 2009 at 10:05 AM, Neal Becker ndbeck...@gmail.com wrote: 1. easy_remove! 2. Various utilities to provide query package management. - easy_install --list (list files installed) -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] What pip uses setuptools for
During the discussion at the open space about distutils, it was noted several tools use Setuptools. Right now I think we all agree (probably PJE too) that Setuptools implements more than we can move into the standard library. Since these tools represent clear use cases, figuring out how to get these tools to not depend on Setuptools will, I think, factor out many of the most important APIs. And it so happens that pip uses just a few things, so factoring out this functionality seems feasible: I call setup.py egg_info. This is currently awkward, because where the egg-info information goes is hard to figure out. But it works, and gives me parseable metadata. (Note, all calls to setup.py are done in a subprocess; I think some tools call setup.py in-process, but subprocesses seemed safer to me and I don't see much of a downside, certainly not performance.) I install using Setuptools, which installs the egg-info metadata, and also records the files that were written. (I write these to a temporary location, rewrite the file paths, and put the list of files into the package's egg-info). I install using setup.py develop, using Setuptools. This calls egg_info, and also handles fixing up a .pth file. I use --no-deps, because I never want any of these installation processes to install anything. I've copied some code out of setuptools.command.egg_info for getting the svn revision. I'm fine maintaining this code (since there's also code for other version control systems). The uses of pkg_resources: I look around at all the packages' dependency_links.txt, as this gives pip a hint of where things came from, and this lets me better implement pip freeze. Some people hate dependency_links, but I don't at all. I get a list of all the installed packages (with versions), for pip freeze. It is very helpful that people use -r1234 for svn checkouts, as I detect this and use this to create sensible frozen versions. Handling version numbers for unreleased packages is important. I use pkg_resources to parse versions, test for which is better, and test for inclusion (whether a version satisfies a requirement). Oh, and I parse requirements. I would like if there was a distinction between stable and unstable versions included in the version discussion. I have some code to normalize project names. The code is awkward, and kind of uses pkg_resources. I'd like this handled more nicely. I think Setutools' case insensitivity is useful, as is some other forms of normalization. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PyPI mirroring and Changing the .egg-info PEPs
Suggestions: Instead of http://pypi.python.org/mirrors how about http://mirrors.pypi.python.org (or pypimirrors.python.org, etc) -- then the mirror list can be hosted separately, and the likelihood of both the mirrors and the main PyPI server being down are much lower. Admittedly, this makes it harder for other indexes to provide their own mirrors, and the location is just entirely magic. But it seems like a useful magic. In the egg-info PEP: They both add other files in the `EGG-INFO` or `.egg-info` directory, and create or modify `.pth` files. `pip` also creates one `.pth` file per installed package, which may lead to slow initialisation of Python. I don't believe this is the case. pip zip creates one .pth file for each zip file, but that's different. pip installs packages flat. Unless you use -e, which invokes setup.py develop, which I believe puts the path in easy-install.pth. Oh, you also use British English; initialization! I only notice that because my spell check is complaining about the quotation ;) As mentioned in another email, I think RECORD should hold relative paths (and PJE's / suggestion also of course seems reasonable). The paths should be relative to the .egg-info directory. - get_egg_info(pkg_name) - path or None Scans all site-packages directories and looks for all `pkg_name.egg-info` directories. Returns the directory path that contains a PKG-INFO that matches `pkg_name` for the `name` metadata. Notice that there should be at most one result. If more than one path matches the pkg_name, a DistutilsError is raised. If the directory is not found, returns None. Currently, if there are multiple paths that might be importable, whichever one is found first with sys.path is imported, and conflicts aren't really considered. raising DistutilsError seems like it changes this. filename can be any value found in `distutils.sdist.EGG_INFO_FILES`. Is there any reason this should be restricted to those filenames? -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 376 for Distutils
On Wed, Feb 25, 2009 at 4:16 PM, P.J. Eby p...@telecommunity.com wrote: I wonder why they don't just use the sitewide distutils.cfg file, which would let them configure user-installed packages to go to somewhere else, e.g.: [install] prefix = /usr/local (And of course the build tools could specify different options.) Some systems do this. I've found this problematic with virtualenv, as people get weird results because they don't realize this configuration file exists. Also there's no way currently in distutils to unset a value, so if something like prefix is given explicitly there's no way to indicate it should be calculated (you just have to give an explicit value for --prefix). -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python Language Summit] Distutils / Packaging survey
On Sat, Jan 31, 2009 at 2:40 AM, David Cournapeau
da...@ar.media.kyoto-u.ac.jp wrote:
Ian Bicking wrote:
On Fri, Jan 30, 2009 at 12:39 PM, Floris Bruynooghe
floris.bruynoo...@gmail.com mailto:floris.bruynoo...@gmail.com
wrote:
I imagine things like libdir, prefix, datadir, docdir and other
things
copied from autoconf. Where the defaults would be something like:
prefix = sys.prefix
libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname
datadir = sys.prefix/share/mypackage
docdir = sys.prefix/share/doc/mypackage
I wouldn't want to use those. What goes in libdir, what goes in
datadir? I don't know, and frankly the distinctions start getting
really arbitrary.
They are not arbitrary - they come from standard usage and have a
rationale, at least on Unix (datadir for arch independent, and libdir
for arch dependent, to simplify).
I'm just about ready to run screaming from this discussion... so no, I want
no part of determining what the right place for these files is, nor do I
find it self-evident.
But you mostly do not need to care, as a developer: .py files would be
considered as data files, extensions as arch-dependent, etc... The main
category which needs special care is documentation, and I think I am not
the only one thinking that's one thing missing in distutils ATM.
I would rather see something like pkg_resources existing API, where
there is some file that maps out how the local names of files (where
they'd be in a checkout) map to their installed location, then the
pkg_resources code could finds the real location of the file.
I am not sure I understand how this would help OS packagers - this does
not sound as the same problem at all.
Sorry, I didn't describe what I meant.
I imagine some file like package-data.conf, containing:
data mypackage/templates/
docs docs/_build/
At least in this example the first word is some tag, and the second is the
directory, or files, or maybe a wildcard or something determining what files
that tag applies to. Everything not declared but present in a package (or
as a module) would default to lib, and everything outside of that (like
the setup.py file) would be ignore.
On installation, you'd write something like
mypackage.egg-info/file-locations.txt, that might look like:
mypackage/templates/ - /usr/share/mypackage/templates/
docs/_build/ - /usr/share/doc/mypackage
(I'm not sure what the syntax would look like, but whatever.) Then when I
did something like pkg_resources.resource_string('mypackage',
'mypackage/templates') it'd look up this file to find the location (in the
absence of the file it'd act like it does currently)
--
Ian Bicking | http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python Language Summit] Distutils / Packaging survey
On Fri, Jan 30, 2009 at 12:39 PM, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: I imagine things like libdir, prefix, datadir, docdir and other things copied from autoconf. Where the defaults would be something like: prefix = sys.prefix libdir = sys.prefix/lib/pythonX.Y/site-packages/pkgname datadir = sys.prefix/share/mypackage docdir = sys.prefix/share/doc/mypackage I wouldn't want to use those. What goes in libdir, what goes in datadir? I don't know, and frankly the distinctions start getting really arbitrary. I would rather see something like pkg_resources existing API, where there is some file that maps out how the local names of files (where they'd be in a checkout) map to their installed location, then the pkg_resources code could finds the real location of the file. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python Language Summit] Distutils / Packaging survey
On Thu, Jan 29, 2009 at 7:53 AM, Josselin Mouette j...@debian.org wrote: Le mercredi 28 janvier 2009 à 13:16 -0600, Ian Bicking a écrit : As you mention, there would have to be some extension to pkg_resources (or an equivalent library) to handle finding these files at runtime. Getting a runtime in place is probably the harder thing, as it is more intrusive for the upstream developers. As I have already explained in the previous discussion, this could easily be solved just like autoconf does, with an automatically generated config.py file that would hold all variables set at build time. Easily solved is a misstatement -- you can lead a horse to water, but you can't make him drink. In this case, you can provide infrastructure for library authors, but you can't make them use it. The solution has to be sufficiently simple that people who don't care at all about FHS or Linux packages won't find it onerous to use, because if they do find it onerous then they won't use it. But anyway, where would this config.py go and what would it look like? -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Uninstall command, the return
On Thu, Jan 29, 2009 at 6:49 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: Next, (in a second step) I was wondering if a uninstall registery could not be a good thing to have, to store a record of the installed files so there's no need to keep the source for uninstallation. This would required a new command, (and a detailed specification of course) pip writes an installation record in Package.egg-info/installed-files.txt (based on the setuptools --record option, with filenames made relative). So... that's similar to it. Of course, to be accurate you have to make sure you don't install over those files. So pip should really be uninstalling before installing something new, and probably be fancy about the whole thing (maybe like Enstaller is doing). But if tools do respect the integrity of those files, it's a reasonably simple record. Well, that and they should be careful about one package overwriting another packages file (which I haven't really seen happen, but of course it *could* happen). -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python Language Summit] Distutils / Packaging survey
On Wed, Jan 28, 2009 at 11:03 AM, zooko zo...@zooko.com wrote: David 2. entry_points doesn't work when the svn checkout is just in PYTHONPATH or ., and fails if there is a non-existent directory on his PYTHONPATH. Probably this is because the package isn't activated, and if it's not activated you can't see its entry point. When a .pth file is on PYTHONPATH, Python won't load it up (it only loads .pth files in some specific locations). So while easy-install.pth would normally activate a package (by adding it to sys.path), with PYTHONPATH that doesn't work. I think the site.py that Setuptools will sometimes create is intended to address this, but it might not always work. Or there might be some entirely different problem I'm unaware of. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Python Language Summit] Distutils / Packaging survey
On Wed, Jan 28, 2009 at 5:05 AM, David Cournapeau da...@ar.media.kyoto-u.ac.jp wrote: I meant that instead of installing almost everything indistinctly like we do now with distutils/setuptools, we should have something like: python setup.py install --bindir=foodir --sbindir=bardir --mandir=mandir E.g. just copying the categories from autoconf (the ones which make sense for python packages). So making a FHS-compliant or something like currently done is the responsibility of the packagers - assuming the directories are correctly set by the developer in the setup.py. The main problem is how to retrieve the different files when it is needed in the package - I would guess pkg_resources would need to be modified as well for that purpose, In the previous discussion (somewhere, I'm too lazy to look it up) people started getting interested in the idea of an improved sdist format. I think the basic idea was to tag files by type. Then (I guess) you'd just do: python setup.py build to get the platform-specific parts of the library built, and then move the files into place based on the descriptions of the files. The actual tool to do this would be external to setup.py and distutils; e.g., you'd have a python-sdist-to-rpm tool, developed entirely separately from distutils or setuptools. Potentially this would make it easier to provide your own file tags, so you could adapt a library without requiring immediate upstream support or patches. As you mention, there would have to be some extension to pkg_resources (or an equivalent library) to handle finding these files at runtime. Getting a runtime in place is probably the harder thing, as it is more intrusive for the upstream developers. -- Ian Bicking | http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] easy_install adds bad interpreter shebang to installed scripts
Tres Seaver wrote: I would way rather see that kind of solution than using 'env': scripts installed by easy_install should *not* use whatever python happens to be found at the moment on PATH. I agree that's necessary, but I don't think anyone has been proposing that (well, except the initial proposal). I'm guessing the script in this case could look like: #!/bin/sh exec path/to/python -c everything that would normally be in the body of the script -- Ian Bicking : i...@colorstudy.com : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] easy_install adds bad interpreter shebang to installed scripts
Phillip J. Eby wrote:
At 06:15 PM 12/17/2008 +0100, Felix Schwarz wrote:
Index: setuptools/command/easy_install.py
===
--- setuptools/command/easy_install.py (Revision 67824)
+++ setuptools/command/easy_install.py (Arbeitskopie)
@@ -1418,8 +1418,12 @@
if options: options = ' '+options
if wininst:
executable = python.exe
+elif sys.platform=='win32':
+executable = nt_quote_arg(executable)
+elif ' ' in executable:
+executable = '/usr/bin/env python'
else:
-executable = nt_quote_arg(executable)
+executable = executable
hdr = #!%(executable)s%(options)s\n % locals()
if unicode(hdr,'ascii','ignore').encode('ascii') != hdr:
# Non-ascii path to sys.executable, use -x to prevent warnings
I'm okay with the change in principle, but in practice, just dropping to
/usr/bin/env python isn't acceptable. It should point env to the
specific sys.executable, just like the fix_jython_executable()
function does.
Does this work?
#!/usr/bin/env /path/to/weird path/python
--
Ian Bicking : i...@colorstudy.com : http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Unsetting something from distutils.cfg
Is there any way, when you have a setting in distutils.cfg, to unset that value? For instance, if you've set: [install] prefix = something Is there a way to get distutils to ignore this setting? Setting it to the empty string doesn't seem to do it, and I don't know of any other way. -- Ian Bicking : i...@colorstudy.com : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Stable versions
I've received a request that pip be able to be restricted to stable versions. It was suggested some kind of --alpha-ok or --beta-ok option, but this seems crude. I'd rather have it be part of the requirement. Maybe Package==stable? But there's also reason to do something like Package=2.0,stable. So while pkg_resources can parse Package==stable easily enough, it's not by itself really expressive enough (I suppose Package=2.0,==stable actually works okay). Also, there needs to be a definition of what versions are stable. And maybe a distinction between beta/rc and development, though I'm less worried about that. Are there definitions of this? Does zc.buildout do this? -- Ian Bicking : i...@colorstudy.com : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Stable versions
Phillip J. Eby wrote: At 06:44 PM 12/16/2008 -0600, Ian Bicking wrote: I've received a request that pip be able to be restricted to stable versions. It was suggested some kind of --alpha-ok or --beta-ok option, but this seems crude. I'd rather have it be part of the requirement. Maybe Package==stable? But there's also reason to do something like Package=2.0,stable. So while pkg_resources can parse Package==stable easily enough, it's not by itself really expressive enough (I suppose Package=2.0,==stable actually works okay). Also, there needs to be a definition of what versions are stable. And maybe a distinction between beta/rc and development, though I'm less worried about that. Are there definitions of this? I would suggest an option that sets a minimum allowable alphanumeric code, e.g. b to require beta or better, c for release candidate, etc. It's easy to scan a parsed version for pre-release tags that match. Setting such an option to final would reject any version containing a pre-release tag. Well, the problem with an option is that it's global. I can imagine wanting to install FirstPackage==stable, and SomethingElse (no specific version requirement). -- Ian Bicking : i...@colorstudy.com : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] pyinstall renamed to pip
I've renamed pyinstall to pip (last renaming, I promise). It now uses commands like pip install something. This will make it easier to add new commands in the future, with entirely different option signatures. New site: http://pip.openplans.org -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Symlinks vs API -- question for developers
Phillip J. Eby wrote: I think Ian's already said this, but the API itself has to do something more, and so far nobody's proposed an API that does anything more than what setuptools does in this area, from the developer point of view. (Except for the request that such an API be in the stdlib and thus avoid an extra dependency... but that of course introduces yet another implementation delay, if it means a new release of Python.) It's probably a bit easier than waiting for a release of Python -- if it's in a PEP, and will be in a release of Python, then the library will be blessed and people will pick it up much more quickly. Realistically most library developers would need to add the package as a requirement for some time, since they won't stop supporting older versions of Python that don't have that package. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Symlinks vs API -- question for developers
in the documentation) is to allow zipped eggs to work. Distributions have no reason to create zipped eggs so they have no reason to submit patches to upstream to support the pkg_resources api. * Distributions, further, don't want to install all-in-one egg directories on the system. The pkg_resources API just gets in the way of doing things correctly in a distribution. I've had to patch code to not use pkg_resources if data is installed in the FHS mandated areas. Far from encouraging distributions to send patches upstream to make modules use pkg_resources this makes distributions actively discourage upstreams from using it. * The API isn't flexible enough. EggTranslations places its data within the metadata store of eggs instead of within the data store. This is because the metadata is able to be read outside of the package in which it is included while the package data can only be accessed from within the package. 8) To a distribution, symlinks are just a hack. We use them for things like php web apps when the web application is hardcoded to accept only one path for things (like the writable state files being intermixed with the program code). Managing a symlink farm is not something distributions are going to get excited over so adoption by distributions that this is the way to work with files won't happen until upstreams move on their own. Further, since the install tool is being proposed as a separate project from the metadata to mark files, the expectation is that the distributions are going to want to write an install tool that manages this symlink farm. For that to happen, you have to get distributions to be much more than simply neutral about the idea of symlinks, you have to have them enthused enough about using symlinks that they are willing to spend time writing a tool to do it. So once again, I think this boils down to these questions: if we have a small library whose sole purpose is to abstract a data store so you can find out where a particular non-code file lives on this system will you use it? Realistically, no. If a distribution packager sends you a patch so the data files are marked correctly and the code can retrieve their location instead of hardcoding an offset against __file__ will you commit it? If it adds a dependency and an abstraction that isn't obvious, then no, I would not commit it. Just marking the files is fine, because it has no impact on other code. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification
Paul Moore wrote: My feeling, by the way, is that system packagers are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers) I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation. Users don't use Python modules, they use applications. Users only care that their applications work, that they can install applications without unnecessary conflicts, that the applications don't break based on unintentional environment changes (e.g., the value of PYTHONPATH). Packagers seem to care a great deal about having applications share libraries on the packaging level, but this is for their own accounting, there's no reason for users to care (except for the too-small-to-matter issue of disk space). Also, packagers seem to jump the gun on this library sharing, as they are concerned about libraries when one (or often zero!) applications depend on the library. Some widely used libraries seem reasonable, but for every widely used library there are a dozen or more niche libraries. Users also don't care about /usr/share or /usr/lib -- the only thing *I* ever care about is /usr/share/doc, /usr/bin, /etc, and maybe a man page. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals
Tarek Ziadé wrote: On Tue, Oct 7, 2008 at 2:42 PM, Phillip J. Eby [EMAIL PROTECTED] wrote: At 10:07 AM 10/7/2008 -0400, Tarek Ziadé wrote: The -m feature of setuptools is nice, but it activates one version at a time, and this is globlal to Python unless each application is handling the version switch, wich is pretty heavy. With or without the -m switch, scripts installed by setuptools will find the version they are specified to use, without the user needing to do anything. So, you can have a default version of an egg (used by the interpreter and non-setuptools scripts), and then some non-default versions that are used by scripts. zc.buildout and virtualenv also have their own ways of accomplishing the same thing, e.g., by hardcoding paths in an installed script. in a plain python setup, If foo 1.2 is the default, and a package wants use foo 1.4, it needs to specifically call pkg_resources.require() in the code, to activate it in sys.path before importing foo in the code. Since each package can list with setuptools its dependencies with versions in install_requires, how hard would it be to automatically call the right require() calls when the package is used ? require() is recursive, so as long as the original script is explicitly loaded (e.g., from a binary script, or something that loads eggs/entry points) then the proper versions will be loaded. Though as far as I know, pkg_resources won't remove other versions of the egg from the path, so it only works if there are no active versions of the eggs. Which isn't how many people install packages, so this feature of require() doesn't get used for much of anything (at least that I've seen). I'll also note that the require in setuptools-generated scripts causes pretty frequent problems for people, all to support this multi-version feature that no one really uses. An example of an easy way to cause the problem, if you do: python setup.py develop; svn up; python setup.py egg_info it'll break any scripts, or if you install a script in an unusual location, or use $PYTHONPATH but don't set $PATH so that you get an unexpected script that doesn't match your libraries -- since pyinstall is using --single-version-externally-managed, I kind of wish I could easily turn off the require() as well (I could monkeypatch setuptools to remove it, but I've been burned by going down that path before). -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] specifying dependencies
Chris Withers wrote: Ian Bicking wrote: Say I have a package that represents an application. We'll call it FooBlog. I release version 1.0. It uses the Turplango web framework (1.5 at the time of release) and the Storchalmy ORM (0.4), and Turplango uses HardJSON (1.2.1). I want my version 1.0 to keep working. So, I figure I'll add the dependencies: Turplango==1.5 Storchalmy==0.4 Why? I would have suggested: Turplango=1.5,2.0 Storchalmy===0.4,0.5 Then when Turplango 1.6 comes out it'll break my code. Then HardJSON 2.0 is released, and Turplango only required HardJSON=1.2, so new installations start installing HardJSON 2.0. But my app happens not to be compatible with that library, and so it's broken. OK... so, I could add HardJSON==1.2.1 in my requirements. Not could, should, in fact must. Relying on a dependency provided by library you're using is suicide. Again, I'd suggest: HardJSON =1.2.1,1.3 What does 1.3 mean? You imply there is a disciplined use of a versioning pattern, and that every release is a guarantee that the versioning has been properly declared. There isn't a common understanding of versions, and it's common that conflicts are released unintentionally. But then a small bug fix, HardJSON 1.2.2 comes out, that fixes a security bug. Turplango releases version 1.5.1 that requires HardJSON=1.2.2. I now have have to update FooBlog to require both Turplango==1.5.1 and HardJSON==1.2.2. Not if you'd followed my advice above. OK, change that to a small bug fix comes out as HardJSON 1.3, and the same problems follow. I don't know what the nature of future releases will be. Later on, I decide that Turplango 1.6 fixes some important bugs, and I want to try it with my app. I can install Turplango 1.6, but I can't start my app because I'll get a version conflict. Not if you'd followed my advice above. Now you've introduced an entirely different requirement -- for some reason I am supposed to have known that HardJSON 1.3 would break my code, but only Turplango 2.0 would cause a conflict. And Turplango 1.6 wouldn't So to even experiment with a new version of the app, I have to check out FooBlog, update setup.py, reinstall (setup.py develop) the package, and then I can start using it. Right, you're developing FooBlog by changing the software it uses, so it seems natural enough to have to edit FooBlog code. You don't have to check those edits into your SCM ;-) But if I've made other hard requirements of packages like HardJSON, I'll have to update all those too. Yes, that's true, and why I recommeded what I did. That said, if you're paranoid enough to specify the exact versions (there's nothing wrong with this ;-) ) then it should be no surprise that you need to edit them... It's not surprising, it's just very annoying. more than 3 libraries involved. I now think it is best to only use version requirements to express known conflicts. Or likely sources of known conflicts, such as major version increases, which is why I suggested what I did above... You presume you can predict likely sources of known conflicts in software that doesn't exist yet. This is simply not true. For future versions of packages you can't really know if they will cause conflicts until they are released. Right, which is why consistency is version numbering for backwards incompatible changes is important. There is no single concept of what backward compatibility even is. You can off something that fixes my specific example, using knowledge that would not be available to you at the time when you were using the code. That doesn't really prove anything -- I could also come up with conflicts that would break any example you could provide. There's no version change so minor that it can't break anything, and there's no version change so major that you should end up with a cascading set of updates that only change dependency information just to accommodate it. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [zc.buildout] running in safe mode
Jim Fulton wrote:
Instead of using open(), etc, to write files, there's an instance of
Maker which holds some of the settings (--interactive, --simulate, a
base directory). Then you do all your file operations like:
maker.ensure_file('path/to/file.txt', content)
If that file exists with different content then the user gets asked
about what to do. It also logs all the writing, shows diffs, can make
backups, etc. You can force overwriting, but that's a keyword
argument that defaults to False, so only if you actually have good
reason to overwrite files (without asking) then that's fine, but you
will start developing the easy way, which is to be safe about this stuff.
In a system in which most data is managed automatically, asking the user
before doing anything that might remove or overwrite data is, in my
experience, counterproductive. It's like a security system that
constantly asks for permission do do things, training users to hit an
OK button very quickly.
In a previous version of buildout, it worked the way you and Tarek
suggest. It asked users before performing any action that caused a part
to be uninstalled. This was extremely annoying. I finally just started
piping the output of the yes command into it.
Again, I can live with people adding an option that causes buildout to
prompt before removing files or directories (or maybe just uninstalling
parts that would cause it to remove files or directories). I know that I
wouldn't use the option myself.
You can also treat that nuisance like a bug and resolve the problem. I
think fassembler has mostly done that, so that only really interesting
problems are noted. We also switched to a system inspired by gentoo
ports, where these queries are deferred until the end of the build.
Yes, it can be annoying, but it doesn't have to be annoying. I'd rather
start at a slightly annoying situation and try to decrease that problem,
than start with a periodically infuriating situation.
--
Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [zc.buildout] running in safe mode
Jim Fulton wrote: I know it is a bad practice for a recipe to return some paths that contains important data in the install() method, because zc.buildout might remove them. Nevertheless, it happens from time to time that a developer lose some content because of a misconfiguration, or a zealous recipe. That is his responsability, and backups are done for that. I don't think backups are the right approach. It's a mistake to have recipes manage precious data. If you really really really think that's a good idea, then the recipe should at least manage uninstall and move precious data aside, rather than remove it. I don't think it is really the user's problem is a recipe misbehaves by allowing precious data to be removed. I'll note fassembler uses a file abstraction layer so that its recipes are safe by default: https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py I think buildout would be a lot more humane if it took the same approach. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [zc.buildout] running in safe mode
Jim Fulton wrote:
On Oct 2, 2008, at 6:15 PM, Ian Bicking wrote:
Jim Fulton wrote:
I know it is a bad practice for a recipe to return some paths that
contains important data in the install() method,
because zc.buildout might remove them.
Nevertheless, it happens from time to time that a developer lose some
content because of a misconfiguration,
or a zealous recipe. That is his responsability, and backups are
done for that.
I don't think backups are the right approach. It's a mistake to have
recipes manage precious data. If you really really really think
that's a good idea, then the recipe should at least manage uninstall
and move precious data aside, rather than remove it.
I don't think it is really the user's problem is a recipe misbehaves
by allowing precious data to be removed.
I'll note fassembler uses a file abstraction layer so that its recipes
are safe by default:
https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py
I think buildout would be a lot more humane if it took the same approach.
I'd be interested to know what you mean by this, but I'm not willing to
read that source to find out.
Can you be a little more specific?
Instead of using open(), etc, to write files, there's an instance of
Maker which holds some of the settings (--interactive, --simulate, a
base directory). Then you do all your file operations like:
maker.ensure_file('path/to/file.txt', content)
If that file exists with different content then the user gets asked
about what to do. It also logs all the writing, shows diffs, can make
backups, etc. You can force overwriting, but that's a keyword argument
that defaults to False, so only if you actually have good reason to
overwrite files (without asking) then that's fine, but you will start
developing the easy way, which is to be safe about this stuff.
--
Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP for distutils
Gael Varoquaux wrote: On Tue, Sep 30, 2008 at 04:01:01PM +0100, Chris Withers wrote: That said, I didn't see any indication of what I consider to be a critical failure in PyPI: No dependency metadata prior to downloading the package. +1. I want to be able do list all the packages an easy_install run will download without running it. Something like the -s option of apt-get. In addition, I want this information to be available programmatically (ie with a good api, not something that expects to be called from the command line) to be able to use it to build dependency graphs, generate conflicts list, or simply tell me that I have requested something that is impossible. There is nothing that I hate more than easy_install failing after having half-installed a package because of a missing dependency. This is one of the reasons I am never too happy when I have to run easy_install. FWIW, pyinstall can collect all the packages before installing any of them. You do have to download all packages, though, as that's the only way to get the metadata. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Catalog-sig] PEP for distutils
A.M. Kuchling wrote: On Tue, Sep 30, 2008 at 10:41:11AM -0500, Ian Bicking wrote: FWIW, pyinstall can collect all the packages before installing any of them. You do have to download all packages, though, as that's the only way to get the metadata. Does the DOAP output for a package not contain enough metadata? No. It probably could hold that information, but currently PyPI doesn't keep any record of requirements, and so the DOAP file it generates doesn't indicate requirements either. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] just use debian
Chris Withers wrote: Tarek Ziadé wrote: Tarek Ziade wrote: For KGS I agree that this is a big work, but there's the need to work at a higher level that in your package Why? You really need to explain to me why the dependency information in each of the packages isn't enough? Because you can keep up with the dependencies changes, removed, or introduced by a package you depend on. Why can this not be expressed in the dependency information in the package? I tried this briefly for a while when Setuptools first came out, and I found it completely unmaintainable. Say I have a package that represents an application. We'll call it FooBlog. I release version 1.0. It uses the Turplango web framework (1.5 at the time of release) and the Storchalmy ORM (0.4), and Turplango uses HardJSON (1.2.1). I want my version 1.0 to keep working. So, I figure I'll add the dependencies: Turplango==1.5 Storchalmy==0.4 Then HardJSON 2.0 is released, and Turplango only required HardJSON=1.2, so new installations start installing HardJSON 2.0. But my app happens not to be compatible with that library, and so it's broken. OK... so, I could add HardJSON==1.2.1 in my requirements. But then a small bug fix, HardJSON 1.2.2 comes out, that fixes a security bug. Turplango releases version 1.5.1 that requires HardJSON=1.2.2. I now have have to update FooBlog to require both Turplango==1.5.1 and HardJSON==1.2.2. Later on, I decide that Turplango 1.6 fixes some important bugs, and I want to try it with my app. I can install Turplango 1.6, but I can't start my app because I'll get a version conflict. So to even experiment with a new version of the app, I have to check out FooBlog, update setup.py, reinstall (setup.py develop) the package, and then I can start using it. But if I've made other hard requirements of packages like HardJSON, I'll have to update all those too. So... that's the kind of thing I encountered with just a couple dependencies, but in practice it was much worse because there were a lot more than 3 libraries involved. I now think it is best to only use version requirements to express known conflicts. For future versions of packages you can't really know if they will cause conflicts until they are released. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] just use debian
Tarek Ziadé wrote: So... that's the kind of thing I encountered with just a couple dependencies, but in practice it was much worse because there were a lot more than 3 libraries involved. I now think it is best to only use version requirements to express known conflicts. For future versions of packages you can't really know if they will cause conflicts until they are released. Exactly, you can't control everything from your package unless you work in an isolated environement like virtualenv or zc.buildout provides, so I can't see any solution unless someone is taking care of it at a higher level :( maybe PyPI though, can automate this, when a package is uploaded, by browsing all dependency and finding relevant conflict ? PyPI knows all the packages out there. At least display those conflicts somehow ? or warn about them. Yes, keeping this version information separate from packages would help, I think. If you find out more information about a conflict it shouldn't require a new release -- new releases take a while to do, and have cascading effects. This kind of metadata isn't so much about the package, as about how the package relates to other packages. If we could somewhat safely have collaborative conflict information that would be nice, though there's different kinds of conflicts so it might be infeasible. It's all too common for a person to just poke around with version stuff until something works, but in a way that is only accurate for the context of their application, and if they submit that information upstream they could easily break other people's setups unnecessarily. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
Tres Seaver wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Greg Ewing wrote:
Guido van Rossum wrote:
Yeah, but *is* dropping backward compatibility an option here?
I'm not sure the concept of backward compatibility
makes sense here. The only kind of distutils replacement
I would be interested in would have a completely different
API, completely different structure and completely
different implementation. Anything less would fail to
fix the problems we want to fix.
Given that, what would it even *mean* for the new tool
to be backward compatible with distutils?
I think a tool which was willing to fake being distutils enough to
extract information from existing 'setup()' calls would probably be
enough, myself, so that:
$ python bbb_extractor_script.py setup.py
would create the equivalent of PKG_INFO / EGG_INFO on disk, where it
could then be used to drive the new installer.
pyinstall does exactly this, like:
python -c import setuptools; __file__='setup.py';
execfile('setup.py') egg_info
well, it's slightly more involved, because the Package.egg-info/
directory is written in the base of the package, which might be
./Package.egg-info, or ./src/Package.egg-info -- to avoid this I add
--egg-base=pyinstall-egg-info so that the egg info always goes in the
same location. Anyway, setuptools already fakes distutils in exactly
the way you describe.
--
Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Fwd: Re: Annoucing distribute project]
be candidates for the stdlib, in much the same way that wsgiref was - the spec is/should be stable, and those parts that are compiler/install-layout specific will need to be maintained in the stdlib anyway, for Python's own build infrastructure. In that sense, distutils 2 would not be so much a rewrite of the distutils, as the separation of them into tools for distributing, and tools for installing, where some of the tools for installing may be community-maintained. I agree. For instance, I don't see a particular advantage to making pyinstall part of the core of Python -- it would increase adoption and add a certain review process, but its usefulness is not contingent on everyone using it. It's more important that there's a consensus (or... movement towards some consensus) about how people write and distribute packages. If some distutils 2 effort just addressed that, and avoided things like installation that mostly can be avoided (though installation needs to be co-developed, of course), it should keep the scope down so that it's not quite as hard to agree on things. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python Package Management Sucks
Rick Warner wrote: Actually, PyPI is replicated. See, for example, http://download.zope.org/simple/. It may be that some of the mirrors should be better advertised. A half-hearted effort. at best, after the problems last year. When I configure a CPAN client (once per user) I create a list of replicas I want to search for any query from a list of hundreds of replicas distributed around the world. Can someone suggest the best way to search among repositories? For instance, try to connect to one, then stop if it gives Connection Refused? If it gives any unexpected error (5xx)? Timing out is a common failure, and a pain in the butt, but I guess there's that too. What does the CPAN client do? -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] pyinstall
I just announced a new installer I've been working on: http://pypi.python.org/pypi/pyinstall (also the not-very-thorough docs) Announcement post: http://www.openplans.org/projects/topp-engineering/blog/2008/09/24/pyinstall-a-new-hope/ Single-file executable for you to try: https://svn.openplans.org/svn/pyinstall/trunk/pyinstall.py I think it's good. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtualenv + python 2.6 (setuptools 2.6 egg)
Brian Clapper wrote: Ian, Been playing with Python 2.6rc2. Have you tried virtualenv with it? Doesn't seem to work for me, since it attempts to download a 2.6 version of the setuptools egg, which doesn't exist. I glanced through the virtualenv source (in between extreme busy-ness at work), and it looked to me as though virtualenv would use a setuptools egg in /path/to/virtualenv.egg/support-files if one existed, but that doesn't seem to happen. The EZ_SETUP_PY embedded script seems to be the culprit, but I haven't dug into it any deeper than that due to work demands. It's quite possible I'm just missing something stupid. If so, feel free to send me on my way. Yes, the missing egg is causing the problem. ez_setup.py doesn't seem to use a tarball fallback (I'm not sure why). Phillip - can there either be a 2.6 egg uploaded, or have ez_setup.py install the tarball when the egg is missing? -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Weird easy_install behavior when reinstalling
I'm getting some weird behavior for a particular package, installed from svn. To reproduce: virtualenv test-env ./test-env/bin/easy_install http://trac-hacks.org/svn/includemacro/0.11 a bunch of stuff is installed... then run: ./test-env/bin/easy_install http://trac-hacks.org/svn/includemacro/0.11 probably the first time you reinstall (and if not then, try a couple times) you get an exception like: $ ./T2/bin/easy_install http://trac-hacks.org/svn/includemacro/0.11 Downloading http://trac-hacks.org/svn/includemacro/0.11 Doing subversion checkout from http://trac-hacks.org/svn/includemacro/0.11 to /tmp/easy_install-SiGKUf/0.11 Processing 0.11 Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-SiGKUf/0.11/egg-dist-tmp-5rw1jx zip_safe flag not set; analyzing archive contents... TracIncludeMacro 2.1 is already the active version in easy-install.pth Installed /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/TracIncludeMacro-2.1-py2.4.egg Processing dependencies for TracIncludeMacro==2.1 Traceback (most recent call last): File ./T2/bin/easy_install, line 7, in ? sys.exit( File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/setuptools/command/easy_install.py, line 1671, in main with_ei_usage(lambda: File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/setuptools/command/easy_install.py, line 1659, in with_ei_usage return f() File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/setuptools/command/easy_install.py, line 1675, in lambda distclass=DistributionWithoutHelpCommands, **kw File /usr/lib/python2.4/distutils/core.py, line 149, in setup dist.run_commands() File /usr/lib/python2.4/distutils/dist.py, line 946, in run_commands self.run_command(cmd) File /usr/lib/python2.4/distutils/dist.py, line 966, in run_command cmd_obj.run() File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/setuptools/command/easy_install.py, line 211, in run self.easy_install(spec, not self.no_deps) File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/setuptools/command/easy_install.py, line 422, in easy_install return self.install_item(None, download, tmpdir, deps, True) File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/setuptools/command/easy_install.py, line 478, in install_item self.process_distribution(spec, dist, deps) File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/setuptools/command/easy_install.py, line 518, in process_distribution distros = WorkingSet([]).resolve( File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 529, in resolve requirements.extend(dist.requires(req.extras)[::-1]) File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 2107, in requires dm = self._dep_map File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 2099, in _dep_map for extra,reqs in split_sections(self._get_metadata(name)): File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 2518, in split_sections for line in yield_lines(s): File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 1813, in yield_lines for ss in strs: File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 2121, in _get_metadata for line in self.get_metadata_lines(name): File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 1140, in get_metadata_lines return yield_lines(self.get_metadata(name)) File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 1137, in get_metadata return self._get(self._fn(self.egg_info,name)) File /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/setuptools-0.6c8-py2.4.egg/pkg_resources.py, line 1195, in _get return self.loader.get_data(path) zipimport.ZipImportError: bad local file header in /home/ianb/src/PoachEggs/T2/lib/python2.4/site-packages/TracIncludeMacro-2.1-py2.4.egg I assume this is while it is trying to read TracIncludeMacro-xxx.egg/EGG-INFO/requires.txt. I've tried opening that egg with zipfile, and it works fine. This problem seems somewhat specific to TracIncludeMacro, though I can't figure out why. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [issue37] del __loader__ causes problems in appengine
New submission from Ian Bicking [EMAIL PROTECTED]: Per this thread, the placeholder .py file created by setuptools can cause problems when it deletes __loader__: http://markmail.org/message/qy2d6awkzcrr5t5s -- files: loader_bdist_egg.patch messages: 148 nosy: [EMAIL PROTECTED] priority: bug status: unread title: del __loader__ causes problems in appengine Added file: http://bugs.python.org/setuptools/file19/loader_bdist_egg.patch ___ Setuptools tracker [EMAIL PROTECTED] http://bugs.python.org/setuptools/issue37 ___Index: setuptools/command/bdist_egg.py === --- setuptools/command/bdist_egg.py (revision 66055) +++ setuptools/command/bdist_egg.py (working copy) @@ -29,7 +29,13 @@ import sys, pkg_resources, imp, __file__ = pkg_resources.resource_filename(__name__,%r) % resource, - del __bootstrap__, __loader__, + try:, + __loader__, + except NameError:, + pass, + else:, + del __loader__, + del __bootstrap__, imp.load_dynamic(__name__,__file__), __bootstrap__(), # terminal \n ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker needpatch keyword, tests, etc.
Phillip J. Eby wrote: At 10:32 AM 8/8/2008 -0500, Chris Galvan wrote: Phillip J. Eby wrote: Hello all. I had some spare time the other day and went through the tracker, reclassifying a few things and marking some with a needpatch keyword. The ones marked needpatch vary from things I have no idea what to do with, to ones where I've practically spelled out the needed patch in the tracker. Issues with patches that passed my initial review have been marked in-progress; these could use some testing before check-in. At this point, I haven't had an opportunity to review the results of the test sprint that was done; if somebody could throw that up as a patch on the tracker, or at least repost a link to where I can find that stuff, that would help. The work done on the test sprint is hosted in this bzr branch. We wanted to get your feedback on what had been done so far to make sure we were heading in the right direction. https://code.launchpad.net/~setuptools/setuptools-test/main My initial reaction is that it's off to a good start, but the tests themselves seem rather shallow; more like smoke tests (i.e., turn it on and see if smoke comes out) than functional tests. I'm thinking it might help to use the setuptools.sandbox facility to log files created, deleted, modified, etc. by the process. That would probably be a better test of what has/hasn't been done than using ellipses on the logs, which is order-dependent as well as having the ability to skip lines where the wrong thing is being done, etc. The way things are being done now, they probably won't be able to test some of the things that are most likely to break (i.e., the complexities of easy_install). (Probably in order to do that I'll need to add a new sandboxing class that creates a mock filesystem and allows before/after expectations to be set.) You could also use ScriptTest: http://pythonpaste.org/scripttest/ -- it doesn't make any attempt to mock anything out, but it does keep track of what a command does. For testing PoachEggs (https://svn.openplans.org/svn/PoachEggs/trunk/poacheggs-tests) I'm creating a scratch virtualenv for the test, then running things inside there. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [ANN] EggFreezer
Alberto Valverde wrote:
Ian Bicking wrote:
Alberto Valverde wrote:
The usage is pretty straightforward:
eggfreezer -o AllTurbogears2 -f
http://turbogears.org/2.0/downloads/current TurboGears2 tg.devtools
That command will try to satisfy all dependencies for TurboGears2 and
tg.devtools (fetching them from local packages if available), using that
url to find links, and bundle them into a file called
AllTurboGears2-${py_version}-${platform}.py.
As long as you are doing platforms, it might be nice to get them right.
Specifically the UCS2/UCS4 distinction, though there might be more
that I'm forgetting. (If there's actually platform-dependent files in
there, if not it'd be nice to leave out the platform entirely.)
I'm using pkg_resources.Distribution's 'platform' attribute to get this,
the algorithm basically iterates over all dependencies and as soon as
one has it set to not None it'll use that. If all are set to None then
no ${platform} is added to the filename. I'm assuming of course that all
distributions have the same string as platform, which I guess it isn't
not too far-fetched, but haven't really checked so I'm sure there's
something I might have overlooked.
BTW, does pkg_resources populate it properly with he UCS2/UCS4
distinction you mention?
No... which makes binary eggs unusable on Linux. I feel like there was
something else that made binary packages on a Mac unreliable, but I
can't remember. Windows binary eggs generally work fine. This is
discussed some here:
http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/
One nuisance is that people don't generally know how their Python was
built (UCS2 or UCS4). I was thinking about making something very
similar to eggfreezer (which I'm unlikely to do now that eggfreezer
exists ;), and generating an install .py file that determines the
platform and downloads the appropriate platform bundle.
--
Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [tg-trunk] Re: [ANN] EggFreezer
differences could occur, but in practice there's maybe 10 platforms instead of 3, and that's not unreasonable. Reading a comment on the philikon article (http://philikon.wordpress.com/2008/06/26/is-there-a-point-to-distributing-egg-files/#comment-47), I also notice that Enthought has done some work on this, it seems by fixing up the binary packages at install time. This seems to be related to an entirely different issue of the location of libraries and binary incompatibilities, which I only slightly understand. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [issue28] Error message is confusing if you accidentally try to instal a directory
New submission from Ian Bicking [EMAIL PROTECTED]: I commonly encounter a problem where a person creates a directory with a name, then tries to install a package with the same name as the directory. easy_install then things they were referring to the directory name. For instance: $ mkdir Pylons $ mkdir Pylons/src/MyProject # etc... $ easy_install Pylons Processing Pylons error: couldn't find a setup script in Pylons I think the error message could just be changed, first to make sure the directory ends with os.path.sep and then saying error: couldn't find a setup script in the directory Pylons/ -- with that error message most people would figure out the problem quickly. -- messages: 63 nosy: [EMAIL PROTECTED] priority: bug status: unread title: Error message is confusing if you accidentally try to instal a directory ___ Setuptools tracker [EMAIL PROTECTED] http://bugs.python.org/setuptools/issue28 ___ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [venv] Re: Changes News / 1.1 / * Added support for Python 2.6.
d2m wrote:
You could go on and add another 3 modules
('linecache','_abcoll','abc') to REQUIRED_MODULES just to find that
setuptools itself cannot be copied to my_env/lib/python2.6/site-
packages -- because there is no such distribution (2.6) right now:
Downloading
http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c8-py2.6.eggtry
run python2.6 ez_setup.py, gets you almost the same error.)
How did you install setuptools and virtualenv to make it functional?
*I* haven't, because I don't have Python 2.6 and haven't tried it with
that. I've added the other modules, but I'm not sure how to deal with
the issue of a lack of a 2.6 setuptools egg.
So, copying the distutils list: what's the best way to handle setuptools
installation on Python 2.6?
--
Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] There should be a no-index on simple pypi index
I just did a search for pyaudio and this link: http://pypi.python.org/simple/PyAudio/ showed up above this link: http://pypi.python.org/pypi/PyAudio/0.2.0 If meta name=robots content=noindex was added to the simple pages that would help. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtual env with exec prefix
Hans Meine wrote: Hi Ian, recently I reported this problem to you: [...] here at our university, python (as well as hundreds of other software packages) is installed in a shared NFS tree with a clear separation of --prefix and --exec-prefix, i.e. all platform-specific stuff goes into the according subdirectories. [...] [2] [EMAIL PROTECTED]:~ - virtualenv enthought-inst New python executable in enthought-inst/bin/python Installing setuptools Complete output from command enthought-inst/bin/python -c #!python \\\Bootstrap setuptoo... /software/python-2.4.4/SuSE-10...4.egg: error: invalid Python installation: unable to open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file or directory) [The message] is right, because it should be /software/python-2.4.4/SuSE-10.2/lib/python2.4/config/Makefile which exists (in the SuSE-10.2 directory). I have now debugged this a little bit (using virtualenv from SVN), and it seems to be a problem with the virtualenv'd python and setuptools. I have written the EZ_SETUP_PY to disk, and when I mimick virtualenv's command for installing setuptools, I get the error: [2] [EMAIL PROTECTED]:~/Programming/enthought - inst/bin/python ../ez_setup.py -v /home/meine/Programming/virtualenv-svn/support-files/setuptools-0.6c8-py2.4.egg error: invalid Python installation: unable to open /software/python-2.4.4/lib/python2.4/config/Makefile (No such file or directory) (NB: inst is the virtual env) OTOH, when I run this with the python binary from its original location, it works fine. Since the error comes from deep within setuptools (AFAICS), should this be discussed on distutils-sig? Hmm... well, does this error only happen when you create a virtualenv from within a virtualenv? Otherwise, I think there's several things in distutils.sysconfig that need to be adjusted. That module isn't really documented, however, and I suspect there's just a bunch of things that will have to be added to over time, not a good general fix. Or... maybe there's some clever way to fix all these, but I'm not sure. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Making setuptools play nicer with virtualenv
Gael Varoquaux wrote: I have just out-clevered myself using setuptools and virtualenv: * install foo using python setup.py develop (foo being ipython). * download some module bar you want to work on in an isolated environment * create this isolated environment using virtualenv bar * in the isolated environment python setup.py develop the bar module. * still in the isolated environment, try to import bar in a script installed by foo (aka ipython) -- fails because foo uses the system python, and virtualenv wants you to use its own python One very easy solution to make this work is to have the setuptools generated scripts use, under unices, #!/usr/bin/env python rather than #!/usr/bin/python. This seems to me like a good solution, in general, to follow the user's expectations. Is this a change that would be possible? Sometimes you want to inherit the environment you've activated, but in my experience usually this isn't what you'll want. I find it easier to just reinstall any tools (like ipython, nose, etc) that I want to use in the virtualenv. In an ideal situation they could share eggs with the system packages, but this only kind of works. (Sometimes, for reasons I don't always understand, easy_install will find and install globally-installed packages, creating an executable bound to the virtualenv.) Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Putting egg-info under version control
Martin Aspeli wrote: That was my impression too. If that's a safe assumption to make, then we can continue to say that *.egg-info goes in svn:ignores. I can't figure out from http://pythonpaste.org/script/developer.html how the files are supposed to be created though. Wiping egg-info and re-running buildout certainly doesn't re-create it. paster create normally adds the .egg-info directory, but ignores all the Setuptools-created files, and some Paste packages (including paster create) add files in that directory. I won't say that's a good idea, and I probably wouldn't go in this direction if starting from scratch... putting just one or two files under version control, but ignoring the rest, is quite awkward. I usually set svn:ignore in the egg-info directory to *, and then you can still opt-in specific files if you want to (or I leave egg-info out of version control entirely). Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] alternatives to zc.buildout?
zooko wrote: On Feb 12, 2008, at 9:35 PM, Ian Bicking wrote: I don't know that many people have used this (or know about it), but I added a command to buildutils (http://knowledgetap.com/hg/buildutils/) to do python setup.py bundle, which takes a package and all its dependencies and puts them together, with a script that adds all the dependencies. It's like what zc.buildout does mechanically, but intended to be used more like py2exe. I think it'd fit the model of managing Python commands and scripts pretty well. That's interesting. What we do is have our setup.py look in a directory named misc/dependencies/ and add any tarballs it finds therein to the dependency_links: http://allmydata.org/trac/tahoe/browser/setup.py#L93 That's similar to bundle; bundle just calls easy_install to install all the dependencies into a particular directory, then adds site.addsitedir(dependency_dir) to the top of any scripts. But as a result you don't have to call setup.py on the host. Figuring out the location of dependency_dir is less than perfect. Both relative and absolute filenames have their problems. Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] alternatives to zc.buildout?
Tres Seaver wrote: My major beef with zc.buildout is perhaps actually a problem with the recipes nearly everybody uses: they blow away hand-edited stuff without warning. In particular, changes to things like the zope.conf file get zapped, because buildout (or the recipe) thinks that the file is its own personal property. This was also something that drove me nuts. It's too bad this is still the case. We have a build tool ourselves, very similar in scope to zc.buildout, though I don't really have intentions at this time to make it a legitimate project for other people to use. But, for the record: http://www.openplans.org/projects/fassembler/ One of the core parts of it is filemaker, which does most of the interaction with the system: https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py It goes to great length to notice changes, even if they aren't due to edits; I find it's nice to know what exactly is going on. It's also got a bit of support for detecting why things changed, by saving the template, and probably will grow support for detecting user changes so some things can be overwritten without warning. With these build things I don't really care who owns what, since that's mostly an abstract concept that the build user won't know and shouldn't really have to know. (filemaker was generally based on the code from paster create) I also pay a lot of attention to logging, as I hate noisy output and of course not enough output is also a problem. I can't remember how zc.buildout acts in that respect. Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Setuptools scripts on x64 Windows
We got a report of problems with Setuptools-generated scripts on x64 Windows XP. I'm pretty sure the problem is in Setuptools: http://pylonshq.com/project/pylonshq/ticket/370 (Submitter is copied) Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtualenv, OS X, and GUIs
Robert Kern wrote: Is there an appropriate mailing list for talking about virtualenv? I've run into a problem where virtualenvs on OS X cannot access the screen to run GUIs. I've found a solution that appears to work and am going to write a patch to let virtualenv do it. I'd like to get some feedback from other OS X virtualenv users, though. I've thought maybe the distutils list is the best place for discussion; at least for the moment (better than private email, where no one else will see it). If activity increases much perhaps a separate mailing list will make sense (not everyone who would want to discuss virtualenv will want to be on the distutils list). I think Ronald Oussoren is on that list, but just in case I'll copy him since he's done the Mac specific stuff in virtualenv so far. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] virtualenv group
I've created a Google Group for virtualenv discussion: http://groups.google.com/group/python-virtualenv/ I also set up a bug tracker on Launchpad: https://bugs.launchpad.net/virtualenv/ -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] easy_install -f confusion
Jeff Hammel and myself were going over a problem with -f seemingly being ignored in easy_install, and it took us a while to realize that you have to do something like easy_install -f 'url1 url2 ...', and you can't do easy_install -f url1 -f url2. It's literally been years that I've been unaware of this, and confused by the results. I think easy_install should give an error if you provide multiple -f's (or make it work if you do so). -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] easy_install prefering filename over #egg
We've been building some custom eggs for the lxml trunk and we uploaded the packages to our wiki, which replaced the .'s with -'s. This messed up the filenames, so we added #egg in an effort to correct that: http://www.openplans.org/projects/opencore/dependencies/lxml-2-0alpha5-py2-4-linux-x86_64.egg#egg=lxml-2.0alpha5-py2.4-linux-x86_64 But it seems that easy_install prefers the base name, not the fragment, and so it parsed this as an lxml 2 egg, with 0alpha5blahblahblah as the platform(?)... I'm not sure exactly how it parsed it, but it got it wrong. It seems like easy_install should prefer fragments over the filename itself, as the fragments will only be there if someone deliberately adds them. Ian ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] easy_install prefering filename over #egg
Ian Bicking wrote: We've been building some custom eggs for the lxml trunk and we uploaded the packages to our wiki, which replaced the .'s with -'s. This messed up the filenames, so we added #egg in an effort to correct that: http://www.openplans.org/projects/opencore/dependencies/lxml-2-0alpha5-py2-4-linux-x86_64.egg#egg=lxml-2.0alpha5-py2.4-linux-x86_64 But it seems that easy_install prefers the base name, not the fragment, and so it parsed this as an lxml 2 egg, with 0alpha5blahblahblah as the platform(?)... I'm not sure exactly how it parsed it, but it got it wrong. It seems like easy_install should prefer fragments over the filename itself, as the fragments will only be there if someone deliberately adds them. Actually, it just occurred to me that there's a link on the page with the #egg, and one without, and probably easy_install was just looking at the link without the fragment. So maybe our plan to use #egg to correct the link just wasn't going to work with the site. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Object loading spec
I mentioned some time ago that it would be nice if everyone could agree how to configure object references. Mostly I just had in mind having Paste Deploy and zc.buildout use the same syntax, but it's something that comes up in ad hoc situations often as well. I wrote up a small implementation to push this idea forward a bit. I'm hoping to get some feedback here. In an unfortunate mishmash of syllables (name suggestions welcome) I named it obconfloader. You can read the description here: http://svn.pythonpaste.org/ObConfLoader/trunk/docs/index.txt I've also copied that description into this email. Status License ObConfLoader is under an `MIT-style permissive license http://svn.pythonpaste.org/ObConfLoader/trunk/docs/license.txt`_. Discussion should occur on `distutils-sig http://www.python.org/community/sigs/current/distutils-sig/`_; please be sure to put ObConfLoader in the subject line. The package is available in a `subversion repository http://svn.pythonpaste.org/ObConfLoader/trunk#egg=ObConfLoader`_, and the trunk can be installed with ``easy_install ObConfLoader==dev``. You can get a checkout with:: svn co http://svn.pythonpaste.org/ObConfLoader/trunk ObConfLoader Introduction ObConfLoader allows you to load objects from strings, typically for use in config files (or from the command line, or other locations where object references are necessary but the format is constrained). It also allows you to utilize `Setuptools entry points http://peak.telecommunity.com/DevCenter/setuptools#entry-points`_ to make public objects that can be referenced. The concept of an entry point group, or API, allows backward compatibility when using entry points. The basic usage is very simple:: from obconfloader import load_ob obj, ep_group = load_ob(a_string, ['mypackage.api']) assert ep_group is None or ep_group == 'mypackage.api' This loads the object from the given string, returning the object and the API it supports. The second argument is a list of entry point groups that you support. You can support multiple entry point groups, or no groups (by leaving out the second argument, or using None). If you leave out the second argument ``ep_group`` will always be None. There are several possible exceptions: ``LoadError``: The parent of all other exceptions; a generic error. ``ConfigSyntaxError``: The string itself is malformed. ``BadGroupError``: The group given doesn't match up with an acceptable input group. ``NotFoundError``: The referenced object can't be found (something like DistributionNotFound, ImportError, AttributeError). ``PythonError``: Some error with a Python expression, Python syntax, etc. Specification Format There are several formats for strings: ``file /path/to/file.py`` This loads the given file, and returns the module object. ``file /path/to/file.py:dotted.name`` This loads the given file, and returns the object ``dotted.name`` inside it. ``file /path/to/file.py:dotted.name [entry.point.group]`` This loads the object, and indicates that the object supports the group ``entry.point.group``. This can be used when multiple APIs are supported, and the object doesn't use the default API. ``python module.name:dotted.name`` This loads the module ``module.name``, and returns the ``dotted.name`` object. Of course you can leave out ``dotted.name`` to return the module, and also add an entry point group. ``egg Distribution`` This loads the distribution (the package) ``Distribution``, and gets the ``main`` entry point, with the given entry point group. ``egg Distribution:entry_point_name`` Instead of the ``main`` entry point, this gets ``entry_point_name``. Other Schemes - You can pass in a dictionary of extra schemes (besides ``file``, ``python``, and ``egg``) with the ``extra_schemes`` argument. All schemes are converted to lower case. The signature of a loader is:: def scheme_loader(string, groups, orig_string, position, group): if problem: raise LoadError(message, orig_string, position) return loaded_obj, group The meaning of the arguments: ``string``: The string you should load, like ``/path/to/file.py:dotted.name``. This leaves out the trailing group and the scheme. ``groups``: The entry point groups accepted. Schemes besides ``egg`` probably will ignore this. ``orig_string``: The full string, with scheme. Used primarily with any ``LoadError`` exceptions, to give the user back the full string they entered. ``position``: The position where the string was found; also used with exceptions. ``group``: If the string contained an explicit group, this will be that group (otherwise None). If you don't have any explicit notion of groups then you can just do the return as in the
Re: [Distutils] Enabling local distutils.cfg usage
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ian Bicking wrote: Tres Seaver wrote: When building a virtual environment, I'd like to be able to store global distutils configuration options which are custom to that environment. However, both setuptools and distutils expect to read / write that file relative to the directory of 'distutils.__file__', which is located in the *source* environment under virtualenv 0.8.4:: This should be working in virtualenv trunk; I had to switch up the way sys.path was constructed some, because the way it was the system path came before the virtualenv path, so a virtual distutils wouldn't be seen. Cool. Where is the trunk SVN, by the way? At http://svn.colorstudy.com/virtualenv/trunk -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Enabling local distutils.cfg usage
Tres Seaver wrote: When building a virtual environment, I'd like to be able to store global distutils configuration options which are custom to that environment. However, both setuptools and distutils expect to read / write that file relative to the directory of 'distutils.__file__', which is located in the *source* environment under virtualenv 0.8.4:: This should be working in virtualenv trunk; I had to switch up the way sys.path was constructed some, because the way it was the system path came before the virtualenv path, so a virtual distutils wouldn't be seen. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtual-python.py, sys.prefix, and Mac
Ronald Oussoren wrote: On Wednesday, September 19, 2007, at 07:26AM, Ian Bicking [EMAIL PROTECTED] wrote: It is in the python.org source tree: Modules/getpath.c That's the file used in the Framework build of Python? I only see some small references to __APPLE__, none of which seem related to Frameworks. Also, if you build Python from source it works fine -- it's only the python that Apple ships that has the problem. They must use some patched version of this file that they use...? Start reading at line 694 in Modules/getpath.c, that's in the head of the release25-maint branch. It's only 695 lines long...? At line 458 it seems like it is looking for lib/python2.5/os.py, in some Framework-specific code. It will look relative to the binary, but only after looking in the Framework first. Simply breaking the Framework lookup somehow would do it. Here's the code to find the Framework locations: pythonModule = NSModuleForSymbol(NSLookupAndBindSymbol(_Py_Initialize)); /* Use dylib functions to find out where the framework was loaded from */ buf = (char *)NSLibraryNameForModule(pythonModule); I'm not sure how to do anything to that though. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Windows Python sys.prefix
Related to the issue I noted with sys.prefix on a Mac, moving the Python executable on Windows also doesn't seem to effect sys.prefix. How does Python on Windows determine sys.prefix? Is there a way to effect that? -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Windows Python sys.prefix
Mark Hammond wrote: How does Python on Windows determine sys.prefix? By consulting the registry. The main use-case for that is when Python is embedded in another executable - often, but not always, via COM. When Python is hosted inside excel.exe, for example, its impossible to calculate sys.path or sys.prefix based on that executable. In that same example, Python's DLL will have been loaded from a directory on the global PATH, which generally means the \Windows\System32 directory - so that too is no help in calculating the path. [IIRC, Python actually tries to use the executable to locate its 'landmark' and only falls back to the registry when that fails - but that sounds exactly like what is happening in your example. The gory details are in PC/getpathp.c] I can certainly create a landmark if at all reasonable. I'm looking at getpathp.c, and it looks like it just looks for lib/os.py ... and I think it walks up from the current directory until it finds it? So right now I have: bin/python.exe lib/python2.5/os.py But perhaps if I just change it to: bin/python.exe lib/os.py it will work? On windows, does it just leave out the /python2.5/ portion of the lib path? Is there a way to effect that? Setting PYTHONHOME in the environment is the only way I'm aware of. That's kind of what workingenv did, which I'm trying to avoid -- you have to worry about, for instance, calling another Python subprocess if you don't want that process to be in the same environment. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Windows Python sys.prefix
After that and a bunch of other little updates, virtualenv (0.8.3) now works on Windows! Now it's just the blasted Mac Framework stuff that stops this from being a complete workingenv replacement. Mark Hammond wrote: I can certainly create a landmark if at all reasonable. I'm looking at getpathp.c, and it looks like it just looks for lib/os.py ... and I think it walks up from the current directory until it finds it? So right now I have: bin/python.exe lib/python2.5/os.py But perhaps if I just change it to: bin/python.exe lib/os.py it will work? Yep, and a quick test here works fine. On windows, does it just leave out the /python2.5/ portion of the lib path? well - it doesn't leave it out as such - it just never adds anything like that :) Mark -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtual-python.py, sys.prefix, and Mac
Ronald Oussoren wrote: On 16 Sep, 2007, at 21:55, Ian Bicking wrote: Ronald Oussoren wrote: On 16 Sep, 2007, at 21:44, Ian Bicking wrote: Ronald Oussoren wrote: On 15 Sep, 2007, at 18:09, Ian Bicking wrote: Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http://pypi.python.org/pypi/virtualenv/). So the basic technique here is to copy the executable into /ENV/bin/python, and then sys.prefix will be '/ENV'. The standard Python installed on a Mac doesn't seem to do this -- the prefix remains '/opt/local/Library/Frameworks/Python.framework/Versions/2.4' regardless. (Custom built Python's on Mac work like normal.) All framework builds behave as you describe, Modules/getpath.c special-cases calculation of sys.prefix for framework builds of Python (the prefix is inside the framework regardless of where the executable is). Is there any way to effect that calculation? I.e., in a normal build that calculation is based on the location of the executable, so virtualenv moves the executable to effect that. Move the framework. I don't really know what that means...? What exactly is the framework? The python framework, that is /Library/Frameworks/Python.framework (or /System/... if you use Apple's build of Python). getpath.c uses some API calls to determine the absolute path of the python framework linked into the current executable and sets sys.prefix to a directory inside that framework. Do you have a reference to the getpath.c that it uses? Windows seems to have something kind of hardcoded, but also a detection scheme, and maybe similarly on Mac there's something I can do to avoid the hardcoded portion. As background info for anyone that's not used to the mac way: OSX supports the usual unix organisation of shared libraries but also has a different method: frameworks. A framework is basicly a directory containing the shared library, header files and resources (the last two are optional) and also supports versioning, although Apple's development tools don't offer full support for that. I should be possible to coax a framework install into support virtual-python by creating a directory structure for a python.framework inside the virtual-python environment and using a simular mechanism as you have now for adding the real stdlib to sys.path. You will have to modify the copied python executable to load this mini-framework because the OSX linker adds absolute paths to shared libraries and frameworks to the executable. The macholib library can be used to do this last task, it is used by py2app for rewriting the linker commands in shared libraries that are used in application bundles. I don't have an example for that handy, but it should be easy enough to extract code from macho_standalone. I noticed in p2app it has a file main.c in it, which I think is the Python interpreter code... maybe it recompiles the interpreter? -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtual-python.py, sys.prefix, and Mac
Ronald Oussoren wrote: On 19 Sep, 2007, at 6:25, Ian Bicking wrote: Ronald Oussoren wrote: On 16 Sep, 2007, at 21:55, Ian Bicking wrote: Ronald Oussoren wrote: On 16 Sep, 2007, at 21:44, Ian Bicking wrote: Ronald Oussoren wrote: On 15 Sep, 2007, at 18:09, Ian Bicking wrote: Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http://pypi.python.org/pypi/virtualenv/). So the basic technique here is to copy the executable into /ENV/bin/python, and then sys.prefix will be '/ENV'. The standard Python installed on a Mac doesn't seem to do this -- the prefix remains '/opt/local/Library/Frameworks/Python.framework/Versions/2.4' regardless. (Custom built Python's on Mac work like normal.) All framework builds behave as you describe, Modules/getpath.c special-cases calculation of sys.prefix for framework builds of Python (the prefix is inside the framework regardless of where the executable is). Is there any way to effect that calculation? I.e., in a normal build that calculation is based on the location of the executable, so virtualenv moves the executable to effect that. Move the framework. I don't really know what that means...? What exactly is the framework? The python framework, that is /Library/Frameworks/Python.framework (or /System/... if you use Apple's build of Python). getpath.c uses some API calls to determine the absolute path of the python framework linked into the current executable and sets sys.prefix to a directory inside that framework. Do you have a reference to the getpath.c that it uses? Windows seems to have something kind of hardcoded, but also a detection scheme, and maybe similarly on Mac there's something I can do to avoid the hardcoded portion. It is in the python.org source tree: Modules/getpath.c That's the file used in the Framework build of Python? I only see some small references to __APPLE__, none of which seem related to Frameworks. Also, if you build Python from source it works fine -- it's only the python that Apple ships that has the problem. They must use some patched version of this file that they use...? -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] virtual-python.py, sys.prefix, and Mac
Andreas Jung wrote: --On 15. September 2007 11:09:44 -0500 Ian Bicking [EMAIL PROTECTED] wrote: Hi all. I'm kind of giving up on workingenv, and have started working from virtual-python as a basis instead (http://pypi.python.org/pypi/virtualenv/). I tried virtualenv - however the python in the new environment contains all eggs in sys.path that are installed for the original Python interpreter. Bug or feature? You have to use --no-site-packages to avoid that. -- Ian Bicking : [EMAIL PROTECTED] : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
