Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On 20 Apr, 2006, at 22:07, Martin v. Löwis wrote: Guido van Rossum wrote: This is another area where API standardization is important; as soon as modules are loaded out of a zip file, the traditional approach of looking relative to os.path.dirname(__file__) no longer works. 2. standardize on file names, not API. If I want to deploy read-only data files, I put them into /usr/share. If I need read-write files, I put them into /var. I did not have such a problem yet on other systems, but I would also try to follow the conventions of these systems. I don't think anyone mentioned this, but part of pkg_resources is an API for loading package-related data. That currently looks for data near the code, which makes Linux packagers unhappy but it should be possible to enhance pkg_resources to look in other locations (such as a directory below /usr/share/python) as well. The data loading API's is one feature of pkg_resources that would make it easier to locate data while at the same time storing data in locations that the various platforms proscribe without hardcoding such knowledge into every program. BTW. Please keep in mind that one of the major platforms for python doesn't have a proper package manager at all. OSX does have an installer application, but that's just that: a GUI for dropping files on you disk. There is no uninstaller, and no real dependency management. Ronald smime.p7s Description: S/MIME cryptographic signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On 4/23/06, Phillip J. Eby [EMAIL PROTECTED] wrote: At 01:19 PM 4/23/2006 +1000, Anthony Baxter wrote: On Sunday 23 April 2006 11:43, Nick Coghlan wrote: Maybe we need something that's the equivalent of alien (rpm - dpkg converter), so that given an egg, one can easily get a native installer for that egg. An 'eggconvert' that used the existing bdist_foo machinery to build system specific packages would probably be an ok summer of code project, no? That's probably not going to be the best way to get from an egg to a system package, since a lot of the bdist_foo commands try to build things from source, and an egg for the specific platform is already going to be built, and won't include source (except for Python modules). Yes. And on Windows at least, building from source can be problematic. Given a binary egg with all the bits included, it's *far* better to just move them around and build an installer from the pieces. It might be possible to unpack the egg into the structure of a distutils build directory, and then use bdist_xxx to build the installer on that (on the basis that the build steps will be ignored as the objects are already there - you may have to fiddle a bit to fool distutils into *not* needing source present to check timestamps against...) Probably you'd want to create something more like Vincenzo Di Massa's easy_deb program, which uses easy_install to find and fetch a source distribution, then builds a .deb from it. You can currently use: That's certainly not what I want, in two ways: 1. Finding and fetching things isn't needed - I'm happy starting with an egg file on the local filesystem 2. Building from source is not always possible, as I explained above. So, for any bdist_foo command that's already implemented in the distutils, you can already get pretty close to this functionality by using a short shell script that just calls easy_install followed by the setup.py. ... assuming that setup.py build works for you. That's not the use case I am thinking of (which is a 3rd party grabbing a binary egg they couldn't build for themselves, and converting it to the local packaging system). What you won't get without writing some more code is dependency support. While I accept that dependency support is very valuable to a lot of people, some of us (the control freaks, maybe :-)) prefer to install dependencies by hand, based on documented requirements. (There's a separate rant here, about the possibility that people stop documenting requirements because setup.py has them, and they get handled by easy_install, but I'll save that for another day...) But if you're thinking of converting egg dependency data to, for example, deb dependency metadata, then yes, that's an issue. For Windows installers the conversion is easy, of course - just delete the metadata :-) Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On 20-apr-2006, at 23:46, Martin v. Löwis wrote: Bob Ippolito wrote: 'There are several binary formats that embody eggs, but the most common is '.egg' zipfile format, because it's a convenient one for distributing projects.' '.egg files are a zero installation format for a Python package;' single modules are also such a zero installation format too. So what? You're simply reading things between the lines that aren't there. How about you describe exactly what parts of the documentation that lead you to believe that eggs are designed to compete with solutions like rpm/msi/deb so that it can be clarified? It's not just the documentation: I firmly believe that many people consider .egg files to be a distribution and package management format. People have commented that some systems (e.g. OSX) doesn't have a usable native packager, so setuptools fills a need here. This shows that people do believe that .egg files are to OSX what .deb files are to Debian. As .egg files work on Debian, too, it is natural that they compete with .deb. Phillip Eby once said (IIRC) that he doesn't want package authors to learn all the different bdist_* commands (which also require access to the target systems sometimes), and that they their life gets easier as they now only have to ship the native Python binary packages, namely .egg files. In this view, rpm/msi/deb have no place anymore, and are obsolete. In the view of at least some Linux packagers nobody but they should create system packages anyway. Personally I think that view is misguided, but the view is there. I can readily believe that package authors indeed see this as a simplification, but I also see an increased burden on system admins in return. So if this attitude (Python Eggs are the preferred binary distribution format) is wrong, it is the attitude that has to change first. Changes to the documentation follow from that. If the attitude is right, I'll have to accept that I have a minority opinion. IMHO python eggs are the preferred distribution format for several use cases, but not all. They are very usefull for systems that lack a proper package manager of their own and for managing a developers sandbox. As a sysadminI'd be a lot less inclined to use eggs to install software on a system with a proper package manager (like most linux distributions) because the eggs will then not be visible in the global view of installed software or play nice with vendor software management tools. Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Ronald Oussoren wrote: On 20-apr-2006, at 23:46, Martin v. Löwis wrote: So if this attitude (Python Eggs are the preferred binary distribution format) is wrong, it is the attitude that has to change first. Changes to the documentation follow from that. If the attitude is right, I'll have to accept that I have a minority opinion. IMHO python eggs are the preferred distribution format for several use cases, but not all. They are very usefull for systems that lack a proper package manager of their own and for managing a developers sandbox. As a sysadminI'd be a lot less inclined to use eggs to install software on a system with a proper package manager (like most linux distributions) because the eggs will then not be visible in the global view of installed software or play nice with vendor software management tools. Maybe we need something that's the equivalent of alien (rpm - dpkg converter), so that given an egg, one can easily get a native installer for that egg. Regards, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On Sunday 23 April 2006 11:43, Nick Coghlan wrote: Maybe we need something that's the equivalent of alien (rpm - dpkg converter), so that given an egg, one can easily get a native installer for that egg. An 'eggconvert' that used the existing bdist_foo machinery to build system specific packages would probably be an ok summer of code project, no? -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
At 01:19 PM 4/23/2006 +1000, Anthony Baxter wrote: On Sunday 23 April 2006 11:43, Nick Coghlan wrote: Maybe we need something that's the equivalent of alien (rpm - dpkg converter), so that given an egg, one can easily get a native installer for that egg. An 'eggconvert' that used the existing bdist_foo machinery to build system specific packages would probably be an ok summer of code project, no? That's probably not going to be the best way to get from an egg to a system package, since a lot of the bdist_foo commands try to build things from source, and an egg for the specific platform is already going to be built, and won't include source (except for Python modules). Probably you'd want to create something more like Vincenzo Di Massa's easy_deb program, which uses easy_install to find and fetch a source distribution, then builds a .deb from it. You can currently use: easy_install --editable --build_directory=somewhere SomePackage And it will find SomePackage, and unpack a source distribution into 'somewhere/somepackage'. So you could then change to that directory and run the package's setup.py with any bdist command you wanted to. So, for any bdist_foo command that's already implemented in the distutils, you can already get pretty close to this functionality by using a short shell script that just calls easy_install followed by the setup.py. What you won't get without writing some more code is dependency support. You also have to deal with the issue of mapping PyPI names to the names used by the relevant packaging system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Anthony Baxter wrote: On Sunday 23 April 2006 11:43, Nick Coghlan wrote: Maybe we need something that's the equivalent of alien (rpm - dpkg converter), so that given an egg, one can easily get a native installer for that egg. An 'eggconvert' that used the existing bdist_foo machinery to build system specific packages would probably be an ok summer of code project, no? It sounds like an excellent one if you can find the right mentor. . . Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Martin v. Löwis wrote: Some libraries (not necessarily in Python) have gone the path of providing a unified API for all kinds of file stream access, e.g. in KDE, any tool can open files over many protocols, without the storage being mounted locally. Maybe a compromise would be to provide an extended version of open() that knew about zip files. This could be a stand-alone facility to complement zip importing, and needn't be tied to distutils or setuptools or anything like that. Code which finds things using module __file__ attributes would work exactly the way it does now as long as it used the special open(). -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Guido van Rossum wrote: You can't blame KDE for providing mechanisms that only work in the KDE world. It's fine for Python to provide Python-specific solutions for issues that have no cross-platform native solution. Also keep in mind that we're talking about resources used internally by the application. They don't *need* to be accessible outside the application. So I don't think the KDE argument applies here. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Martin v. Löwis wrote: Greg Ewing wrote: The resources name is actually quite a common meme; I believe it goes back to the original Macintosh, I can believe that history. Still, I thought a resource is something you can exhaust; Haven't you heard the term renewable resource ?-) In the real world, yes, most resources will eventually become exhausted, but I don't think it's a necessary part of the meaning. It's just something that you exploit, or make use of. BTW, in the game programming industry the in-vogue term at the moment seems to be asset, which has even more inappropriate connotations. Or perhaps not, if you're a commercial entity that attaches a dollar value to all your intellectual property... the fork should have been named data fork Except that's what they call the *other* fork (equivalent to the only fork on systems that don't have forks). or just second fork. But then the relevant Toolbox module would have to have been called the Second Fork Manager, which sounds like an API for use by dining philosophers. :-) FWIW, Apple seem to be deprecating the use of resource forks these days, and the Resource Manager, which is a bit sad. It was *fun* programming the Mac back when it was quirky as hell and like nothing else on the planet. Frustrating at times, but fun! -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Martin v. Löwis wrote: I can readily believe that package authors indeed see this as a simplification, but I also see an increased burden on system admins in return. There are two conflicting desires here. Package authors don't want to have to make M different kinds of package for M different systems. System administrators don't want to have to deal with N different kinds of package originating from N different software development environments. What can be done about this? Maybe some kind of babel server in the middle, which knows how to convert between all the different packaging formats. Authors submit packages in the form which is easiest for them to produce, and users request packages in the form which is easiest for them to install. The server does whatever is necessary to convert between them. Perhaps something along these lines could be incorporated into the cheeseshop? So you could ask e.g. for your favourite package in .rpm format, and if it only knew how to get it in .egg format it would feed it to a handy bdist-bot for conversion. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Phillip J. Eby wrote: By now, however, the term is popularly used with GUI toolkits of all kinds to mean essentially read-only data files that are required by a program or library to function, but which are not directly part of the code. It's just occurred to me that there's another thread to the history of this term: the X window system's use of the term resource to mean essentially a user-configurable setting. This is probably where the usage you're referring to comes from, since Gtk and the ilk are fairly deeply rooted in X. This is really quite a different idea, although there's some overlap because X applications are typically distributed with a resources file containing default values for the settings, and won't work without there being some version of this file available. However, the sense we're talking about is more like the Mac one, where resources are an integral part of the program just as much as the code is, and have every right to be kept with the code. Indeed, in a classic 68K Mac application, the code literally WAS a resource. The 68K machine code was kept in the resource fork in resources of type CODE. The data fork of an application was usually empty. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Ronald Oussoren wrote: That doesn't require Eggs to be single-file zipfiles; deleting a directory would work just as will (and I believe it will work with ez_install, too). Egg directories (which are basically just the same as packages using .pth files) also work for this and that's what I usually install. Setuptools can work with python extension inside .zip files, but I'm not entirely comfortable with that. It's primarily the .egg *files* that I dislike. I'm can accept the .egg directories. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Greg Ewing wrote: Guido van Rossum wrote: You can't blame KDE for providing mechanisms that only work in the KDE world. It's fine for Python to provide Python-specific solutions for issues that have no cross-platform native solution. Also keep in mind that we're talking about resources used internally by the application. They don't *need* to be accessible outside the application. So I don't think the KDE argument applies here. They might need to be available outside Python. Two use cases: 1. The application embeds an sqlite database. Even though it knows how to get at the data, it can't use it, because the sqlite3 library won't accept .../foo-1.0.egg/resources/the_data (or some such) as a database name, if foo-1.0.egg is a ZIP file. If the installed application was a set of files, that would work. 2. The application embeds an SGML DTD (say, HTML). In order to perform validation, it invokes nsgmls on the host system. It cannot pass the SGML catalog to nsgmls (using the -C option) since you can't refer to DTD files inside ZIP files inside an SGML catalog. If this was a bunch of installed files, it would work. 3. The application includes an SSL certificate. It can't pass it to socket.ssl, since OpenSSL expects a host system file name, not a fragment of a zip file. If this was installed as files, it would work. This is precisely what happens in KDE: you have konqueror happily browse an SMB directory, double-click on a .xls file, OpenOffice starts and can't access the file it was started with. It doesn't matter to the user that there is a good reason for that. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
At 06:43 PM 4/21/2006 +0200, Martin v. Löwis wrote: They might need to be available outside Python. Two use cases: 1. The application embeds an sqlite database. Even though it knows how to get at the data, it can't use it, because the sqlite3 library won't accept .../foo-1.0.egg/resources/the_data (or some such) as a database name, if foo-1.0.egg is a ZIP file. If the installed application was a set of files, that would work. 2. The application embeds an SGML DTD (say, HTML). In order to perform validation, it invokes nsgmls on the host system. It cannot pass the SGML catalog to nsgmls (using the -C option) since you can't refer to DTD files inside ZIP files inside an SGML catalog. If this was a bunch of installed files, it would work. 3. The application includes an SSL certificate. It can't pass it to socket.ssl, since OpenSSL expects a host system file name, not a fragment of a zip file. If this was installed as files, it would work. In all of these cases, the applications could use pkg_resources.resource_filename(), which returns a true OS filename, even for files inside of .zip files. Of course, it does this by extracting to a cache directory in such cases, and is only suitable for read-only access. But it works fine for such cases as these. Passing a resource directory name results in an operating system directory name being returned as well, with all the contents (recursively extracted) therein. If the application is *not* in a zip file, then resource_filename() simply returns the obvious results by __file__ manipulation, so the author need not concern him or herself with this in code. They just use resource_string() or resource_string() wherever possible, and resort to resource_filename() when working with tools such as the above, that cannot handle anything but files. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
I don't understand it.
Have you read the manuals?
You mean,
http://peak.telecommunity.com/DevCenter/setuptools
Yes, I did. However, this would only enable me to use it (should I
ever find a need). What I really need to understand is how all this
stuff works inside.
Now, I haven't actually *tried* to learn it so far. Fredrik's question
was not who thinks he could learn it (I certainly could), I
read his question as who does understand it, as of today. I can't
faithfully answer yes to that question.
So far, I believe nobody has said yes, I do understand it, and could
maintain it if Phillip Eby became unavailable.
Please define magic.
Functionality with no obvious implementation, or functionality
implemented in an unobvious way.
Better, please point to the API functions or
classes, or the setup commands documented in the manual, to show me what
these things are that appear to be magic.
Here are some that I either already knew, or discovered just now:
1. setuptools manages to replace pydoc on earlier Python versions.
MAL assumed it does so by overwriting the files. This would be
bad practice, but is the obvious solution.
You then told him that this is not what actually happens (so
it's already magic); instead, you arrange that Python finds
your version before it finds its own.
Again, this is still magic: How does it do that? If you append
to sys.path, Python would still find its own version before
it finds yours. So perhaps you insert another path at
sys.path[0]? This would also be bad practice, but it would
be the next-obvious approach.
But I fully expect that you are now telling me that this is
still not how it works.
2. setuptools manages to install code deep inside a subdirectory
of site-python, and manages to install multiple versions
simultaneously.
How does that work? The first part (packages outside sys.path)
can be solved with a .pth file (which I already consider bad
practice, as it predates the package concept of Python);
but that can't work for multiple versions: magic.
I (think I) know the answer: there is a single .pth file,
and that is *edited* on package installation time. This
is a completely non-obvious approach. If Python needs
to support multiple versions of a package (and there
is apparently a need), that should be supported
properly, with a clear design, in a PEP.
3. namespace packages. From the documentation, I see that
you pass namespace_packages= to setup, and that you
put
__import__('pkg_resources').declare_namespace(__name__)
into the namespace package. How does that work?
The documentation only says
This code ensures that the namespace package machinery
is operating and that the current package is registered
as a namespace package.
Also, why not
import pkg_resources
pkg_resources.declare_namespace(__name__)
What does declare_namespace actually do?
4. zip_safe. Documentation says
If this argument is not supplied, the bdist_egg command
will have to analyze all of your project's contents
for possible problems each time it buids an egg.
How? Sounds like it is solving the halting problem, if
it manages to find possible problems.
There do exist implementation details that are not included in user
documentation, but this is going to be true of any system. These details
are sometimes necessarily complex due to distutils limitations, behavioral
quirks of deployed packages using distutils, and the sometimes baroque
variations in sys.path resolution across platforms, Python versions, and
invocation methods.
I trust that there is a good reason for each of them. However,
for inclusion in the standard library, some of these should go away:
- if you have distutils limitations, remove them
- differences in Python versions shouldn't matter: you always
know what the Python version is
- if there are baroque variations in sys.path resolution across
platforms, they should be removed, or modernized.
Not sure what invocation methods are.
This magic might do the right thing in many cases, and it might
indeed help the user that the magic is present, yet I still do believe
that magic is inherently evil: explicit is better than implicit.
Are documented defaults implicit or magic?
Documented defaults are explicit.
With respect to you and MAL, I think that some of your judgments regarding
setuptools may have perhaps been largely formed at a time last year when,
among other things:
* That documentation section I just referenced didn't exist yet
I don't think I ever complained about the lack of documentation. I would
(personally) not trust the documentation, anyway, but always look at the
code to find out how it actually works. I read documentation primarily
to find out what the *intent* was.
These are significant changes that are directly relevant to the objections
that you and he raised (regarding startup time, path
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On Thursday 20 April 2006 17:42, Martin v. Löwis wrote: 4. .egg files. -1 As far as I understand it, an egg file is just a zipimport format zip file with some additional metadata. You can also install the egg files in an unpacked way, if you prefer that. I don't understand the objection here - it's no better or worse than countless packages in site-packages, and if it gives us multiple versions of the same code, all the better. -- Anthony Baxter [EMAIL PROTECTED] It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On 4/20/06, Martin v. Löwis [EMAIL PROTECTED] wrote: 3. package resources. I dislike the term resources (it is not about natural gas, water, main memory, or processor cycles, right?); instead, this seems to provide access to embedded data files. Apparently, there is a need for it, so be it. -0 The resources name is actually quite a common meme; e.g. the Google internal API for this is in a module named resources.py (Google has its own set of functionality not unlike eggs). I'm also surprised you're -0 on the functionality. Lots of code has associated data files that must be distributed together with it, and IMO this is an absolutely essential feature for any packaging and distribution tool. This is another area where API standardization is important; as soon as modules are loaded out of a zip file, the traditional approach of looking relative to os.path.dirname(__file__) no longer works. - I disliked the fact that nobody explicitly approved inclusion of setuptools. Now that Anthony Baxter did, I'm fine. That's my fault. I told Phillip in no unclear terms that I wanted it. That was enough of an approval for me (and for Neal and Anthony, who were aware AFAIK). But I forgot to clarify this to python-dev. (Like with the proposal to turn print into a function, I assumed it was obvious that it was a good idea. :-) - I still fear that the code of setuptools will turn out to be a maintenance nightmare. AFAICT Phillip has explicitly promised he will maintain it (or if he doesn't, I expect that he would gladly do so if you asked). This has always been sufficient as a guarantee that a new module isn't orphaned. Beyond that, this objection is FUD; the circumstances of the original distutils aren't likely to be replicated here. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Anthony Baxter wrote: 4. .egg files. -1 As far as I understand it, an egg file is just a zipimport format zip file with some additional metadata. You can also install the egg files in an unpacked way, if you prefer that. I don't understand the objection here - it's no better or worse than countless packages in site-packages, and if it gives us multiple versions of the same code, all the better. It is worse: each .egg file makes an additional sys.path entry (if I understand correctly); every import statement will traverse every every package. I'm not sure precisely how all this works, but I wouldn't be surprised if the zip directory is read over and over again. Compare that to countless packages in site-packages: import foo will *just* look for foo.py, foo.so, and the directory foo. I understand there is a second API to importing, some kind of require() call. I consider that even worse: it obsoletes the language's mechanism for modules, and defines its own modularization. However, this isn't really my objection to .egg files. I dislike them because they compete with platform packages: .rpm, .msi, .deb. Package authors will refuse to produce them, putting the burden of package maintenance (what packages are installed, what are their dependencies, when should I remove a package) onto the the end user/system administrator. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Guido van Rossum wrote: This is another area where API standardization is important; as soon as modules are loaded out of a zip file, the traditional approach of looking relative to os.path.dirname(__file__) no longer works. Certainly. However, I get two conclusions out of this: 1. don't load packages out of .zip files. It's not that bad if software on the user's disk occupies multiple files, as long as there is a convenient way to get rid of them at once. Many problems go away if you just say no to zipimport. 2. standardize on file names, not API. If I want to deploy read-only data files, I put them into /usr/share. If I need read-write files, I put them into /var. I did not have such a problem yet on other systems, but I would also try to follow the conventions of these systems. With these combined, I can use any API I like to operate on the files. distutils already has support for that. Some libraries (not necessarily in Python) have gone the path of providing a unified API for all kinds of file stream access, e.g. in KDE, any tool can open files over many protocols, without the storage being mounted locally. I consider this approach flawed: once I leave the realm of KDE programs, this all stops working. It is the operating system's job to provide unified access to storage, not the programming language's or the job of a library. - I still fear that the code of setuptools will turn out to be a maintenance nightmare. AFAICT Phillip has explicitly promised he will maintain it (or if he doesn't, I expect that he would gladly do so if you asked). This has always been sufficient as a guarantee that a new module isn't orphaned. He has, and it is. Still, for whatever reason, I feel particularly uneasy here (and yes, that is my fear, my uncertainty, and my doubt). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On 20-apr-2006, at 21:53, Martin v. Löwis wrote: However, this isn't really my objection to .egg files. I dislike them because they compete with platform packages: .rpm, .msi, .deb. As far as I understand the issues they compete up to a point, but should also make it easier to create platform packages that contain proper the proper dependencies because those are part of machine-readable meta-data instead of being written down in readme files. Oddly enough that was also the objection from one linux distribution maintainer: somehow his opinion was that the author of a package couldn't possibly know the right depedencies for it. As for platform packages: not all platforms have useable packaging systems. MacOSX is one example of those, the system packager is an installer and doesn't include an uninstaller. Eggs make it a lot easier to manage python software in such an environment (and please don't point me to Fink or DarwinPorts on OSX, those have serious problems of their own). Package authors will refuse to produce them, putting the burden of package maintenance (what packages are installed, what are their dependencies, when should I remove a package) onto the the end user/system administrator. Philip has added specific support for them: it is possible to install packages in the tradition way but with some additional files that tell setuptools about installed packages. Maybe 'python setup.py install' should default to installing in that mode (as someone else already suggest), with either on option or a seperate command to install as eggs. Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Bob Ippolito wrote: They DO NOT compete any more than source packages do. eggs are packages plus metadata, nothing more. What eggs do and what rpm/msi/deb does are orthogonal. It's entirely reasonable that in the future rpm/msi/deb will simply be a delivery mechanism for eggs. That might be your view, but it apparently isn't the view of the inventor(s). From http://peak.telecommunity.com/DevCenter/setuptools Create Python Eggs - a single-file importable distribution format http://peak.telecommunity.com/DevCenter/PythonEggs 'Eggs are to Pythons as Jars are to Java...' 'There are several binary formats that embody eggs, but the most common is '.egg' zipfile format, because it's a convenient one for distributing projects.' '.egg files are a zero installation format for a Python package;' So the .egg inventors do view .egg files (i.e. the .egg zipfile format) as a distribution format, just like rpm/msi/deb *are* distribution formats (none of them zero installation, though, you always have to perform some deployment activity). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On 4/20/06, Martin v. Löwis [EMAIL PROTECTED] wrote: 1. don't load packages out of .zip files. It's not that bad if software on the user's disk occupies multiple files, as long as there is a convenient way to get rid of them at once. Many problems go away if you just say no to zipimport. You forget that importing from a zip file is often much faster than importing from the filesystem. That's why zipimport was added in the first place. 2. standardize on file names, not API. If I want to deploy read-only data files, I put them into /usr/share. If I need read-write files, I put them into /var. What about Windows? Putting data files in the same directory as source files solves a lot of problems of making the data follow the source no matter how/where it is installed (or if it's not installed at all, like when running code straight out of svn). Anyway, perhaps it's a matter of choice. It's clear to me that many developers prefer to do it this way. You don't. This is an area that has enough external constraints that I'm uncomfortable telling developers they can't do it that way. A standard API to access resources by name (with perhaps a registry for defining additional ways to find them) makes a lot of sense to me, and I don't think we're exactly inventing something novel here. [...] Some libraries (not necessarily in Python) have gone the path of providing a unified API for all kinds of file stream access, e.g. in KDE, any tool can open files over many protocols, without the storage being mounted locally. I consider this approach flawed: once I leave the realm of KDE programs, this all stops working. It is the operating system's job to provide unified access to storage, not the programming language's or the job of a library. I don't see it that way. All operating system access is mediated via the Python library anyway; and in many cases the Python library provides additional abstractions over what the operating system natively provides (e.g. many APIs in the os and os.path modules). You can't blame KDE for providing mechanisms that only work in the KDE world. It's fine for Python to provide Python-specific solutions for issues that have no cross-platform native solution. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Ronald Oussoren wrote: As far as I understand the issues they compete up to a point, but should also make it easier to create platform packages that contain proper the proper dependencies because those are part of machine-readable meta-data instead of being written down in readme files. Oddly enough that was also the objection from one linux distribution maintainer: somehow his opinion was that the author of a package couldn't possibly know the right depedencies for it. What he can't possibly know is the *name* of the package he depends on. For example, a distutils package might be called 'setuptools', so developers of additional packages might depend on 'setuptools'. However, on Debian, the dependency should be different: The package should depend on either 'python-setuptools' or 'python2.3-setuptools', depending on details which are off-topic here. As for platform packages: not all platforms have useable packaging systems. MacOSX is one example of those, the system packager is an installer and doesn't include an uninstaller. Eggs make it a lot easier to manage python software in such an environment (and please don't point me to Fink or DarwinPorts on OSX, those have serious problems of their own). Isn't uninstallation just a matter of deleting a directory? If I know that I want to uninstall the Python package 'foo', I just delete its directory. Now, with setuptools, I might have multiple versions installed, so I have to chose (in Finder) which of them I want to delete. That doesn't require Eggs to be single-file zipfiles; deleting a directory would work just as will (and I believe it will work with ez_install, too). Package authors will refuse to produce them, putting the burden of package maintenance (what packages are installed, what are their dependencies, when should I remove a package) onto the the end user/system administrator. Philip has added specific support for them: it is possible to install packages in the tradition way but with some additional files that tell setuptools about installed packages. As a system administrator, I don't *want* to learn how to install Python packages. I know how to install RPMs (or MSIs, or whatever system I manage); this should be good enough. If this Python junk comes with its own installer procedure, I will hate it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Greg Ewing wrote: The resources name is actually quite a common meme; I believe it goes back to the original Macintosh, which was the first and only computer in the world to have files with something called a resource fork. The resource fork contained pieces of data called resources. I can believe that history. Still, I thought a resource is something you can exhaust; the fork should have been named data fork or just second fork. Then Microsoft stole the name, and before you knew, everyone was using it. It's all been downhill from there. :-) Right. I'm not asking that the name is changed in setuptools - I'm just complaining about the state of the world, and showing my lack of intuition for the English language. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Bob Ippolito wrote: 'There are several binary formats that embody eggs, but the most common is '.egg' zipfile format, because it's a convenient one for distributing projects.' '.egg files are a zero installation format for a Python package;' single modules are also such a zero installation format too. So what? You're simply reading things between the lines that aren't there. How about you describe exactly what parts of the documentation that lead you to believe that eggs are designed to compete with solutions like rpm/msi/deb so that it can be clarified? It's not just the documentation: I firmly believe that many people consider .egg files to be a distribution and package management format. People have commented that some systems (e.g. OSX) doesn't have a usable native packager, so setuptools fills a need here. This shows that people do believe that .egg files are to OSX what .deb files are to Debian. As .egg files work on Debian, too, it is natural that they compete with .deb. Phillip Eby once said (IIRC) that he doesn't want package authors to learn all the different bdist_* commands (which also require access to the target systems sometimes), and that they their life gets easier as they now only have to ship the native Python binary packages, namely .egg files. In this view, rpm/msi/deb have no place anymore, and are obsolete. I can readily believe that package authors indeed see this as a simplification, but I also see an increased burden on system admins in return. So if this attitude (Python Eggs are the preferred binary distribution format) is wrong, it is the attitude that has to change first. Changes to the documentation follow from that. If the attitude is right, I'll have to accept that I have a minority opinion. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
At 10:59 PM 4/20/2006 +0200, Martin v. Löwis wrote: Bob Ippolito wrote: They DO NOT compete any more than source packages do. eggs are packages plus metadata, nothing more. What eggs do and what rpm/msi/deb does are orthogonal. It's entirely reasonable that in the future rpm/msi/deb will simply be a delivery mechanism for eggs. That might be your view, but it apparently isn't the view of the inventor(s). For the record, Bob *is* the co-inventor of eggs; he was the only person who responded to my call on the distutils-sig for the creation of a format for distribution of plugins, and it is largely due to his efforts that the egg runtime exists in the first place. In addition, it was he who convinced me that they would be good for installing arbitrary Python libraries, not just plugins for extensible applications. It's true that a lot of our initial discussions and even some of the documentation seem to blur the line a bit as to what eggs are. Clarity often takes time to develop. In fact, it wasn't until the Debian packaging flap on distutils-sig last fall that we were forced to clarify our thoughts enough to realize that distribution packaging was always just a means to an end - a way of reifying the idea of a library or plugin as a self-contained and self-describing unit. Distribution and installation are certainly a part of the overall picture, but the essential nature of eggs is to make *project releases* truly first-class objects in Python. A lot of your arguments (and some of MAL's) against eggs as unnecessary have been based on the (incorrect) theory that Python packages (importable unit) constitute the appropriate grain of modularity for system packages (distributable unit). But ever since the distutils existed, it has been clear that distribution units are not a 1:1 mapping with packages in the Python sense of the word. The stdlib itself gives lie to that concept, since there are many packages in it, yet it is distributed as a single project release. Eggs, however represent project releases as tangible objects: the 1.1 release of FooBar is a thing that can be installed, put on sys.path, searched for, depended on, and so forth. This is a valuable concept to have access to from Python code, independent of how these project releases actually end up on a system, or how they are physically arranged when they get there. It is intimately related to delivery and installation, of course. But to claim that they're competing with system packages is to miss the point. If you run bdist_rpm on a setuptools-based setup script, for example, you will get an RPM that looks just like any other RPM for a Python package, merely with the addition of an .egg-info directory that contains introspectable data. (And the same thing happens for bdist_wininst and any other bdist_* tool that uses setup.py install --root to build its target image.) In other words, if the delivery and installation part is handled by some other tool, then only the true egg remains: a project release as a discoverable and introspectable unit. That's not to say that the eggs didn't *originate* as a way to solve all of the problems in one fell swoop, but the point is that eggs do *NOT* compete with existing system packaging tools within their domain of competence. What I'm opposed to in making setuptools install things the distutils way by default is that there is no easy path to clean upgrade or installation in the absence of a system packaging tool like RPM or deb or what-have-you. I am not opposed to doing the classic style of installation by default *forever*, but only until setuptools can handle uninstallation and upgrades in that format. Right now, it's a lot easier to uninstall things if they are all in one directory. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
At 11:26 PM 4/20/2006 +0200, Martin v. Löwis wrote: Greg Ewing wrote: The resources name is actually quite a common meme; I believe it goes back to the original Macintosh, which was the first and only computer in the world to have files with something called a resource fork. The resource fork contained pieces of data called resources. I can believe that history. Still, I thought a resource is something you can exhaust; the fork should have been named data fork or just second fork. I suspect that the distinction is that data sounds too much like something belonging to the *user* of the program, whereas resources can reasonably be construed to be something that belongs to the program itself. By now, however, the term is popularly used with GUI toolkits of all kinds to mean essentially read-only data files that are required by a program or library to function, but which are not directly part of the code. Interestingly enough, there is another resource tool for Python out there, that actually works by converting resource files to strings in .py files, so that you can then just import them. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
On 4/20/06, Martin v. Löwis [EMAIL PROTECTED] wrote: I thought a resource is something you can exhaust; the fork should have been named data fork or just second fork. Hmm... I don't think that the English word resource necessarily implies that it can be exhausted. In US businesses, people are often also referred to as resources. I've come to think of it as simply what you need. The definition at http://www.answers.com/resource has this: 1. Something that can be used for support or help: The local library is a valuable resource. 2. An available supply that can be drawn on when needed. Often used in the plural. 3. The ability to deal with a difficult or troublesome situation effectively; initiative: a person of resource. 4. Means that can be used to cope with a difficult situation. Often used in the plural: needed all my intellectual resources for the exam. 5. 1. resources The total means available for economic and political development, such as mineral wealth, labor force, and armaments. 2. resources The total means available to a company for increasing production or profit, including plant, labor, and raw material; assets. 3. Such means considered individually. #1 and #2 especially explain the usage we're considering. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Guido van Rossum wrote: On 4/20/06, Martin v. Löwis [EMAIL PROTECTED] wrote: 1. don't load packages out of .zip files. It's not that bad if software on the user's disk occupies multiple files, as long as there is a convenient way to get rid of them at once. Many problems go away if you just say no to zipimport. You forget that importing from a zip file is often much faster than importing from the filesystem. That's why zipimport was added in the first place. This is true if you only have a single ZIP file, e.g. python24.zip on sys.path. It is not if you have dozens of ZIP files on sys.path, since you have to take the initial scan time plus the memory for caching the ZIP file contents into account. Each ZIp file adds to Python startup time, since the ZIP files on sys.path are scanned for every Python startup. Here's an example of the effect of having just 20 eggs on sys.path: tmp/eggs time python -S -c '0' 0.014u 0.006s 0:00.02 50.0% 0+0k 0+0io 0pf+0w tmp/eggs unsetenv PYTHONPATH tmp/eggs time python -S -c '0' 0.006u 0.003s 0:00.01 0.0% 0+0k 0+0io 0pf+0w (Reference: http://mail.python.org/pipermail/distutils-sig/2005-December/005739.html) Startup time effectively doubles, even if you don't use any of the installed eggs in your script. The main reason for adding zip imports was to simplify deploying (complete) applications to users, e.g. have the complete Python stdlib in one file. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
Phillip J. Eby wrote: What I'm opposed to in making setuptools install things the distutils way by default is that there is no easy path to clean upgrade or installation in the absence of a system packaging tool like RPM or deb or what-have-you. I am not opposed to doing the classic style of installation by default *forever*, but only until setuptools can handle uninstallation and upgrades in that format. Right now, it's a lot easier to uninstall things if they are all in one directory. If that's all it takes, have a look at the uninstall implementation in mxSetup.py (e.g. from egenix-mx-base). This will work with any half decent distutils setup.py file, since it uses the original install process as basis for finding the files to uninstall. (Not my idea: credits go to Thomas Heller.) BTW, if there's interest, I can look into porting the stuff in mxSetup.py into stock distutils. That'll give it uninstall, some autoconf and support for building Unix C libs as part of the process (plus a bunch of other things). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 21 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] magic in setuptools (Was: setuptools in the stdlib)
At 12:32 AM 4/21/2006 +0200, M.-A. Lemburg wrote: Phillip J. Eby wrote: What I'm opposed to in making setuptools install things the distutils way by default is that there is no easy path to clean upgrade or installation in the absence of a system packaging tool like RPM or deb or what-have-you. I am not opposed to doing the classic style of installation by default *forever*, but only until setuptools can handle uninstallation and upgrades in that format. Right now, it's a lot easier to uninstall things if they are all in one directory. If that's all it takes, have a look at the uninstall implementation in mxSetup.py (e.g. from egenix-mx-base). I have. As far as I can tell, it requires you to save the original setup.py, and for any dynamic considerations of that setup.py to still be in effect. What's actually needed is to save the list of outputs at *installation* time, so it's not practical to use your implementation for setuptools' purposes. I thought I'd explained this when you suggested this on the distutils-sig previously, but perhaps I forgot to. As well, there's another consideration for easy_install, which virtualizes the installation process for packages that don't use setuptools. Such packages may have customized their installation process by extending the distutils, *without* overriding get_outputs(). Since few people actually use the --record option for anything important, nobody notices when it breaks. It's often hard to implement correctly in the first place, so this is probably why most tools that wrap distutils prefer to use --root instead of --record anyway... which means that there's even less incentive to write correct get_outputs() methods. BTW, if there's interest, I can look into porting the stuff in mxSetup.py into stock distutils. That'll give it uninstall, some autoconf and support for building Unix C libs as part of the process (plus a bunch of other things). I imagine quite a lot of those things would be useful, except that I have personally noticed some issues through these extensions not following distutils standards. (Oh, the irony! :)) Specifically, I have noticed that your autoconf command does not use the 'build' command as the basis for obtaining its default '--compiler', which caused me quite a bit of trouble when trying to easy_install some of your packages. At least your mx_build_ext command handles this correctly. In general, distutils commands take their defaults from either the build or install master commands, as this allows users to have a single place to configure default options. But I digress. I do think there are a lot of useful things in your work and it was on my list to steal as many of your ideas as possible as soon as I got to working on setuptools things that needed them. :) I certainly wouldn't object to them being in the distutils, though I'm likely to nitpick certain little things like the aforementioned compiler defaults. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
