Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Steve Holden writes: Not only that, but the Cygwin packaging system appears to be extremely difficult to organize a package for. Really? I Don't Do Windows[tm], but the people who did installers and stuff for XEmacs releases never had problems with it. It was much more painful to create the .exe-style Windows installers. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Sun, 2009-04-05 at 18:08 +0900, Stephen J. Turnbull wrote: Steve Holden writes: Not only that, but the Cygwin packaging system appears to be extremely difficult to organize a package for. Really? I Don't Do Windows[tm], but the people who did installers and stuff for XEmacs releases never had problems with it. It was much more painful to create the .exe-style Windows installers. Back when I was maintaining setup.exe was when XEmacs started using setup.exe to do installers; it must have been fairly straight forward because we first heard of it when it was complete :). The following may have changed, but I doubt it has changed dramatically - the setup.exe system is kindof trivial: There is a .lst file which is a .INI format file listing packages and direct dependencies. - each package is a .tar.(gz|bz2) which is unpacked on disk, and [optional] post-install, pre-removal scripts inside the tarball. Doing an installer for something not part of Cygwin requires a one-time fork of the setup.exe program, to change the master source for .lst files, and thats about it. Beyond that its all maintaining whatever set of packages and dependencies you have. If you are installing things for Cygwin itself you can just depend directly on things Cygwin ships in your .lst file; and not ship a setup.exe at all - setup.exe can source from many places to satisfy dependencies. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: That's not entirely true; Cygwin comes with a package management tool that probably could be used to set up a repository of python packages for native Windows: http://sources.redhat.com/cygwin-apps/setup.html Ah, ok. It has the big disadvantage of not being Microsoft-endorsed, though. In that sense, it feels very much like easy_install (which also does dependencies). Not only that, but the Cygwin packaging system appears to be extremely difficult to organize a package for. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Op dinsdag 24-03-2009 om 20:51 uur [tijdzone +0100], schreef Martin v. Löwis: The Windows story is indeed sad, as none of the Windows packaging formats provides support for dependencies That's not entirely true; Cygwin comes with a package management tool that probably could be used to set up a repository of python packages for native Windows: http://sources.redhat.com/cygwin-apps/setup.html This package manager is in no way dependent on Cygwin, supports (basic) dependencies, etc. Of course some people would have to take care of the packaging work (just like happens for most open source OS distros and for Mac OS X already). It seems like XEmacs is already using a fork of that installer for the same purpose. -- Jan Claeys ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
That's not entirely true; Cygwin comes with a package management tool that probably could be used to set up a repository of python packages for native Windows: http://sources.redhat.com/cygwin-apps/setup.html Ah, ok. It has the big disadvantage of not being Microsoft-endorsed, though. In that sense, it feels very much like easy_install (which also does dependencies). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
David Cournapeau wrote: I won't argue for setuptools' implementation of multi-version. It sucks. But multi-version can be done well. Sonames in C libraries are a simple system that does this better. I would say simplistic instead of simple :) what works for C won't necessarily work for python - and even in C, library versioning is not used that often except for a few core libraries. Library versioning works in C because C model is very simple. It already breaks for C++. I'm not sure what you're talking about here. Library versioning is used for practically every library on a Linux system. My limited exposure to the BSDs and Solaris was the same. (If you're only talking Windows, well; does windows even have Sonames?) I can name only one library that isn't versioned in Fedora right now and may have heard of five total. Perhaps you are thinking of library symbols? If so, there are only a few libraries that are using that. But specifying backwards compatibility via soname is well known and ubiquitous. More high-level languages like C# already have a more complicated scheme (GAC) - and my impression is that it did not work that well. The SxS for dll on recent windows to handle multiple version is a nightmare too in my (limited) experience. Looking at C#/Mono/.net for examples is perfectly horrid. They've taken inferior library versioning and bad development practices and added technology (the GAC) as the solution. If you want an idea of what python should avoid at all costs, look to that arena for your answer. * Note that setuptools' multi-version implementation shares some things in common with the GAC. For instance, using directories to separate versions instead of filenames. setuptools' implementation could be made better by studying the GAC and taking things like caching of lookups from it but I don't encourage this... I think the design itself is flawed. -Toshio signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
2009/3/25 Steve Holden st...@holdenweb.com: As far as memory serves, Mike built the installers precisely by using distutils to build Windows installers. He then had to suffer criticism from people who suggested this was inappropriately complex for pure Python modules. In so far as end users won't wish to install individual Python modules this argument may have had some merit, but I personally thought the criticism unjustified since Mike's technique gave a uniform install procedure for everything. I've lost the context for this discussion completely, but can I offer a couple of points from a Windows user's point of view: 1. Using distutils to build Windows installers for pure-python packages is, in my view, *entirely* appropriate. The uniform install procedure (and more importantly, uniform *un*install procedure) is a key benefit for me. I can (and do) build my own installers for pure-python modules which do not come with an installer (easy enough, precisely because it's pure Python) but I feel that it's considerate and helpful of distributors to offer bdist_wininst (or bdist_msi) installers even when all it saves is a bit of my time. 2. Setuptools messes this clean picture up, for reasons I cannot understand, but which annoy me intensely. When setuptools is involved, it changes the default bdist_wininst behaviour somehow, so that the installers become version-specific. This increases the burden on distributors, which as a result means that you are less likely to find bdist_wininst installers for setuptools-using pure python packages (yes, I know, they give you eggs, and easy_install, yadda yadda...) So where setuptools is involved, things do start to become inappropriately complex - but that is a setuptools problem, NOT an issue with providing installers. 3. Setuptools, unfortunately, has divided the Python distribution community quite badly. I'd be curious to know whether the people who suggested this was inappropriately complex you mention above were setuptools supporters. My (unfounded) suspicion is that the argument was having to use a separate installer is more complex than just using easy_install - which is a highly debatable (and frequently debated!) point of view. I don't think anyone was suggesting that py2exe would be helpful for writing installers--simply to reduce complexity from the user's point of view and bundle dependencies in isolation from other applications. It seems to do that quite well. py2exe is good at creating bundled applications with no external dependencies. It doesn't make installers for these apps, just a directory containing the exe and all supporting files. That's a completely different scenario, as you say. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Stephen J. Turnbull wrote: Batteries included is the status quo for the (C) Python project, and I personally don't see a strong reason to change that policy until those who favor a package-manager-based solution have a package manager that satisfies them! Beautifully put :) And a package based solution that satisfies everyone (or even a significant majority), be they on Windows, OS X, .deb based Linux, .rpm based Linux, some other Linux or *nix, be they a mere user or the machine admin... I think that's a problem that still has a lot of development work in front of it ;) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 25, 2009, at 9:01 AM, Nick Coghlan wrote: And a package based solution that satisfies everyone (or even a significant majority), be they on Windows, OS X, .deb based Linux, .rpm based Linux, some other Linux or *nix, be they a mere user or the machine admin... I think that's a problem that still has a lot of development work in front of it ;) I don't think that will ever happen. I regularly use three *nix systems and each has a different package management systems and I don't think I even want a unified package manager. Tools like setuptools, zc.buildout, etc. seem great for developers but not very good for distributions. At last year's Pycon I think there was agreement from the Linux distributors that distutils, etc. just wasn't very useful for them. Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBScpFE3EjvBPtnXfVAQJ6FwP9GdT9gVyfDp1aCLlfXsw61hcwU+6E3sX2 kvuqRYyXeTlecvRWnhc0YKzLjIke+4K+z7DlKxKazWaOsgkLTxxKrB1HTOwqKsiA +PxIrHK5tSKbvoBLphFQVgtDcqzb07xxxsz0KsTxWR/mskvR0eYf1/XZl3+b+9sQ M1POnOxXCN0= =8VhZ -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Barry Warsaw wrote: Tools like setuptools, zc.buildout, etc. seem great for developers but not very good for distributions. At last year's Pycon I think there was agreement from the Linux distributors that distutils, etc. just wasn't very useful for them. It's decent for modules but has limitations that we run up against somewhat frequently. It's a horror for applications. -Toshio signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tools like setuptools, zc.buildout, etc. seem great for developers but not very good for distributions. At last year's Pycon I think there was agreement from the Linux distributors that distutils, etc. just wasn't very useful for them. I think distutils is different here - it not only helps creating distributions (i.e. deployable package files), but also allows direct installation. This, in turn, is used to build the packages for Linux distributions. E.g. debian/rules often contains a setup.py install call in its build step (and there is even a CDBS python-distutils.mk fragment) In that sense, distutils is for Python what make is for C. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Thu, Mar 26, 2009 at 12:26 PM, Martin v. Löwis mar...@v.loewis.de wrote: Tools like setuptools, zc.buildout, etc. seem great for developers but not very good for distributions. At last year's Pycon I think there was agreement from the Linux distributors that distutils, etc. just wasn't very useful for them. I think distutils is different here - it not only helps creating distributions (i.e. deployable package files), but also allows direct installation. This, in turn, is used to build the packages for Linux distributions. E.g. debian/rules often contains a setup.py install call in its build step (and there is even a CDBS python-distutils.mk fragment) In that sense, distutils is for Python what make is for C. It is more like the whole autotools suite (at least for unix), and that's the problem: distutils does everything quite poorly, and once you need to do something that distutils can't do out of the box, you are in a no man's land because distutils is almost impossible to extend (everything is done internally, with no way to recover data short of rewriting everything or monkey patching). To take a recent example, I wanted to add the ability to install a clib extension (pure C, no python API), so that it can be used by other projects: that would take 2 minutes with any build tool out there, but is almost impossible in distutils, unless you rewrite your own build_clib and/or install commands. Even autotools is more enjoyable, which is quite an achievement :) If distutils was split into different modules (one for the build, one for the compiler/platform configuration, one for the installation), which could be extended, tweaked, it would be much better. But the distutils design makes this inherently very difficult (commands). cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Thu, Mar 26, 2009 at 5:32 AM, David Cournapeau courn...@gmail.com wrote: If distutils was split into different modules (one for the build, one for the compiler/platform configuration, one for the installation), which could be extended, tweaked, it would be much better. But the distutils design makes this inherently very difficult (commands). I am not sur why the command design is a problem here. And I think Distutils features are not far from what you need, if you look at things like customize_compiler, which is called by build_clib. I'm ready to discuss your use case in Distutils-SIG, if you have a complete example etc. Cheers Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Thu, Mar 26, 2009 at 2:01 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Thu, Mar 26, 2009 at 5:32 AM, David Cournapeau courn...@gmail.com wrote: If distutils was split into different modules (one for the build, one for the compiler/platform configuration, one for the installation), which could be extended, tweaked, it would be much better. But the distutils design makes this inherently very difficult (commands). I am not sur why the command design is a problem here. For several reasons: - options handling cannot be done correctly. If you need to pass some options specific to the build, you have to pass it to build_clib and build_ext, etc... Example: I would really like to add options like --with-libfoo ala autoconf so that the packager can simply say where to look for a library (headers, .so, etc). This cannot be done easily in distutils (no persistence, no easy way to communicate between commands) - the whole concept of commands is bogus for a build tool. The correct way to do builds is with a DAG, to handle dependencies. And I think Distutils features are not far from what you need, if you look at things like customize_compiler, which is called by build_clib. The whole customize_compiler is awful. You cannot call it when you want, but only at some arbitrary time in the execution, which is not documented. You have to create your own command, because you can't call it in setup.py directly. You may have to call initialize_something_which_has_nothing_to_do_with_compiler(), which may break on windows because the MS compiler abstraction is totally different than the unix one. It is actually hard to believe if you never had to deal with it: so many trivial things which are one line of code in every other tool are difficult, obscure, magic or damn impossible in distutils. Modifiying compiler flag ? You have to create a new compiler class. Installing docs ? You have to create your own install class. etc... I'm ready to discuss your use case in Distutils-SIG, if you have a complete example etc. Ok, I will give you the example on the distutils ML, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Seems to me that while all this is fine for developers and Python users it's completely unsatisfactory for people who just want to use Python applications. For them it's much easier if each application comes with all dependencies including the interpreter. I think it depends on your engineering principles. If you absolutely have to use the latest version of any software package just because it is there, you will certainly end up with a wobbly setup where only a single combination of libraries can cooperate at all. OTOH, if you develop with backwards compatibility in mind, it may be more painful to develop, but much easier to deploy. Many of the larger libraries and applications (Twisted, Roundup, MoinMoin) support a wide range of Python releases, and an equally wide range of underlying libraries (in case they need any) - so it can be done. Of course, if you have just a single installation for your application, developing it portably across various versions could be wasted effort. However, in that case, I found it sufficient to just make it work on (the Debian packages of) that single installation. Every time I upgrade to a new Debian release, I have to see what breaks in my applications. Most of the time, very little if any effort is needed. So I disagree that application developers homogeneously prefer packaging everything into a single stand-alone package. For some, it's just not an option, for others, it's unnecessary. I do develop applications myself, and had only once in ten years the desire to package everything in a stand-alone way, and then ended up using freeze. I'm genuinely curious what the scenarios are where people desire such packaging - I did hear the desire often, but never fully explained. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
I do develop applications myself, and had only once in ten years the desire to package everything in a stand-alone way, and then ended up using freeze. I'm genuinely curious what the scenarios are where people desire such packaging - I did hear the desire often, but never fully explained. There is one case where I understand that desire: py2exe on Windows (which is similar to my own case, except that I wanted to support Solaris). It's the case where you produce an application to download by end users, and you don't want to bother end users with installing Python first. However, I think that py2exe also solves this case sufficiently. One might debate whether freeze would be better, or whether it would be better to produce MSI instead of exe, but in essence, the result will be always the same. I don't think a cross-platform packaging solution can help here in any way. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Steve Holden writes: Seems to me that while all this is fine for developers and Python users it's completely unsatisfactory for people who just want to use Python applications. ? I use lots of Python applications under Mac OS X, MacPorts, Debian, and Gentoo without looking too closely at what the cat drags in. In fact, in many cases I don't even know and don't care that they're Python apps. As long as the cat stays in her corner, I'm happy. For them it's much easier if each application comes with all dependencies including the interpreter. This may seem wasteful, but it removes many of the version compatibility issues that otherwise bog things down. For that version of the application. As soon as you start thinking about version upgrades of the application, the package distribution problem reappears (in the sense that somebody must manage a whole set of packages in the application's installed tree, make sure they get removed at upgrade, deal with multiple concurrent installations so that beta versions can be tested and compared with one's current production installation, etc). This is really the same as the problem that Python currently faces and solves with the batteries-included stdlib, just specialized to a particular application. But those applications are similarly likely to experience bitrot in their bundled dependencies, and *some* but not all users will find that somewhere between annoying and disastrous, just like the Python-with-stdlib distribution itself. Degree of each kind of issue will change, of course, but the issues do remain. As far as I can *all* of these various solutions have their uses, and each is going to be strongly favored by some groups of users, and more or less unacceptable to others. These preferences may or may not be correlated with specific apps written in Python. Batteries included is the status quo for the (C) Python project, and I personally don't see a strong reason to change that policy until those who favor a package-manager-based solution have a package manager that satisfies them! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: I do develop applications myself, and had only once in ten years the desire to package everything in a stand-alone way, and then ended up using freeze. I'm genuinely curious what the scenarios are where people desire such packaging - I did hear the desire often, but never fully explained. There is one case where I understand that desire: py2exe on Windows (which is similar to my own case, except that I wanted to support Solaris). It's the case where you produce an application to download by end users, and you don't want to bother end users with installing Python first. However, I think that py2exe also solves this case sufficiently. One might debate whether freeze would be better, or whether it would be better to produce MSI instead of exe, but in essence, the result will be always the same. I don't think a cross-platform packaging solution can help here in any way. I'm not convinced we do need a cross-platform packaging solution, so I may have explained my views badly. I regard application developers as Python users, so I did not intend to suggest that the requirement for stand-alone installation came from them. My main concern is that if Linux and Unix (Lunix) application installation results, as is the case with setuptools, in the download and/or installation of arbitrary support packages then we may end up condemning Python app users to our own version of DLL hell (package purgatory?). As far as Lunix distributions are concerned I'd be happy enough to rely on the distributed Python, and let people who replace it suffer the consequences and maintain their own Python packages as required. So I suppose that we *could* take the interpreter as a given. But I don't think that a normal Lunix user should ever have to involve themselves directly with distutils or setuptools (it's OK to invoke them from the installation routine, but not to require the user to unpack the application then run setup.py). I am afraid that distutils, and setuptools, are not really the answer to the problem, since while they may (as intended) guarantee that Python applications can be installed uniformly across different platforms they also more or less guarantee that Python applications are installed differently from all other applications on the platform. Mike Driscoll did some work providing Windows installers for various Python packages and extension modules, and people were amused that he provided executable installers for pure Python libraries. But I saw that as a sensible decision, since it meant that Windows users (and even Windows Python application developers) used the same mechanism to install everything. I don't claim to have all the answers, but distutils and setuptools are for the savvy users - people who build distros and those who maintain their own systems independent of their distribution's packaging standard. As time goes by, however, and the Lunix installed base continues to grow, the majority of users will expect to rely on the standard installation mechanisms of their distribution, and that will never be setuptools or distutils. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Tue, Mar 24, 2009 at 8:53 PM, Steve Holden st...@holdenweb.com wrote: I'm not convinced we do need a cross-platform packaging solution, so I may have explained my views badly. I regard application developers as Python users, so I did not intend to suggest that the requirement for stand-alone installation came from them. My main concern is that if Linux and Unix (Lunix) application installation results, as is the case with setuptools, in the download and/or installation of arbitrary support packages then we may end up condemning Python app users to our own version of DLL hell (package purgatory?). There already is a bit of a DLL hell in python. The whole idea to solve the dependency problems by installing multiple version of the same software is fundamentally flawed, it just does not work for general deployment on multiple machines. Many systems outside python, with more resource, have tried - and failed. By enabling a general, system-wide installation of multiple version of the same package, setuptools has made the situation worse. I am quite puzzled than many people don't realize this fundamental issue, it is a simple combinatory problem. If the problem is to get a recent enough version of the library, then the library would better be installed locally, for the application. If it is too much a problem because the application depends on billions of libraries which are 6 months old, the problem is to allow such a dependency in the first place. What kind of nightmare would it be if programs developed in C would required a C library which is 6 months old ? That's exactly what multiple-versions installations inflict on us. That's great for testing, development. But for deployment on end-user machines, the whole thing is a failure IMO. I am afraid that distutils, and setuptools, are not really the answer to the problem, since while they may (as intended) guarantee that Python applications can be installed uniformly across different platforms they also more or less guarantee that Python applications are installed differently from all other applications on the platform. I think they should be part of the solution, in the sense that they should allow easier packaging for the different platforms (linux, windows, mac os x and so on). For now, they make things much harder than they should (difficult to follow the FHS, etc...). But otherwise, I agree. Python applications which care about non-savy users should be distributed as .dmg, .exe, .rpm, .deb. There will always be some work to do that correctly: there is no way to provide a general, automatic environment to build installers which provide a good experience on all platforms. AFAIK, It does not even exist in the commercial landscape, so I see little chance to see this in the python ecosystem. cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Stephen J. Turnbull wrote: Chris Withers writes: - debian has an outdated and/or broken version of your package. True, but just as for the package system you are advocating, it's quite easy to set up your apt to use third-party repositories of Debian-style packages. The question is whether those repositories exist. Introducing yet another, domain-specific package manager will make it less likely that they do, and it will cause more work for downstream distributors like Debian and RH. I haven't seen this mentioned so -- For many sites (including Fedora, the one I work on), the site maintains a local yum/apt repository of packages that are necessary for getting certain applications to run. This way we are able to install a system with a distribution that is maintained by other people and have local additions that add more recent versions only where necessary. This has the following advantages: 1) We're able to track our changes to the base OS. 2) If the OS vendor releases an update that includes our fixes, we're able to consume it without figuring out on which boxes we have to delete what type of locally installed file (egg, jar, gem, /usr/local/bin/program, etc). 3) We're using the OS vendor package management system for everything so junior system admins can bootstrap a new machine with only familiarity with that OS. We don't have to teach them about rpm + eggs + gems + where to find our custom repositories of each. 4) If we choose to, we can separate out different repositories for different sets of machines. Currently we have the main local repo and one repo that only the builders pull from. -Toshio signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Steve Holden wrote: Seems to me that while all this is fine for developers and Python users it's completely unsatisfactory for people who just want to use Python applications. For them it's much easier if each application comes with all dependencies including the interpreter. This may seem wasteful, but it removes many of the version compatibility issues that otherwise bog things down. The upfront cost of bundling is lower but the maintenance cost is higher. For instance, OS vendors have developed many ways of being notified of and dealing with security issues. If there's a security issue with gtkmozdev and the python bindings to it have to be recompiled, OS vendors will be alerted to it and have the opportunity to release updates on zero day, the day that the security announcement goes out. Bundled applications suffer in two ways here: 1) the developers of the applications are unlikely to be on vendor-sec and so the opportunity for zero day fixes is lower. 2) the developer becomes responsible for fixing problems with the libraries, something that they often do not. This is especially true when developers start depending, not only on newer features of some libraries, but older versions of others (for API changes). It's not clear to many developers that requiring a newer version of a library is at least supported by upstream whereas requiring an older version leaves them as the sole responsible party. 3) Over time, bundled libraries tend to become forked versions. And worse, privately forked versions. If three python apps all use slightly different older versions of libfoo-python and have backported fixes, added new features, etc it is a nightmare for a system administrator or packager to get them running with a single version from the system library or forward port them. And because they're private forks the developers lose out on collaborating on security, bugfixes, etc because they are doing their work in isolation from the other forks. -Toshio signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Tue, 24 Mar 2009 at 07:59, Toshio Kuratomi wrote: 3) Over time, bundled libraries tend to become forked versions. And worse, privately forked versions. If three python apps all use slightly different older versions of libfoo-python and have backported fixes, added new features, etc it is a nightmare for a system administrator or packager to get them running with a single version from the system library or forward port them. And because they're private forks the developers lose out on collaborating on security, bugfixes, etc because they are doing their work in isolation from the other forks. This is one of the things that I really disliked about Java when I had to work with it: the culture there is that you ship a zip bundle that has all the libraries in it, at some unknown version or another. I fortunately did not run into any of those security or version-drift/bug issues in the relatively short time I worked with it, but I could feel those issues looming in the background and it made my skin crawl :( I'm very happy that Gentoo keeps the libraries separate when it packages Java applications. -- R. David Murray http://www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
2009/3/24 Toshio Kuratomi a.bad...@gmail.com: Steve Holden wrote: Seems to me that while all this is fine for developers and Python users it's completely unsatisfactory for people who just want to use Python applications. For them it's much easier if each application comes with all dependencies including the interpreter. This may seem wasteful, but it removes many of the version compatibility issues that otherwise bog things down. The upfront cost of bundling is lower but the maintenance cost is higher. For instance, OS vendors have developed many ways of being notified of and dealing with security issues. If there's a security issue with gtkmozdev and the python bindings to it have to be recompiled, OS vendors will be alerted to it and have the opportunity to release updates on zero day, the day that the security announcement goes out. I don't think bundling should be compared to depending on the system libraries, but as a lesser evil compared to requiring multiple, system-wide installed libraries. 3) Over time, bundled libraries tend to become forked versions. And worse, privately forked versions. If three python apps all use slightly different older versions of libfoo-python and have backported fixes, added new features, etc it is a nightmare for a system administrator or packager to get them running with a single version from the system library or forward port them. And because they're private forks the developers lose out on collaborating on security, bugfixes, etc because they are doing their work in isolation from the other forks. This is a purely technical problem, and can be handled by good source control systems, no ? cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
David Cournapeau wrote: 2009/3/24 Toshio Kuratomi a.bad...@gmail.com: Steve Holden wrote: Seems to me that while all this is fine for developers and Python users it's completely unsatisfactory for people who just want to use Python applications. For them it's much easier if each application comes with all dependencies including the interpreter. This may seem wasteful, but it removes many of the version compatibility issues that otherwise bog things down. The upfront cost of bundling is lower but the maintenance cost is higher. For instance, OS vendors have developed many ways of being notified of and dealing with security issues. If there's a security issue with gtkmozdev and the python bindings to it have to be recompiled, OS vendors will be alerted to it and have the opportunity to release updates on zero day, the day that the security announcement goes out. I don't think bundling should be compared to depending on the system libraries, but as a lesser evil compared to requiring multiple, system-wide installed libraries. Well.. I'm not so sure it's even a win there. If the libraries are installed system-wide, at least the consumer of the application knows: 1) Where to find all the libraries to audit the versions when a security issue is announced. 2) That the library is unforked from upstream. 3) That all the consumers of the library version have a central location to collaborate on announcing fixes to the library. With my distribution packager hat on, I can say I dislike both multiple versions and bundling but I definitely dislike bundling more. 3) Over time, bundled libraries tend to become forked versions. And worse, privately forked versions. If three python apps all use slightly different older versions of libfoo-python and have backported fixes, added new features, etc it is a nightmare for a system administrator or packager to get them running with a single version from the system library or forward port them. And because they're private forks the developers lose out on collaborating on security, bugfixes, etc because they are doing their work in isolation from the other forks. This is a purely technical problem, and can be handled by good source control systems, no ? No. This is a social problem. Good source control only helps if I am tracking upstream's trunk so I'm aware of the direction that their changes are headed. But there's a wide range of reasons that application developers that bundle libraries don't do that: 1) not enough time in a day. I'm working full-time on making my application better. Plus I have to update all these bundled libraries from time to time, testing that the updates don't break anything. I don't have time to track trunk for all these libraries -- I barely have time to track releases. 2) My release schedule doesn't mesh with all of the upstream libraries I'm bundling. When I want to release Foo-1.0, I want to have some assurance that the libraries I'm bundling with will do the right thing. Since releases see more testing than trunk, tracking trunk for twenty bundled libraries is a lot less attractive than tracking release branches. 3) This doesn't help with the fact that my bundled version of the library and your bundled version of the library are being developed in isolation from each other. This needs central coordination which people who believe bundling libraries are very unlikely to pursue. -Toshio signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 R. David Murray wrote: On Tue, 24 Mar 2009 at 07:53, Steve Holden wrote: I don't claim to have all the answers, but distutils and setuptools are for the savvy users - people who build distros and those who maintain their own systems independent of their distribution's packaging standard. As time goes by, however, and the Lunix installed base continues to grow, the majority of users will expect to rely on the standard installation mechanisms of their distribution, and that will never be setuptools or distutils. I'll note that such users are mostly going to be installing desktop applications, which make a good case for the batteries included story, but one which ignores the system Python altogether, and bundles the interpreter. I think that building such a self-contained bundle would be eased by a strong story for managing out-of-core distributions. +1. Or any other Python-only package manager. And that goes at least double for Windows :) I just did an install for a customer yesterday (actually, the customer did it, with me as air traffic controller), which started from the current Plone Windows installer, and then used the zc.buildout machinery shipped with Plone to add on the extra distributions (seven or so community-maintained, plus one containing custom code). The buildout was driven from a text-based config file I supplied to the customer, which I developed and tested first on Linux. This method was *way* less painful and error-prone than downloading and running eight separate installers, *and* it ensured repeatability, because the extra distributions were all installed from a PyPI-style index I set up for the customer: I can recreate the environment at will (well, all the bits on disk under the main installation directory). If I need to fix one of the distributions which make up the application, ar add a new one, I will roll a new 'sdist' for it (no C extensions in play), upload it to the index, rebuild the metadata, and tell the customer to re-run the 'buildout.exe'. I am (among other things) a software developer, and I am quite capable of maintaining my own python installation outside the system one, and do so for testing and development. But for anything else, I rely on the system package manager to install my Python packages. If the extension hasn't been packaged for my distribution, I typically will either not use it, or I will install it somewhere other than the system default location. At which point you can benefit from the distributions-with-dependencies- in-an-index format provided, with varying levels of success, by setuptools today. I think the confusion-in-terms here may be that many Python developers don't use a large number of out-of-core distributions, and thus are mostly OK with managing the few they do use as part of their application: the rest they get from a batteries include Python. Mostly that is a reasonable choice, although I note that the Python 2.5 releases before 2.5.2 broke applicatinos using the tarfile module, and the 2.6 release broke applications using asyncore / asynchat: if those packages were shipped outside the stdlib, I could have fixed the breakage by updating them separately. Those of us who do need to deploy applications with larger dependency sets need tools which make it possible to do the deployments repeatably. The use .deb / .rpm / Windows installer answer just moves the effort around (and adds to it, in my experience), because in that world I end up running the equivalent of a PyPI index anyway: I can't rely on an upstream distro to package the correct versions for me, or to avoid breaking my application by the packaging choices they make. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJyRPu+gerLs4ltQ4RAjgPAKDB12YNxmAJX7VFN9WGvMR3H8KPHgCgt0NN vCKUCTkVcxjP09/LQVBP8G4= =2qbi -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David Cournapeau wrote: There already is a bit of a DLL hell in python. The whole idea to solve the dependency problems by installing multiple version of the same software is fundamentally flawed, it just does not work for general deployment on multiple machines. Many systems outside python, with more resource, have tried - and failed. By enabling a general, system-wide installation of multiple version of the same package, setuptools has made the situation worse. I am quite puzzled than many people don't realize this fundamental issue, it is a simple combinatory problem. Many of us using setuptools extensively tend to adopt an isolated environment strategy (e.g., pip, virtualenv, zc.buildout). We don't install the packages used by different applications into shared directories at all. Instead, each environment uses a restricted subset of packages known to work together. If the problem is to get a recent enough version of the library, then the library would better be installed locally, for the application. If it is too much a problem because the application depends on billions of libraries which are 6 months old, the problem is to allow such a dependency in the first place. What kind of nightmare would it be if programs developed in C would required a C library which is 6 months old ? That's exactly what multiple-versions installations inflict on us. That's great for testing, development. But for deployment on end-user machines, the whole thing is a failure IMO. It is wildly successful, even on platforms such as Windows, when you abandon the notion that separate applications should be sharing the libaries they need. I am afraid that distutils, and setuptools, are not really the answer to the problem, since while they may (as intended) guarantee that Python applications can be installed uniformly across different platforms they also more or less guarantee that Python applications are installed differently from all other applications on the platform. I think they should be part of the solution, in the sense that they should allow easier packaging for the different platforms (linux, windows, mac os x and so on). For now, they make things much harder than they should (difficult to follow the FHS, etc...). FHS is something which packagers / distributors care about: I strongly doubt that the end users will ever notice, particularly for silliness like 'bin' vs. 'sbin', or architecture-specific vs. 'noarch' rules. But otherwise, I agree. Python applications which care about non-savy users should be distributed as .dmg, .exe, .rpm, .deb. There will always be some work to do that correctly: there is no way to provide a general, automatic environment to build installers which provide a good experience on all platforms. AFAIK, It does not even exist in the commercial landscape, so I see little chance to see this in the python ecosystem. As a counter-example, I offer the current Plone installer[1], which lays down a bunch of egg-based packages in a cross-platform way (Windows, MacOSX, Linux, BSDs). It uses zc.buildout, which makes configuration-driven (repeatable) installation of add-ons easy. [1] http://plone.org/products/plone Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJyRZH+gerLs4ltQ4RAu3qAKDXhkY73S2htl6Ytz+ZPVyRgYSHLACdGo+N XNMrpg7ue0SX3ODgiOjHTmg= =Rk1D -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Wed, Mar 25, 2009 at 1:45 AM, Toshio Kuratomi a.bad...@gmail.com wrote: David Cournapeau wrote: 2009/3/24 Toshio Kuratomi a.bad...@gmail.com: Steve Holden wrote: Seems to me that while all this is fine for developers and Python users it's completely unsatisfactory for people who just want to use Python applications. For them it's much easier if each application comes with all dependencies including the interpreter. This may seem wasteful, but it removes many of the version compatibility issues that otherwise bog things down. The upfront cost of bundling is lower but the maintenance cost is higher. For instance, OS vendors have developed many ways of being notified of and dealing with security issues. If there's a security issue with gtkmozdev and the python bindings to it have to be recompiled, OS vendors will be alerted to it and have the opportunity to release updates on zero day, the day that the security announcement goes out. I don't think bundling should be compared to depending on the system libraries, but as a lesser evil compared to requiring multiple, system-wide installed libraries. Well.. I'm not so sure it's even a win there. If the libraries are installed system-wide, at least the consumer of the application knows: 1) Where to find all the libraries to audit the versions when a security issue is announced. 2) That the library is unforked from upstream. 3) That all the consumers of the library version have a central location to collaborate on announcing fixes to the library. Yes, those are problems, but installing multi libraries have a lot of problems too: - quickly, by enabling multiple version installed, people become very sloppy to handle versions of the dependencies, and this increases a lot the number of libraries installed - so the advantages above for system-wide installation becomes intractable quite quickly - bundling also supports a real user-case which cannot be solved by rpm/deb AFAIK: installation without administration privileges. - multi-version installation give very fragile systems. That's actually my number one complain in python: setuptools has caused me numerous headache, and I got many bug reports because you often do not know why one version was loaded instead of another one. So I am not so convinced multiple-version is better than bundling - I can see how it sometimes can be, but I am not sure those are that important in practice. No. This is a social problem. Good source control only helps if I am tracking upstream's trunk so I'm aware of the direction that their changes are headed. But there's a wide range of reasons that application developers that bundle libraries don't do that: 1) not enough time in a day. I'm working full-time on making my application better. Plus I have to update all these bundled libraries from time to time, testing that the updates don't break anything. I don't have time to track trunk for all these libraries -- I barely have time to track releases. Yes, but in that case, there is nothing you can do. Putting everything in one project is always easier than splitting into modules, coding and deployment-wise. That's just one side of the speed of development vs maintenance issue IMHO. 3) This doesn't help with the fact that my bundled version of the library and your bundled version of the library are being developed in isolation from each other. This needs central coordination which people who believe bundling libraries are very unlikely to pursue. As above, I think that in that case, it will happen whatever tools are available, so it is not a case worth being pursued. cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Wed, Mar 25, 2009 at 2:20 AM, Tres Seaver tsea...@palladion.com wrote: Many of us using setuptools extensively tend to adopt an isolated environment strategy (e.g., pip, virtualenv, zc.buildout). We don't install the packages used by different applications into shared directories at all. Instead, each environment uses a restricted subset of packages known to work together. Is that a working solution when you want to enable easy installation on a large number of customers ? In those discussions, I often see different solutions depending on the kind of projects people do. I don't know anything about plone, but I can imagine the deployment issues are quite different from the projects I am involved in (numpy and co). Everytime I tried to understand what buildout was about, I was not even sure it could help for my own problems at all. It seems very specific to web development - I may completely miss the point ? virtualenv, pip, yolk, those are useful tools for development/testing, but I don't see how they can help me to make the installation of a numpy environment easier on many different kind of platforms. If the problem is to get a recent enough version of the library, then the library would better be installed locally, for the application. If it is too much a problem because the application depends on billions of libraries which are 6 months old, the problem is to allow such a dependency in the first place. What kind of nightmare would it be if programs developed in C would required a C library which is 6 months old ? That's exactly what multiple-versions installations inflict on us. That's great for testing, development. But for deployment on end-user machines, the whole thing is a failure IMO. It is wildly successful, even on platforms such as Windows, when you abandon the notion that separate applications should be sharing the libaries they need. Well, I may not have been clear: I meant that in my experience, deploying something with several dependencies was easier with bundling than with a mechanism ala setuptools with *system-wide* installation of multiple versions of the same library. So I think we agree here: depending on something stable (python stdlib + a few well known things) system-wide is OK, for anything else, not sharing is easier and more robust in the current state of things, at least when one needs to stay cross platform. Almost every deployment problem I got from people using my own softwares was related to setuptools, and in particular the multiple version thing. For end-users who know nothing about python package mechanism, and do not care about it, that's really a PITA to debug, and give bad mouth taste. The fact that those problems happen when my software was not even *using* setuptools/etc... was a real deal breaker for me, and I am strongly biased against setuptools ever since. FHS is something which packagers / distributors care about: I strongly doubt that the end users will ever notice, particularly for silliness like 'bin' vs. 'sbin', or architecture-specific vs. 'noarch' rules. That's not silly, and that's a bit of a fallacy. Of course end users do not care about the FHS in itself, but following the FHS enables good integration in the system, which end users do care about. I like finding my doc in /usr/share/doc whatever thing I install - as I am sure every window user like to find his installed software in the panel control. As a counter-example, I offer the current Plone installer[1], which lays down a bunch of egg-based packages in a cross-platform way (Windows, MacOSX, Linux, BSDs). It uses zc.buildout, which makes configuration-driven (repeatable) installation of add-ons easy. But zc.buildout is not a solution to deploy applications, right ? In my understanding, it is a tool to deploy plone instances on server/test machines, but that's quite a different problem from installing applications for users who may not even know what python is. cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David Cournapeau wrote: On Wed, Mar 25, 2009 at 2:20 AM, Tres Seaver tsea...@palladion.com wrote: Many of us using setuptools extensively tend to adopt an isolated environment strategy (e.g., pip, virtualenv, zc.buildout). We don't install the packages used by different applications into shared directories at all. Instead, each environment uses a restricted subset of packages known to work together. Is that a working solution when you want to enable easy installation on a large number of customers ? In those discussions, I often see different solutions depending on the kind of projects people do. I don't know anything about plone, but I can imagine the deployment issues are quite different from the projects I am involved in (numpy and co). Plone is downloaded and installed on many-many systems, across all the mainline platforms. In each case (since Plone 3.2), the installer is based on (and includes) zc.buildout, and documents[1] how to add new bits to the installed Plone by modifying the buildout.cfg file. Everytime I tried to understand what buildout was about, I was not even sure it could help for my own problems at all. It seems very specific to web development - I may completely miss the point ? I think so: it is largely a way to get repeatable / scripted deployment of software to disk. It uses setuptools to install Python package distributions, but also can use other means (e.g, configure-make-make install to install a C library such as libxml2). The end result is a self-contained directory tree: - - Scripts in the 'bin' directory are configured to have the specific Python pacakges (and versions) they require on the PYTHONPATH. - - By convention, released package distributions are installed into the 'eggs' subdirectory', which is *not* on the PYTHONPATH, nor is it a 'site' directory for Python. - - Other bits are typically in their own subdirectories, often under 'parts'. virtualenv, pip, yolk, those are useful tools for development/testing, but I don't see how they can help me to make the installation of a numpy environment easier on many different kind of platforms. When not doing Plone / Zope-specific work (where zc.buildout is a de facto standard), I use 'virtualenv' to create isolated environments into which I install the libraries for a given application. If your application ships as Python package distributions, then documenting the use of 'virtualenv' as a supported way to install it might reduce your support burden. You can even ship a virtualenv-derived script which pre-installs your own packages into such an environment, isolated from the other pacakges installed on the machine. If the problem is to get a recent enough version of the library, then the library would better be installed locally, for the application. If it is too much a problem because the application depends on billions of libraries which are 6 months old, the problem is to allow such a dependency in the first place. What kind of nightmare would it be if programs developed in C would required a C library which is 6 months old ? That's exactly what multiple-versions installations inflict on us. That's great for testing, development. But for deployment on end-user machines, the whole thing is a failure IMO. It is wildly successful, even on platforms such as Windows, when you abandon the notion that separate applications should be sharing the libaries they need. Well, I may not have been clear: I meant that in my experience, deploying something with several dependencies was easier with bundling than with a mechanism ala setuptools with *system-wide* installation of multiple versions of the same library. So I think we agree here: depending on something stable (python stdlib + a few well known things) system-wide is OK, for anything else, not sharing is easier and more robust in the current state of things, at least when one needs to stay cross platform. You can think of zc.buildout or the virtualenv-based script as a form of bundling, which bootstraps from another already-installed Python, but remains isolated from anything in its 'site-packages'. Almost every deployment problem I got from people using my own softwares was related to setuptools, and in particular the multiple version thing. I never even use that switch manually. zc.buildout does, but that is because it wants to control the PYTHONPATH by generating the script code: it doesn't ask users to tweak that. For end-users who know nothing about python package mechanism, and do not care about it, that's really a PITA to debug, and give bad mouth taste. The fact that those problems happen when my software was not even *using* setuptools/etc... was a real deal breaker for me, and I am strongly biased against setuptools ever since. I don't know why anybody who was not writing a packaging tool, or packaging a library for something like .deb /
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tres Seaver wrote: David Cournapeau wrote: I am afraid that distutils, and setuptools, are not really the answer to the problem, since while they may (as intended) guarantee that Python applications can be installed uniformly across different platforms they also more or less guarantee that Python applications are installed differently from all other applications on the platform. I think they should be part of the solution, in the sense that they should allow easier packaging for the different platforms (linux, windows, mac os x and so on). For now, they make things much harder than they should (difficult to follow the FHS, etc...). FHS is something which packagers / distributors care about: I strongly doubt that the end users will ever notice, particularly for silliness like 'bin' vs. 'sbin', or architecture-specific vs. 'noarch' rules. That's because you're thinking of a different class of end-user than FHS is targeting. Someone who wants to install a web application on a limited number of machines (one in the home-desktop scenario) or someone who makes their living helping people to install the software they've written has a whole different view on things than someone who's trying to install and maintain the software on fifteen computer labs in a campus or the person who is trying to write software that is portable to tens of different platforms in their spare time and every bit of answering end user's questions, tracking other upstreams for security bugs, etc, is time taken away from coding. Following FHS means that the software will work for both end-users who don't care about the nitty-gritty of the FHS and system administrators of large sites. Disregarding the FHS because it is silliness means that system administrators are going to have to special-case your application, decide not to install it at all, or pay someone else to support it. Note that those things do make sense sometimes. For instance, when an application is not intended to be distributed to a large number of outside entities (facebook, flikr, etc) or when your revenue stream is making money from installing and administering a piece of software for other companies. -Toshio signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Wed, Mar 25, 2009 at 3:15 AM, Tres Seaver tsea...@palladion.com wrote: Everytime I tried to understand what buildout was about, I was not even sure it could help for my own problems at all. It seems very specific to web development - I may completely miss the point ? I think so: it is largely a way to get repeatable / scripted deployment of software to disk. It uses setuptools to install Python package distributions, but also can use other means (e.g, configure-make-make install to install a C library such as libxml2). The end result is a self-contained directory tree: - - Scripts in the 'bin' directory are configured to have the specific Python pacakges (and versions) they require on the PYTHONPATH. - - By convention, released package distributions are installed into the 'eggs' subdirectory', which is *not* on the PYTHONPATH, nor is it a 'site' directory for Python. - - Other bits are typically in their own subdirectories, often under 'parts'. Ok - but I don't think it helps much, see below. When not doing Plone / Zope-specific work (where zc.buildout is a de facto standard), I use 'virtualenv' to create isolated environments into which I install the libraries for a given application. If your application ships as Python package distributions, then documenting the use of 'virtualenv' as a supported way to install it might reduce your support burden. I now realize why we don't understand each other - in my case, the one doing the installation is the user, who cannot be assumed to know much about python.q11 That's what I mean by application deployment vs webapp deployment. Ideally, the user has to enter one command/click one button, and he can use the application - he may not even be a programmer (I deploy things based on numpy/scipy for scientific people, who often have little patience for things which take more than 1 minute to set up software-wise). That's the user case I am mostly interested in - and I think it is quite general, and quite different from plone deployment kind of things. You can think of zc.buildout or the virtualenv-based script as a form of bundling, which bootstraps from another already-installed Python, but remains isolated from anything in its 'site-packages'. Yes, I know what virtualenv is, I use it sometimes - but it is definitely too complicated for the people I want to distribute my software to. I never even use that switch manually. zc.buildout does, but that is because it wants to control the PYTHONPATH by generating the script code: it doesn't ask users to tweak that. Well, that's the thing: neither do I :) but if my software is a dependency of another software, then I am bothered for problems with software which are not used at all by my own package (because setuptools makes an egg of my software automatically, screw things up, and I am the one blamed for it). I don't know why anybody who was not writing a packaging tool, or packaging a library for something like .deb / .rpm, would even use the multi-version option for setuptools: I don't see any sane way to install conflicting requirements into a shared 'site-packages'. But that's the problem: it often happens *even if you don't use setuptools yourself*. I would not be surprised if that's one reason why so many people have a seemingly unfair bias against setuptools. That's certainly the reason in my case. cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
David Cournapeau wrote: On Wed, Mar 25, 2009 at 1:45 AM, Toshio Kuratomi a.bad...@gmail.com wrote: David Cournapeau wrote: 2009/3/24 Toshio Kuratomi a.bad...@gmail.com: Steve Holden wrote: Seems to me that while all this is fine for developers and Python users it's completely unsatisfactory for people who just want to use Python applications. For them it's much easier if each application comes with all dependencies including the interpreter. This may seem wasteful, but it removes many of the version compatibility issues that otherwise bog things down. The upfront cost of bundling is lower but the maintenance cost is higher. For instance, OS vendors have developed many ways of being notified of and dealing with security issues. If there's a security issue with gtkmozdev and the python bindings to it have to be recompiled, OS vendors will be alerted to it and have the opportunity to release updates on zero day, the day that the security announcement goes out. I don't think bundling should be compared to depending on the system libraries, but as a lesser evil compared to requiring multiple, system-wide installed libraries. Well.. I'm not so sure it's even a win there. If the libraries are installed system-wide, at least the consumer of the application knows: 1) Where to find all the libraries to audit the versions when a security issue is announced. 2) That the library is unforked from upstream. 3) That all the consumers of the library version have a central location to collaborate on announcing fixes to the library. Yes, those are problems, but installing multi libraries have a lot of problems too: - quickly, by enabling multiple version installed, people become very sloppy to handle versions of the dependencies, and this increases a lot the number of libraries installed - so the advantages above for system-wide installation becomes intractable quite quickly This is somewhat true. Sloppiness and increased libraries are bad. But there are checks on this sloppiness. Distributions, for instance, are quite active about porting software to use only a subset of versions. So in the open source world, there's a large number of players interested in keeping the number of versions down. Using multiple libraries will point people at where work needs to be done whereas bundling hides it behind the monolithic bundle. - bundling also supports a real user-case which cannot be solved by rpm/deb AFAIK: installation without administration privileges. This is only sortof true. You can install rpms into a local directory without root privileges with a commandline switch. But rpm/deb are optimized for system administrators so the documentation on doing this is not well done. There can also be code issues with doing things this way but those issues can affect bundled apps as well. And finally, since rpm's primary use is installing systems, the toolset around it builds systems. So it's a lot easier to build a private root filesystem than it is to cherrypick a single package. It should be possible to create a tool that merges a system rpmdb and a user's local rpmdb using the existing API but I'm not aware of any applications built to do that yet. - multi-version installation give very fragile systems. That's actually my number one complain in python: setuptools has caused me numerous headache, and I got many bug reports because you often do not know why one version was loaded instead of another one. I won't argue for setuptools' implementation of multi-version. It sucks. But multi-version can be done well. Sonames in C libraries are a simple system that does this better. So I am not so convinced multiple-version is better than bundling - I can see how it sometimes can be, but I am not sure those are that important in practice. Bundling is always harmful. Whether multiple versioning is any better is certainly debatable :-) No. This is a social problem. Good source control only helps if I am tracking upstream's trunk so I'm aware of the direction that their changes are headed. But there's a wide range of reasons that application developers that bundle libraries don't do that: 1) not enough time in a day. I'm working full-time on making my application better. Plus I have to update all these bundled libraries from time to time, testing that the updates don't break anything. I don't have time to track trunk for all these libraries -- I barely have time to track releases. Yes, but in that case, there is nothing you can do. Putting everything in one project is always easier than splitting into modules, coding and deployment-wise. That's just one side of the speed of development vs maintenance issue IMHO. 3) This doesn't help with the fact that my bundled version of the library and your bundled version of the library are being developed in isolation from each other. This needs central coordination which people who believe bundling
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
But I don't think that a normal Lunix user should ever have to involve themselves directly with distutils or setuptools (it's OK to invoke them from the installation routine, but not to require the user to unpack the application then run setup.py). I completely agree. It would be best if the Linux distribution provided packages for most of the stuff (in RPM or deb format as appropriate), and volunteers provided a repository of packages in a format that the native package management tool actually understands (rather than PyPI - although it might be useful to have PyPI provide a Debian package repository for all projects that upload .deb files) I am afraid that distutils, and setuptools, are not really the answer to the problem, since while they may (as intended) guarantee that Python applications can be installed uniformly across different platforms they also more or less guarantee that Python applications are installed differently from all other applications on the platform. Ah, so we are clearly in agreement here. Mike Driscoll did some work providing Windows installers for various Python packages and extension modules, and people were amused that he provided executable installers for pure Python libraries. But I saw that as a sensible decision, since it meant that Windows users (and even Windows Python application developers) used the same mechanism to install everything. The Windows story is indeed sad, as none of the Windows packaging formats provides support for dependencies (MSI has some support, but as far as I understand it, it's pretty useless). But yes, for Windows, you want .exe or .msi installers, not something proprietary. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
If the problem is to get a recent enough version of the library, then the library would better be installed locally, for the application. If it is too much a problem because the application depends on billions of libraries which are 6 months old, the problem is to allow such a dependency in the first place. What kind of nightmare would it be if programs developed in C would required a C library which is 6 months old ? That's exactly what multiple-versions installations inflict on us. That's great for testing, development. But for deployment on end-user machines, the whole thing is a failure IMO. I see this as a consequence of Web 2.0, and open source: Publish early, publish often, make the pieces you publish as tiny as possible to maximize reuse :-( If people invent a new marshaling format every year (XML-RPC, SOAP, JSON, YAML, ...), you *have* to use a just-published library to follow the hype curve. If you have been raised in this tradition, you are shocked by the number of modules in the Python library, and wish for this dinosaur to die. Apparently, deployment to end user machines is not so much of an issue usually, since the mess will live on the web servers, and not on everybody's end user machine. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
This is only sortof true. You can install rpms into a local directory without root privileges with a commandline switch. But rpm/deb are optimized for system administrators so the documentation on doing this is not well done. There can also be code issues with doing things this way but those issues can affect bundled apps as well. And finally, since rpm's primary use is installing systems, the toolset around it builds systems. So it's a lot easier to build a private root filesystem than it is to cherrypick a single package. It should be possible to create a tool that merges a system rpmdb and a user's local rpmdb using the existing API but I'm not aware of any applications built to do that yet. Building private root kind of defeat the purpose :) Deploying linux packages in a reliable way without requiring admin privileges is an interesting experience. The tools just don't support this well - or there exists some magical tools that I've never seen mention of. I won't argue for setuptools' implementation of multi-version. It sucks. But multi-version can be done well. Sonames in C libraries are a simple system that does this better. I would say simplistic instead of simple :) what works for C won't necessarily work for python - and even in C, library versioning is not used that often except for a few core libraries. Library versioning works in C because C model is very simple. It already breaks for C++. More high-level languages like C# already have a more complicated scheme (GAC) - and my impression is that it did not work that well. The SxS for dll on recent windows to handle multiple version is a nightmare too in my (limited) experience. I'm confused -- if it will happen whatever tools are available, how does good source control solve the issue? I'm saying that this is not an issue that can be solved by having good source control... it's a social issue that has to be solved by people learning to avoid bad practices. I meant that whatever technology is available, bundling everything will always be easier. And sometimes, given the time/resource constraints, that's even the only realistic option. I could tell you many stories about wasted hours related to some fortran libraries which were hopelessly broken (missing symbols) on most distributions, or ABI conflicts - for which bundling sometimes is the only path to keep its sanity (short of giving up support for the platform). When you need to solve the problem now because you want to demo things tomorrow, not in 6 months, that's the only solution. It is not always bad practice. IMHO, one should focus on making it easier to avoid bundling everything - robust and simple dependency scheme, adapting distutils installation scheme to the various OS conventions, be it FHS, windows, etc... But we can't and shouldn't prevent it totally, and tools are already there to help minimizing the problems of bundling. For multiple system-wide libraries, I have yet to encounter anything which makes it somewhat reliable - it has only caused problems for me, and not solved any single problem. cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
David Cournapeau wrote:
On Wed, Mar 25, 2009 at 3:15 AM, Tres Seaver tsea...@palladion.com wrote:
Everytime I tried to understand what buildout was about, I was not
even sure it could help for my own problems at all. It seems very
specific to web development - I may completely miss the point ?
I think so: it is largely a way to get repeatable / scripted deployment
of software to disk. It uses setuptools to install Python package
distributions, but also can use other means (e.g, configure-make-make
install to install a C library such as libxml2). The end result is a
self-contained directory tree:
- - Scripts in the 'bin' directory are configured to have the specific
Python pacakges (and versions) they require on the PYTHONPATH.
- - By convention, released package distributions are installed into the
'eggs' subdirectory', which is *not* on the PYTHONPATH, nor is it a
'site' directory for Python.
- - Other bits are typically in their own subdirectories, often under
'parts'.
Ok - but I don't think it helps much, see below.
When not doing Plone / Zope-specific work (where zc.buildout is a de
facto standard), I use 'virtualenv' to create isolated environments into
which I install the libraries for a given application. If your
application ships as Python package distributions, then documenting the
use of 'virtualenv' as a supported way to install it might reduce your
support burden.
I now realize why we don't understand each other - in my case, the one
doing the installation is the user, who cannot be assumed to know much
about python.q11 That's what I mean by application deployment vs
webapp deployment. Ideally, the user has to enter one command/click
one button, and he can use the application - he may not even be a
programmer (I deploy things based on numpy/scipy for scientific
people, who often have little patience for things which take more than
1 minute to set up software-wise).
That's the user case I am mostly interested in - and I think it is
quite general, and quite different from plone deployment kind of
things.
OK, but I'm still not convinced that the Plone installers don't fit this
case. It's just that they also allow add-on installation afterward:
the main Plone application is installed via a GUI wizard (on Windows /
OSX).
You can think of zc.buildout or the virtualenv-based script as a form of
bundling, which bootstraps from another already-installed Python, but
remains isolated from anything in its 'site-packages'.
Yes, I know what virtualenv is, I use it sometimes - but it is
definitely too complicated for the people I want to distribute my
software to.
Have you tried using the script you can generate from it? E.g.,:
MY_ADDON_CODE = \
...
def after_install(options, home_dir):
subprocess.call([join(home_dir, 'bin', 'easy_install'),
'MyPackage'])
from virtualenv import create_bootstrap_script
f = open('/tmp/myinstaller.py', 'w')
f.write(create_bootstrap_script(MY_ADDON_CODE)
f.close()
Users would then be able to run that script and get a virtualenv, with
your 'MyPackage' distribution already installed in it.
I never even use that switch manually. zc.buildout does, but that is
because it wants to control the PYTHONPATH by generating the script
code: it doesn't ask users to tweak that.
Well, that's the thing: neither do I :) but if my software is a
dependency of another software, then I am bothered for problems with
software which are not used at all by my own package (because
setuptools makes an egg of my software automatically, screw things up,
and I am the one blamed for it).
Hmm, if your package uses pure distutils facilities, then it should be
installable by setuptools as an egg without problems. Why would your
users be trying to to '--multi-version' installs of your package?
I don't know why anybody who was not writing a packaging tool, or
packaging a library for something like .deb / .rpm, would even use the
multi-version option for setuptools: I don't see any sane way to
install conflicting requirements into a shared 'site-packages'.
But that's the problem: it often happens *even if you don't use
setuptools yourself*. I would not be surprised if that's one reason
why so many people have a seemingly unfair bias against setuptools.
That's certainly the reason in my case.
Again: I've *never* seen packages installed in multi-version fashion,
except in the very different mode (not in sys.path, nor in a directory
significant to site.py) used by zc.buildout.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis mar...@v.loewis.de writes: I see this as a consequence of Web 2.0, and open source: Publish early, publish often, make the pieces you publish as tiny as possible to maximize reuse :-( I'm all in favour of re-use. But re-use should mean having the library in a *single* predictable location (per declared version) on the system, so that any code that needs it can declare the dependency and find it there, and so that bugs in the library only need to be found and fixed in *one* place. If people invent a new marshaling format every year (XML-RPC, SOAP, JSON, YAML, ...), you *have* to use a just-published library to follow the hype curve. That's still no reason why that new version of the library can't be installed to a central, predictable location. If you have been raised in this tradition, you are shocked by the number of modules in the Python library, and wish for this dinosaur to die. It's more shocking to realise that there are even *more* libraries installed privately by applications, where they can't be easily found and fixed by a single update of the library. -- \ “The process by which banks create money is so simple that the | `\ mind is repelled.” —John Kenneth Galbraith, _Money: Whence It | _o__) Came, Where It Went_, 1975 | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On 25/03/2009 6:51 AM, Martin v. Löwis wrote: Mike Driscoll did some work providing Windows installers for various Python packages and extension modules, and people were amused that he provided executable installers for pure Python libraries. But I saw that as a sensible decision, since it meant that Windows users (and even Windows Python application developers) used the same mechanism to install everything. The Windows story is indeed sad, as none of the Windows packaging formats provides support for dependencies (MSI has some support, but as far as I understand it, it's pretty useless). But yes, for Windows, you want .exe or .msi installers, not something proprietary. Isn't this discussion slightly at cross-purposes? * py2exe doesn't create *installers*, just the target application. It (poorly) manages dependencies on the build machine. There doesn't seem to be a need for runtime dependency management in this tool. * The creation of an *installer* is something quite different. An installer for a py2exe based tool also doesn't need dependency management. An installer for a pure-python package that made no attempt to bundle dependencies might be nice, but I don't quite see how that falls outside the scope of distutils/setuptools/etc. In other words, I don't see why the installer can't bootstrap the 'normal' dependency management which would be used if the package was installed any other way or on other platforms. * distutils already has the ability to create Windows installer executables for pure-python apps/libs. I agree it would be nice if it was an MSI but that is an implementation detail rather than implementation requirement. How were Mike's packages fundamentally different than that? Cheers, Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
* The creation of an *installer* is something quite different. An installer for a py2exe based tool also doesn't need dependency management. Right. I wasn't really talking about py2exe (anymore), but about installers for libraries. An installer for a pure-python package that made no attempt to bundle dependencies might be nice, but I don't quite see how that falls outside the scope of distutils/setuptools/etc. In other words, I don't see why the installer can't bootstrap the 'normal' dependency management which would be used if the package was installed any other way or on other platforms. Perhaps that could be a solution. However, in package management systems that solve this properly, you also have proper uninstallation, which includes: - uninstallation is rejected if packages still depend on the to-be-removed package (or else offers to remove the relying packages as well) - uninstallation reference-counts, causing an automatically-installed package to be uninstalled if it is no longer needed, or else offers to compute-then-uninstall all packages which are no longer needed. The .exe/.msi installers do support uninstallation, but, alas, no dependency management. * distutils already has the ability to create Windows installer executables for pure-python apps/libs. I agree it would be nice if it was an MSI but that is an implementation detail rather than implementation requirement. Right - plus, distutils also supports creating MSIs. How were Mike's packages fundamentally different than that? Dunno. Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On 2009-03-24 23:47, Martin v. Löwis wrote: An installer for a pure-python package that made no attempt to bundle dependencies might be nice, but I don't quite see how that falls outside the scope of distutils/setuptools/etc. In other words, I don't see why the installer can't bootstrap the 'normal' dependency management which would be used if the package was installed any other way or on other platforms. Perhaps that could be a solution. However, in package management systems that solve this properly, you also have proper uninstallation, which includes: - uninstallation is rejected if packages still depend on the to-be-removed package (or else offers to remove the relying packages as well) - uninstallation reference-counts, causing an automatically-installed package to be uninstalled if it is no longer needed, or else offers to compute-then-uninstall all packages which are no longer needed. The .exe/.msi installers do support uninstallation, but, alas, no dependency management. Question is: who really needs such dependency management ? * It may be helpful to developers who wrap up 3rd party code for an application (e.g. Miro). * It may also help users that want to install a few plugins for an already installed application (e.g. Zope). * It will help users who use OSes that rely on software management tools to keep the initial distribution size small and prefer sharing over application isolation (e.g. Ubuntu). * It won't simplify things if such a system gets in the way of how the users or developer usually works or wants to work in a project. * It introduces dependencies on network resources that may potentially not be trusted. * If the package dependencies are not managed with lots of quality assurance, it can easily ruin your complete installation or simply prevent you from installing two sets of packages at the same time. There are both lots of reasons for wanting dependency checking and against such dependency checking. As a result, there is no definite answer as to whether it's good or bad and there is no single system that would satisfy all users/developers. Instead, there needs to be freedom of choice and distutils provides this freedom of choice by allowing you to ship .exes, .msi installers, binary drop-in ZIP archives, RPM packages, Debian packages, etc. etc. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 24 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: * The creation of an *installer* is something quite different. An installer for a py2exe based tool also doesn't need dependency management. Right. I wasn't really talking about py2exe (anymore), but about installers for libraries. An installer for a pure-python package that made no attempt to bundle dependencies might be nice, but I don't quite see how that falls outside the scope of distutils/setuptools/etc. In other words, I don't see why the installer can't bootstrap the 'normal' dependency management which would be used if the package was installed any other way or on other platforms. Perhaps that could be a solution. However, in package management systems that solve this properly, you also have proper uninstallation, which includes: - uninstallation is rejected if packages still depend on the to-be-removed package (or else offers to remove the relying packages as well) - uninstallation reference-counts, causing an automatically-installed package to be uninstalled if it is no longer needed, or else offers to compute-then-uninstall all packages which are no longer needed. The .exe/.msi installers do support uninstallation, but, alas, no dependency management. * distutils already has the ability to create Windows installer executables for pure-python apps/libs. I agree it would be nice if it was an MSI but that is an implementation detail rather than implementation requirement. Right - plus, distutils also supports creating MSIs. How were Mike's packages fundamentally different than that? Dunno. As far as memory serves, Mike built the installers precisely by using distutils to build Windows installers. He then had to suffer criticism from people who suggested this was inappropriately complex for pure Python modules. In so far as end users won't wish to install individual Python modules this argument may have had some merit, but I personally thought the criticism unjustified since Mike's technique gave a uniform install procedure for everything. I don't think anyone was suggesting that py2exe would be helpful for writing installers--simply to reduce complexity from the user's point of view and bundle dependencies in isolation from other applications. It seems to do that quite well. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tres Seaver wrote: Given that the out-of-the-box Python install already has facilities for retrieving text over the net and executing that text, the notion of locking down a machine to include only the bits installed in the stock Python install is just security theatre; such a machine shouldn't have Python installed at all (nor a C compiler, etc.) Indeed, in the real world this locking down is done at the firewall level. As for packaging in this scenario, that's what private package servers are for... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Lie Ryan wrote: Some companies have /very/ strict policies on running anything on live server, including scripts you write yourself. The problem is if the script goes awry, it might disturb the stability or even security of the server. Yes, we as a profession right software and have responsibilities. Get over it. It's what dev servers, uat, backups and dr are for... I see no relation between this and packaging other than that any packaging story needs to support privtae distribution servers. When the server administrator is already freaked out about adding an script developed by in-house employee, what about adding an external module? Then he's a muppet, plain and simple. If he's not, then he will have tested the whole setup before hand and got signoff from the developers and users who are responsbile for doing so. All of this has very little to do with packaging. Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Paul Moore wrote: I am not. What I *am* doing is saying (obliquely, I admit) is that for a package management system to be decent enough for stripping down the stdlib to not be an issue, it has to address these points (which clearly it can't). Sure it can, either by supporting offline bundles or by having sets of packages that are marked as Python Approved! or some such and so all have the same license. Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
R. David Murray wrote: I disagree. One of the jobs I've had is release management for internal software projects that depend on various external pieces. Release integration tested against specific versions of those external packages, and those were the packages that needed to wind up on the system when the release was installed. I've done systems depending on both perl and python, and let me tell you, python is way, _way_ easier to manage. With python, I have a dependency on a particular python version, and then maybe one or two add on packages. Well, python already has tools available to do exactly this.: buildout from a private egg repository will do exactly what you're after. However, its built on top of setuptools, which is flawed, and it's not blessed as official core python, so there's lots of room for improvement! Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: In some environments, each new component must be approved. Once python is approved, the standard library is OK, but adding 7 packages from pypi requires 7 more sets of approvals. True, but as I mentioend elsewhere, I myself haven't done a python project where I only needed python and the standard lib for many years... I was always able to get what I need through aptitude. Yes, well known for its ability to run on Windows and Mac OS... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Trent Mick wrote: Apologies for jumping in mid-thread here. FYI: We're (where we == ActiveState here) looking at spending more effort on Python of late. Some of our thoughts are on add modules: whether added to the ActivePython core or easily addable via an equivalent to ActivePerl's ppm (package manager) is still be batted around. Buildout is the closest thing python has to sanity at the moment. I'm curious as to people's thoughts. I'll also be at PyCon in Chicago getting thoughts. I'll be there, look forward to it. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
2009/3/23 Chris Withers ch...@simplistix.co.uk: Paul Moore wrote: I am not. What I *am* doing is saying (obliquely, I admit) is that for a package management system to be decent enough for stripping down the stdlib to not be an issue, it has to address these points (which clearly it can't). Sure it can, either by supporting offline bundles or by having sets of packages that are marked as Python Approved! or some such and so all have the same license. OK, I'll drop out of the discussion at this point. We clearly have such different experience that we aren't understanding each others' points - and the misunderstandings aren't moving the discussion forwards. Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Chris Withers chris at simplistix.co.uk writes: Well, python already has tools available to do exactly this.: buildout from a private egg repository will do exactly what you're after. However, its built on top of setuptools, which is flawed, and it's not blessed as official core python, so there's lots of room for improvement! Could you explain how buildout is an improvement over other systems? Its documentation seems full of generic wording (parts etc.) that I can't make sense of. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Antoine Pitrou wrote: Chris Withers chris at simplistix.co.uk writes: Well, python already has tools available to do exactly this.: buildout from a private egg repository will do exactly what you're after. However, its built on top of setuptools, which is flawed, and it's not blessed as official core python, so there's lots of room for improvement! Could you explain how buildout is an improvement over other systems? Its documentation seems full of generic wording (parts etc.) that I can't make sense of. It has a couple of differentiators from a stock distutils or setuptools-based installation: - Distributions are compiled and installed as eggs, but in a directory which is neither on the sys.path nor one of those marked as a 'site' directory. zc.buildout *does* use the dependency information, if present in setup.py, to fetch dependent distributions (like easy_install with the --multi-version option). - Scripts generated from the eggs get a generated prologue which sets up the sys.path expressing the requirements spelled out for that script. - It externalizes much of the how to build it information out of 'setup.py' into a separate INI-style configuration file, - It uses recipes as extensions, which enable a lot of tasks which are unsupported or poorly supported by distutils / setuptools (e.g., installing non-Python software using configure-make-make install, generating config files, etc.) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJx9+q+gerLs4ltQ4RAlheAJ0Wq98Q3+SwgeaZthN2FrVYLyle2wCg3NiG QX6ojgLlSmBJY1g48gMLimM= =pkMq -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
In some environments, each new component must be approved. Once python is approved, the standard library is OK, but adding 7 packages from pypi requires 7 more sets of approvals. True, but as I mentioend elsewhere, I myself haven't done a python project where I only needed python and the standard lib for many years... I was always able to get what I need through aptitude. Yes, well known for its ability to run on Windows and Mac OS... Is it? I had never heard before that aptitude works well on Windows (although it does work on OSX). In any case, I rarely need to install Python packages on Windows or Mac, so this causes no problem to me. If I need something on Windows, I usually download the Windows installer for it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: In some environments, each new component must be approved. Once python is approved, the standard library is OK, but adding 7 packages from pypi requires 7 more sets of approvals. True, but as I mentioend elsewhere, I myself haven't done a python project where I only needed python and the standard lib for many years... I was always able to get what I need through aptitude. Yes, well known for its ability to run on Windows and Mac OS... Is it? I had never heard before that aptitude works well on Windows (although it does work on OSX). I'm actually not quite sure if that was a sarcastic response to Chris's original sarcasm, or if the original sarcasm just didn't cross the language barrier very well... In any case, I rarely need to install Python packages on Windows or Mac, so this causes no problem to me. If I need something on Windows, I usually download the Windows installer for it. I believe that was Chris's point as well - unlike Linux, getting new packages directly from third parties rather than from a distro provided repository is still the norm on Windows and OS X. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
I was always able to get what I need through aptitude. Yes, well known for its ability to run on Windows and Mac OS... Is it? I had never heard before that aptitude works well on Windows (although it does work on OSX). I'm actually not quite sure if that was a sarcastic response to Chris's original sarcasm, or if the original sarcasm just didn't cross the language barrier very well... The former, in this case. I believe that was Chris's point as well - unlike Linux, getting new packages directly from third parties rather than from a distro provided repository is still the norm on Windows and OS X. Sure. However, I don't think that Chris' proposed solution (whatever that is) would improve my life - it would likely make things worse for me (already the introduction of eggs made the life worse for Debian package maintainers, at least initially - i.e. for a few years). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: In some environments, each new component must be approved. Once python is approved, the standard library is OK, but adding 7 packages from pypi requires 7 more sets of approvals. True, but as I mentioend elsewhere, I myself haven't done a python project where I only needed python and the standard lib for many years... I was always able to get what I need through aptitude. Yes, well known for its ability to run on Windows and Mac OS... Is it? I had never heard before that aptitude works well on Windows (although it does work on OSX). Hmm, sorry, let me clarify: sarcasm Yes, well known for its ability to run on Windows and Mac OS... /sarcasm ;-) aptitude also won't help when: - your customer is deploying onto managed machines running RHEL - debian has an outdated and/or broken version of your package. The latter is a standard problem with Zope/Apache/Postgres in debian and causes much gnashing of teeth by people trying to support them. The Debian guys love buggering around with other people's packaging, not caring that it makes supporting stuff so much harder. In any case, I rarely need to install Python packages on Windows or Mac, so this causes no problem to me. If I need something on Windows, I usually download the Windows installer for it. Yeah, 20 windows installers for different python packages whose versions may or may not conflict also seems a little silly ;-) A cross-plaftorm, platform agnostic, python-centric package management system is what's needed. Setuptools comes close, but I wonder if it's impossible to get it to do the last bits of what's needed (uninstall being the big one, and knowing what version of what package you have installed!) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Tue, Mar 24, 2009 at 6:47 AM, Martin v. Löwis mar...@v.loewis.de wrote: already the introduction of eggs made the life worse for Debian package maintainers, at least initially - i.e. for a few years. It still is, FWIW, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Chris Withers writes: aptitude also won't help when: - your customer is deploying onto managed machines running RHEL True. - debian has an outdated and/or broken version of your package. True, but just as for the package system you are advocating, it's quite easy to set up your apt to use third-party repositories of Debian-style packages. The question is whether those repositories exist. Introducing yet another, domain-specific package manager will make it less likely that they do, and it will cause more work for downstream distributors like Debian and RH. The latter is a standard problem with Zope/Apache/Postgres in debian and causes much gnashing of teeth by people trying to support them. The Debian guys love buggering around with other people's packaging, not caring that it makes supporting stuff so much harder. Well, I can't speak for the Debian Zope/Apache/Postgres maintainers, but I assure you the Emacs maintainers do care. But their hands are full trying to implement what you keep proposing as a solution, viz., a batteries-included distribution built on top of a package system. If you don't like Debian, fine, as an upstream vendor, I don't either. But an awful lot of my users *do* like[1] Debian (or Ubuntu or another Debian-derived distro). I see no alternative to cooperating with them (though I sometimes complain loudly throughout the processwink). If you want to see where the kind of thing you propose can lead, take a look at the Debian Emacs Policy document, which is designed to deal with one fork that takes the batteries-included approach, and another that has gone a long way in the direction of unbundling packages. Note that while Python doesn't have a political fork of the kind that Emacs does, it *does* have multiple blessed technical forks, and your suggestion involves the creation of yet more forks (ie, the distributions with bundled packages, versus those without). Whether the technical differences among Python implementations and packaging strategies will lead to the kind of issues that form the background of the Debian Emacs Policy, I don't know. But you don't know either. A cross-plaftorm, platform agnostic, python-centric package management system is what's needed. That's what you (think you[2]) need, but that statement rudely ignores the stated requirements of many other users. What you are doing here is divisive, not unifying. Setuptools comes close, but I wonder if it's impossible to get it to do the last bits of what's needed (uninstall being the big one, and knowing what version of what package you have installed!) Why wonder, when you can try an implementation and report the results? I guess you mean you hope somebody else will do the work (not only of development, but of maintaining the package repository). Well, somebody else has *already* done the work, but unfortunatelywink they defined the work in their own way. The result is the batteries-included stdlib. Footnotes: [1] For values of like including but not limited to see no superior alternative to. [2] If your users include Debian and RHEL users, you may find that they are not so happy with yet more PMS. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tres Seaver wrote: Steven D'Aprano wrote: Tres, for some reason every time you reply to the list, you send TWO copies instead of one: To: python-dev@python.org CC: Python Dev python-dev@python.org Could you please fix that? I can try: I normally post via gmane, and leave python-dev CC'ed so that folks who read via the list don't have their replies to me fall off list (which happens on some lists, anyway). I will trim the CC in the future, and hope for the best. That's what works best for me, anyway. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) I don't like the standard library to shrink. It's good that batteries are included. If a decent package management system *was* included, this wouldn't be an issue.. Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Steve Holden wrote: Perhaps we could encourage more jumbo distributions, like Enthought's and ActiveState's. I suspect many people would rather be able to maintain their Python functionality as a single product. I think you'll find it split.. People who use and love things like zc.buildout do so because they want to free package maintainers to do their own release cycles and not have individual packages held back by the need to release the whole project in one go. However, yes, I'm sure there are just as many people who want to install just one thing and have it all there. (although they'll be sadly disappointed when they realise whatever it is they need (lxml, PIL, xlrd,xlwt) isn't there. That said, a decent package management system in the core *and* the jumbo installers you mention would likely keep both camps happy ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Michael Foord wrote: I have mixed feelings. It is great that the batteries are included, but some batteries are showing their age or not maintained (who maintains IDLE? - does the calendar module really warrant being in the standard library? - imaplib is really not useful and IMAPClient which isn't in the standard library is much better [1]). Wow, interesting case in point ;-) I actually stumbled into imaplib, found it not useful and gave up and solved the problem a completely different way. Had I not tripped over it, I might have found IMAPClient! Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Jim Jewett wrote: - python to grow a decent, cross platform, package management system As stated, this may be impossible, because of the difference in what a package should mean on Windows vs Unix. If you just mean a way to add python packages from pypi as with EasyInstall, then maybe. I meant package in the python sense, which has a clear definition, unless I'm mistaken? In some environments, each new component must be approved. Once python is approved, the standard library is OK, but adding 7 packages from pypi requires 7 more sets of approvals. True, but as I mentioend elsewhere, I myself haven't done a python project where I only needed python and the standard lib for many years... That said, it may make sense to just give greater prominence to existing repackagers, such as ActiveState or Enthought. Right, getting ActivePython or similar approved might be the way to go, but I'm betting it depends on the project... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
2009/3/13 Chris Withers ch...@simplistix.co.uk: If a decent package management system *was* included, this wouldn't be an issue.. Remember that a decent package management system needs to handle filling in all the forms and arranging approvals to get authorisation for packages when you download them. And no, I'm *not* joking. People in a locked-down corporate environment really do benefit from just having to get the OK for Python, and then knowing that they have all they need. Even ignoring the above, your decent package management system has to deal with systems with no internet connectivity - just copying the Python installer onto my pen drive to put on my Mum's laptop so I can write some admin jobs for her, is a lot easier than having to pick and choose from PyPI what to download before I start. -1 on slimming down the stdlib. (OTOH, I've no problem with seeing an improved package system - just don't use it as an excuse to cripple the stdlib!) Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Fri, 13 Mar 2009 at 09:58, Chris Withers wrote: Martin v. L?wis wrote: In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) I don't like the standard library to shrink. It's good that batteries are included. If a decent package management system *was* included, this wouldn't be an issue.. I disagree. One of the jobs I've had is release management for internal software projects that depend on various external pieces. Release integration tested against specific versions of those external packages, and those were the packages that needed to wind up on the system when the release was installed. I've done systems depending on both perl and python, and let me tell you, python is way, _way_ easier to manage. With python, I have a dependency on a particular python version, and then maybe one or two add on packages. With perl, I have perl, and then I have a gadzillion cpan modules. I don't know how many a gadzillion is, because what I wound up doing was making a local copy of the cpan archive, checking that in to the repository, and writing up some scripts that made sure I pulled the actual install from my cpan snapshot and supported the developers in updating that snapshot when we were building a new version. (Nor was that the only problem with perlwhat idiot decided it was OK to interactively prompt for things during a batch install process?! And without providing any way to script the answers, at least that I could find!) So I'm +1 for keeping the Python stdlib as comprehensive as sensible. (Please note that last word...I've no objection to pruning things that are no longer serving a useful purpose, or that are better managed outside the core.) -- R. David Murray http://www.bitdance.com___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
R. David Murray wrote: On Fri, 13 Mar 2009 at 09:58, Chris Withers wrote: Martin v. L�wis wrote: In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) I don't like the standard library to shrink. It's good that batteries are included. If a decent package management system *was* included, this wouldn't be an issue.. I disagree. One of the jobs I've had is release management for internal software projects that depend on various external pieces. Release integration tested against specific versions of those external packages, and those were the packages that needed to wind up on the system when the release was installed. I've done systems depending on both perl and python, and let me tell you, python is way, _way_ easier to manage. With python, I have a dependency on a particular python version, and then maybe one or two add on packages. With perl, I have perl, and then I have a gadzillion cpan modules. I don't know how many a gadzillion is, because what I wound up doing was making a local copy of the cpan archive, checking that in to the repository, and writing up some scripts that made sure I pulled the actual install from my cpan snapshot and supported the developers in updating that snapshot when we were building a new version. (Nor was that the only problem with perlwhat idiot decided it was OK to interactively prompt for things during a batch install process?! And without providing any way to script the answers, at least that I could find!) So I'm +1 for keeping the Python stdlib as comprehensive as sensible. (Please note that last word...I've no objection to pruning things that are no longer serving a useful purpose, or that are better managed outside the core.) Just for clarity, when I said a jumbo distribution I meant one with all necessary libraries to run and support a specified set of python functionalities. The point is *not* to add other libraries (which invites version creep and undermines stability) but to have everything you need for a given (core plus) set of functionality. I believe my original message acknowledged that this wouldn't be everyone's cup of tea, but if a good* set of applications were analyzed and a distribution built to support just those (presumably Python) subsystems, you would have a good core that you can augment with the system-installed Python if you are so minded. A jumbo shouldn't try to replace the system-installed Python because hopefully different Python (jumbo) distributions would coexist on the same system, supporting those Python elements for which their configuration is acceptable. I would not expect core developers to have to give the jumbos much mind, except in so far as they might require support for (slightly?) different versions of Python. A 1.5.2 process can talk to a 3.1 process without any problems at all as long as the application protocol is unambiguous. Why this insistence on trying to do everything with a single interpreter? Sure, it might use more resources to have three different versions of PIL in use from three different jumbos, but let's cross that bridge when we come to it. Naturally, in Python there are already several environments that will compute the required library subset necessary to support an application, though at present they do it only across a single Python version and application. However, writing a simple GUI or command-line program to call the other Python modules will give you a single analyzable module tree. You don't even have to distribute the GUI if you don't want ... So I don't see jumbo as replacing batteries included. More like comes with 14v 300AH accumulator and support for domain name and email services or suitable for GeoDjango virtuals under VirtualBox with NAT addressing. For non-Python stuff (e.g. PostgreSQL, though SQLite is plenty good enough for experiments) I think the API has to be stable enough to accommodate some version variations. regards Steve * This is not a democracy: use your own prejudices to decide. -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul Moore wrote: 2009/3/13 Chris Withers ch...@simplistix.co.uk: If a decent package management system *was* included, this wouldn't be an issue.. Remember that a decent package management system needs to handle filling in all the forms and arranging approvals to get authorisation for packages when you download them. And no, I'm *not* joking. People in a locked-down corporate environment really do benefit from just having to get the OK for Python, and then knowing that they have all they need. You are plainly joking: nothing in Python should know or care about the various bureaucratic insanities in some workplaces. Given the *existing* stdlib and network connectivity, nothing any corporate security blackshirt can do will prevent an even moderately-motivated person from executing arbitrary code downloaded from elsewhere. In that case, what is the point in trying to help those who impose such craziness? Even ignoring the above, Which the language and library should do. your decent package management system has to deal with systems with no internet connectivity - just copying the Python installer onto my pen drive to put on my Mum's laptop so I can write some admin jobs for her, is a lot easier than having to pick and choose from PyPI what to download before I start. Nobody is arguing that there should be *no* batteries in the stdlib: we are talking about the ones which are leaking acid already, or which might get there soon due to lack of maintenance. -1 on slimming down the stdlib. (OTOH, I've no problem with seeing an improved package system - just don't use it as an excuse to cripple the stdlib!) Part of this discussion is about not *expanding* the stdlib: give a reasonable packaging story, leaving a given component out of the library is a defensible choice. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJumAJ+gerLs4ltQ4RApC7AKDVsIxfBlw6CWWLa+VhaASyDz+LFQCfQp5I yzrdYPo1FbXGAB90Ucf/Le8= =bCTx -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul Moore wrote: 2009/3/13 Chris Withers ch...@simplistix.co.uk: If a decent package management system *was* included, this wouldn't be an issue.. Remember that a decent package management system needs to handle filling in all the forms and arranging approvals to get authorisation for packages when you download them. And no, I'm *not* joking. People in a locked-down corporate environment really do benefit from just having to get the OK for Python, and then knowing that they have all they need. You are plainly joking: nothing in Python should know or care about the various bureaucratic insanities in some workplaces. Given the *existing* stdlib and network connectivity, nothing any corporate security blackshirt can do will prevent an even moderately-motivated person from executing arbitrary code downloaded from elsewhere. In that case, what is the point in trying to help those who impose such craziness? I (and most people, I presume) would not run arbitrary program downloaded from somewhere else on a corporate server that holds many important customer data even when there is no technical or even bureaucratic restriction, maybe I will sneak around on a workstation but definitely not on the server especially if I love my job and want to keep it (I'm a student though so that applies to me in the future). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lie Ryan wrote: Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul Moore wrote: 2009/3/13 Chris Withers ch...@simplistix.co.uk: If a decent package management system *was* included, this wouldn't be an issue.. Remember that a decent package management system needs to handle filling in all the forms and arranging approvals to get authorisation for packages when you download them. And no, I'm *not* joking. People in a locked-down corporate environment really do benefit from just having to get the OK for Python, and then knowing that they have all they need. You are plainly joking: nothing in Python should know or care about the various bureaucratic insanities in some workplaces. Given the *existing* stdlib and network connectivity, nothing any corporate security blackshirt can do will prevent an even moderately-motivated person from executing arbitrary code downloaded from elsewhere. In that case, what is the point in trying to help those who impose such craziness? I (and most people, I presume) would not run arbitrary program downloaded from somewhere else on a corporate server that holds many important customer data even when there is no technical or even bureaucratic restriction, maybe I will sneak around on a workstation but definitely not on the server especially if I love my job and want to keep it (I'm a student though so that applies to me in the future). I'm not arguing that employees should violate their employers' policies: I'm arguing that Python itself shouldn't try to cater to such policies. Note that I'm not talking about running code pushed on me by malware authors, either: I'm talking about ordinary software development activities like using a script from a cookbook, or using a well-tested and supported library, rather than NIH. Given that the out-of-the-box Python install already has facilities for retrieving text over the net and executing that text, the notion of locking down a machine to include only the bits installed in the stock Python install is just security theatre; such a machine shouldn't have Python installed at all (nor a C compiler, etc.) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJunUx+gerLs4ltQ4RAojAAKCdoliiVDoGoKzfGXNuQUZVmoPrhgCfXeSa pGCKI3wLt9W1A4ccnINSdLs= =3H9u -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lie Ryan wrote: Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paul Moore wrote: 2009/3/13 Chris Withers ch...@simplistix.co.uk: If a decent package management system *was* included, this wouldn't be an issue.. Remember that a decent package management system needs to handle filling in all the forms and arranging approvals to get authorisation for packages when you download them. And no, I'm *not* joking. People in a locked-down corporate environment really do benefit from just having to get the OK for Python, and then knowing that they have all they need. You are plainly joking: nothing in Python should know or care about the various bureaucratic insanities in some workplaces. Given the *existing* stdlib and network connectivity, nothing any corporate security blackshirt can do will prevent an even moderately-motivated person from executing arbitrary code downloaded from elsewhere. In that case, what is the point in trying to help those who impose such craziness? I (and most people, I presume) would not run arbitrary program downloaded from somewhere else on a corporate server that holds many important customer data even when there is no technical or even bureaucratic restriction, maybe I will sneak around on a workstation but definitely not on the server especially if I love my job and want to keep it (I'm a student though so that applies to me in the future). I'm not arguing that employees should violate their employers' policies: I'm arguing that Python itself shouldn't try to cater to such policies. Basically you're saying: Python is designed not to work on such environment. Note that I'm not talking about running code pushed on me by malware authors, either: I'm talking about ordinary software development activities like using a script from a cookbook, or using a well-tested and supported library, rather than NIH. Some companies have /very/ strict policies on running anything on live server, including scripts you write yourself. The problem is if the script goes awry, it might disturb the stability or even security of the server. Given that the out-of-the-box Python install already has facilities for retrieving text over the net and executing that text, the notion of locking down a machine to include only the bits installed in the stock Python install is just security theatre; such a machine shouldn't have Python installed at all (nor a C compiler, etc.) When the server administrator is already freaked out about adding an script developed by in-house employee, what about adding an external module? Of course all of this does not (usually) apply to regular workstation. A messed up workstation only means a reinstall, a messed up server may mean company reputation. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Lie Ryan wrote: Tres Seaver wrote: I'm not arguing that employees should violate their employers' policies: I'm arguing that Python itself shouldn't try to cater to such policies. Basically you're saying: Python is designed not to work on such environment. No, I'm saying that it isn't Python's responibility to enable that kind of policy. If it happens to be good *for Python* to have a a package installation / upgrade machinery (the real point of the discussion), then it will be up to the paranoid admin to figure out how to disable that feature: it isn't the problem of the Python developers. There are real costs to batteries included, especially for modules which don't get used much. One such cost is that an unused module tends to bitrot over time; another is that the presence of a module in the stdlib may shadow other, better modules / packages which are not in the stdlib. Those costs need to be balanced against the undoubted benefits, when making the choice to add or remove a module from the stdlib. Note that I'm not talking about running code pushed on me by malware authors, either: I'm talking about ordinary software development activities like using a script from a cookbook, or using a well-tested and supported library, rather than NIH. Some companies have /very/ strict policies on running anything on live server, including scripts you write yourself. The problem is if the script goes awry, it might disturb the stability or even security of the server. I understand that such policies exist, and why. However, I don't think their existence is a relevant constraint on the design or implementation of Python. Given that the out-of-the-box Python install already has facilities for retrieving text over the net and executing that text, the notion of locking down a machine to include only the bits installed in the stock Python install is just security theatre; such a machine shouldn't have Python installed at all (nor a C compiler, etc.) When the server administrator is already freaked out about adding an script developed by in-house employee, what about adding an external module? An admin who is that paranoid shouldn't be giving people shell access, either: given shell acccess, network connectivity, and the existing stdlib the admin's policy is unenforceable as a technical measure. Even if the user can't create a file anywhere on the filesystem, the interpreter prompt is enough to allow the user to evade the policy. Heck, the *bash* prompt is enough to wreck it, e.g. using here documents, even without network connectivity. As an aisde: anybody who is installing packages from PyPI on a production box (rather than using an index under their own control) isn't sufficiently paranoid: it can and does happen that people re-upload changed packages to PyPI without changing the version, for instance, not to mention removing older releases. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJuooU+gerLs4ltQ4RAsdnAKCSkKc94bHvHBIrILampl9+Mksz8wCeJSBe +Yl5HVmwQ6StGTcWNmDiSjE= =qGID -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
I don't like the standard library to shrink. It's good that batteries are included. If a decent package management system *was* included, this wouldn't be an issue.. You can prove anything with a false premise... I believe that a package management system that is decent cannot possibly be included in Python (IOW, any packaging system included in Python cannot be decent enough to allow removal of things from the standard library) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Chris Withers wrote: Jim Jewett wrote: - python to grow a decent, cross platform, package management system As stated, this may be impossible, because of the difference in what a package should mean on Windows vs Unix. If you just mean a way to add python packages from pypi as with EasyInstall, then maybe. I meant package in the python sense, which has a clear definition, unless I'm mistaken? Unfortunately, you are mistaken: the term package is highly confusing and ambiguous in the python sense. It could be a package-as-import-sees-it, or it could be a package-as-pypi-sees-it. For the latter, the distutils inventors tried to coin the term distribution, but that didn't quite catch on. It *is* the Python *Package* index, and people do often refer to the things it indexes as packages. In some environments, each new component must be approved. Once python is approved, the standard library is OK, but adding 7 packages from pypi requires 7 more sets of approvals. True, but as I mentioend elsewhere, I myself haven't done a python project where I only needed python and the standard lib for many years... I was always able to get what I need through aptitude. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Chris Withers wrote: That said, it may make sense to just give greater prominence to existing repackagers, such as ActiveState or Enthought. Right, getting ActivePython or similar approved might be the way to go, but I'm betting it depends on the project... Apologies for jumping in mid-thread here. FYI: We're (where we == ActiveState here) looking at spending more effort on Python of late. Some of our thoughts are on add modules: whether added to the ActivePython core or easily addable via an equivalent to ActivePerl's ppm (package manager) is still be batted around. I'm curious as to people's thoughts. I'll also be at PyCon in Chicago getting thoughts. Cheers, Trent -- Trent Mick trentm at activestate.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
2009/3/13 Tres Seaver tsea...@palladion.com: Paul Moore wrote: 2009/3/13 Chris Withers ch...@simplistix.co.uk: If a decent package management system *was* included, this wouldn't be an issue.. Remember that a decent package management system needs to handle filling in all the forms and arranging approvals to get authorisation for packages when you download them. And no, I'm *not* joking. People in a locked-down corporate environment really do benefit from just having to get the OK for Python, and then knowing that they have all they need. You are plainly joking: nothing in Python should know or care about the various bureaucratic insanities in some workplaces. I am not. What I *am* doing is saying (obliquely, I admit) is that for a package management system to be decent enough for stripping down the stdlib to not be an issue, it has to address these points (which clearly it can't). In other words, the problems inherent in restricting the stdlib aren't ones which are solvable by a package management system. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tres Seaver wrote: You are plainly joking: nothing in Python should know or care about the various bureaucratic insanities in some workplaces. Given the *existing* stdlib and network connectivity, nothing any corporate security blackshirt can do will prevent an even moderately-motivated person from executing arbitrary code downloaded from elsewhere. In that case, what is the point in trying to help those who impose such craziness? Network connectivity isn't a given, even today. So yes, there are environments that are secure (i.e. no network connectivity), and there are environments where developers are trusted (shock, horror) to actually follow company policy and get all licenses vetted by their Contracts group before installing downloaded software on company machines. Given that even some of the core developers work in environments like that, then yes, I believe Python can and should take reasonable steps to enable its use in such situations. And the most reasonably step Python can take on that front is to continue to provide a relatively powerful standard library *even if* a flexible and otherwise useful package management approach is added at some stage. If someone else decides to create a MinimalPython which consists solely of something like easy_install and whatever is needed to run it (i.e. the opposite of the fat bundles from folks like ActiveState and Enthought), then more power to them. But I don't believe the official releases from python.org should go that way. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nick Coghlan wrote: Tres Seaver wrote: You are plainly joking: nothing in Python should know or care about the various bureaucratic insanities in some workplaces. Given the *existing* stdlib and network connectivity, nothing any corporate security blackshirt can do will prevent an even moderately-motivated person from executing arbitrary code downloaded from elsewhere. In that case, what is the point in trying to help those who impose such craziness? Network connectivity isn't a given, even today. So yes, there are environments that are secure (i.e. no network connectivity), and there are environments where developers are trusted (shock, horror) to actually follow company policy and get all licenses vetted by their Contracts group before installing downloaded software on company machines. Given that even some of the core developers work in environments like that, then yes, I believe Python can and should take reasonable steps to enable its use in such situations. And the most reasonably step Python can take on that front is to continue to provide a relatively powerful standard library *even if* a flexible and otherwise useful package management approach is added at some stage. My inclination would be to leave the stdlib largely as is, except that occostonally I would argue for ripping out a particular obsolete / bitrotted module. A couple of other points: - - Absent a sufficiently powerful package management system, the pressure to add modules to the stdlib (or keep them) is higher because it is harder for *all* Python users to add them, or replace them if dropped. - - The choice to add or remove a module to / from the stdlib should be made on the merits of the module, without regard to the kind of specialized deployment policies you outline. - - Changing the stdlib in a new release of Python is probably irrelevant for the kind of environments you allude to, as there is likely as much review involved to approve a new version of Python as there was in approving it in the first place: of the few I know of today, all are still running Python 2.1.x and / or 2.2.x for this reason. If someone else decides to create a MinimalPython which consists solely of something like easy_install and whatever is needed to run it (i.e. the opposite of the fat bundles from folks like ActiveState and Enthought), then more power to them. But I don't believe the official releases from python.org should go that way. Note that I am *not* advocating scrubbing / exploding the stdlib. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJuy9Y+gerLs4ltQ4RAranAJ4rCXgq0opHPki6OmlABbaqE3D1sQCeJ7Zt Em6VMK1u+6+xYsoqixwfoJ4= =YzN7 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Tres, for some reason every time you reply to the list, you send TWO copies instead of one: To: python-dev@python.org CC: Python Dev python-dev@python.org Could you please fix that? -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Steven D'Aprano wrote: Tres, for some reason every time you reply to the list, you send TWO copies instead of one: To: python-dev@python.org CC: Python Dev python-dev@python.org Could you please fix that? I can try: I normally post via gmane, and leave python-dev CC'ed so that folks who read via the list don't have their replies to me fall off list (which happens on some lists, anyway). I will trim the CC in the future, and hope for the best. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJuzWG+gerLs4ltQ4RAsQfAKCBGfeI6FEP8cNbOdh0zFhLjj65CgCgiLZb 725QgMYFCyhdM6OP5+SC04U= =yRwI -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
I'd like to suggest that any new candidate for the standard library be discussed and then set aside for a cooling off period of ONE YEAR. If then folks can all agree the library is not only Goodness, but of general interest, especially for bootstrapping small projects, then take a vote, or the BDFL can decide. A key criteria should be, Will the new library help small projects get started by providing basic capabilities without introducing a steep learning curve? These are all thoughts that I can sympathize with. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Thu, Mar 05, 2009 at 01:30:25PM -0500, Barry Warsaw wrote: Ubuntu (and probably Debian): apt-get install python-lxml Tested in Debian: yes, the incantation works. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/p...@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Guido van Rossum wrote: Based on the sad example of BerkeleyDB, which was initially welcomed into the stdlib but more recently booted out for reasons having to do with the release cycle of the external dependency and other issues typical for large external dependencies, I think we should be very careful with including it in the standard library. Yes. My experience of these kinds of libraries (bdb, lxml, etc) is that having them in the Python stdlib is a bad thing. Why? Because python (quite rightly, as I'm being convinced!) has a very conservative policy of changes is 3rd point releases. This, however, means you end up getting 'stuck with a release of something like lxml that you can't upgrade to get new features because you, say, use a debian-packages python which only upgrades when Debian next decide to do release... In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) I don't like the standard library to shrink. It's good that batteries are included. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) I don't like the standard library to shrink. It's good that batteries are included. I have mixed feelings. It is great that the batteries are included, but some batteries are showing their age or not maintained (who maintains IDLE? - does the calendar module really warrant being in the standard library? - imaplib is really not useful and IMAPClient which isn't in the standard library is much better [1]). If a library is well maintained then there seems to be little point in moving it into the standard library as it may actually be harder to maintain, and if a library has no active developers then do we really want it in the standard library... On the other hand there are some standard tools that a significant portion of the community use (Python Imaging Library and the PyWin32 extensions for example) that aren't in the standard library. I think other developers have similar mixed feelings, or at least there are enough people on both sides of the fence that it is very hard to make changes. Perhaps this is the way it should be. Michael [1] http://freshfoo.com/wiki/CodeIndex Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
I have mixed feelings. It is great that the batteries are included, but some batteries are showing their age or not maintained (who maintains IDLE? - does the calendar module really warrant being in the standard library? - imaplib is really not useful and IMAPClient which isn't in the standard library is much better [1]). I certainly agree that little-used modules should be removed (by means of a proper deprecation procedure). I do think it is fairly important that IDLE remains included (whether or not gpolo takes ownership of it). As for imaplib: I can't comment on the library itself, since I never use it. However, given the number of bug reports that we receive, it seems there is a fairly significant followership of it. If a library is well maintained then there seems to be little point in moving it into the standard library as it may actually be harder to maintain It depends. For quickly evolving libraries, it might be harder to maintain them in the core, as you can't release as quickly as you desire. In other cases, it simplifies maintenance: whenever a systematic API change is added, all standard library modules get typically updated by whoever makes the systematic change. That is more productive than having each individual maintainer to figure out what to change in response. However, I don't think of the maintainer point of view that much: it's rather the end users (i.e. application developers) who I worry about: Should we remove regular expressions from Python, just because the library doing it is unmaintained? On the other hand there are some standard tools that a significant portion of the community use (Python Imaging Library and the PyWin32 extensions for example) that aren't in the standard library. I continue having the same position: if the authors of those respective libraries would like to contribute it to the core (and eventually drop the out-of-core fork), then I would be happy to let them do that. Of course, Guido's cautioning wrt. external libraries that those depend on still applies: if there are strong external dependencies, the library would have to be really important to the community to still include it. I think other developers have similar mixed feelings, or at least there are enough people on both sides of the fence that it is very hard to make changes. Perhaps this is the way it should be. I think so, yes. Decisions will be made on a case-by-case basis, going either direction one time or the other. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis wrote: In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) I don't like the standard library to shrink. It's good that batteries are included. Perhaps we could encourage more jumbo distributions, like Enthought's and ActiveState's. I suspect many people would rather be able to maintain their Python functionality as a single product. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Perhaps we could encourage more jumbo distributions, like Enthought's and ActiveState's. I suspect many people would rather be able to maintain their Python functionality as a single product. I think the concept of jumbo distribution has been lost. I mentioned it to one of the Enthought people (sorry, forgot who exactly), and he said he had never heard the term. Looking back, it seems that you have to be a commercial enterprise to produce such a thing. There is the python.org distribution, where many volunteers maintain it, and then there are the two (?) free-commercial distributions (ActivePython, and EPD). I'm skeptical that there can be motivation for creating another community jumbo distribution - why would anybody put efforts into maintaining it? You don't get much credit for release engineering - except from fellow release engineers. In addition, you have the Linux distributions, which you can also count as jumbo Python distributions (and also jumbo Perl, jumbo Java, jumbo LISP - at least for Debian :-). Again, many of these are commercially based, although there still seems to be space for multiple community distributions (Debian, Gentoo). This is precisely the reason why I want Python to continue including its batteries. If we give that up, it will not come back, and users get frustrated that they have to collect stuff from many places, and that the stuff doesn't fit together completely. What that means for BeautifulSoup, I don't know. First, its authors would have to offer contributing it, and then experts should also agree that this would be a useful inclusion. Some apparently say that html5lib would be a better choice. If that's the thing that is currently on release 0.11, then I think we should take no action at this point - I don't want to include anything that has version 0.11. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis mar...@v.loewis.de wrote: If a library is well maintained then there seems to be little point in moving it into the standard library as it may actually be harder to maintain It depends. For quickly evolving libraries, it might be harder to maintain them in the core, as you can't release as quickly as you desire. In other cases, it simplifies maintenance: whenever a systematic API change is added, all standard library modules get typically updated by whoever makes the systematic change. That is more productive than having each individual maintainer to figure out what to change in response. This is a complicated issue. But two sub-threads seem to be about (1) modules dependent (or wrapping) a large, complicated third-party library under active development, and (2) hard-to-routinely-test modules, like imaplib. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Martin v. Löwis mar...@v.loewis.de wrote: then there are the two (?) free-commercial distributions (ActivePython, and EPD). Apple's system Python seems to qualify; it comes with a huge additional library, including numpy and Twisted. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Michael Foord wrote: Chris Withers wrote: ... love to see ... but ... not optimistic - python to grow a decent, cross platform, package management system As stated, this may be impossible, because of the difference in what a package should mean on Windows vs Unix. If you just mean a way to add python packages from pypi as with EasyInstall, then maybe. - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included In some environments, each new component must be approved. Once python is approved, the standard library is OK, but adding 7 packages from pypi requires 7 more sets of approvals. On the other hand, if there were a way to say The PSF explicitly endorses Packages X, Y, and Z as worthy of the stdlib; they are distributed separately for administrative reasons, then the initial request could be for Python plus officially endorsed addons That said, it may make sense to just give greater prominence to existing repackagers, such as ActiveState or Enthought. If a library is well maintained then there seems to be little point in moving it into the standard library The official endorsement is in many cases more important than shared distribution. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Jim Jewett wrote: That said, it may make sense to just give greater prominence to existing repackagers, such as ActiveState or Enthought. Perhaps we should think about linking to the relevant ActiveState/Enthought distributions from the pydotorg download pages? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
In light of this, what I'd love to see (but sadly can't really help with, and am not optimistic about happening) is for: - python to grow a decent, cross platform, package management system - the standard library to actually shrink to a point where only libraries that are not released elsewhere are included I'd be interested to know how many users of python also felt this way ;-) I don't like the standard library to shrink. It's good that batteries are included. I have mixed feelings. It is great that the batteries are included, but some batteries are showing their age or not maintained (who maintains IDLE? - does the calendar module really warrant being in the standard library? - imaplib is really not useful and IMAPClient which isn't in the standard library is much better [1]). If a library is well maintained then there seems to be little point in moving it into the standard library as it may actually be harder to maintain, and if a library has no active developers then do we really want it in the standard library... On the other hand there are some standard tools that a significant portion of the community use (Python Imaging Library and the PyWin32 extensions for example) that aren't in the standard library. I think other developers have similar mixed feelings, or at least there are enough people on both sides of the fence that it is very hard to make changes. Perhaps this is the way it should be. Batteries Included should not be interpreted as Soup and Nuts or the Kitchen Sink. There are a lot of good, no, outstanding libraries out there that are excellent candidates for the standard library. IMO, the standard library should include those things that would be truly helpful getting small projects started. Beyond that the needs of the project will likely be specialized to the point that one should be using libraries beyond the standard libraries. Heck, there are things I use frequently that I'd like to be standard, but could be an imposition on the rest of the Python community. The last thing I want to see is a library so large that we spend time figuring out how to cull unwanted items. Beautiful Soup and html5lib are good, but specialized libraries, and not, IMO, of *general* interest. The same goes for lxml, which by the way, I like and have used. ...but it doesn't have to be in the standard library to be useful. I'd like to suggest that any new candidate for the standard library be discussed and then set aside for a cooling off period of ONE YEAR. If then folks can all agree the library is not only Goodness, but of general interest, especially for bootstrapping small projects, then take a vote, or the BDFL can decide. A key criteria should be, Will the new library help small projects get started by providing basic capabilities without introducing a steep learning curve? Larry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
This is precisely the reason why I want Python to continue including its batteries. If we give that up, it will not come back, and users get frustrated that they have to collect stuff from many places, and that the stuff doesn't fit together completely. I concur. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Ivan Krstić wrote: On Mar 4, 2009, at 12:32 PM, James Y Knight wrote: I think html5lib would be a better candidate for an imrpoved HTML parser in the stdlib than BeautifulSoup. While we're talking about alternatives, Ian Bicking appears to swear by lxml: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ I second that. ;) And, BTW, I wouldn't mind getting lxml into the stdlib either. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Thu, Mar 5, 2009 at 2:39 AM, Stefan Behnel stefan...@behnel.de wrote: Ivan Krstić wrote: On Mar 4, 2009, at 12:32 PM, James Y Knight wrote: I think html5lib would be a better candidate for an imrpoved HTML parser in the stdlib than BeautifulSoup. While we're talking about alternatives, Ian Bicking appears to swear by lxml: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ I second that. ;) And, BTW, I wouldn't mind getting lxml into the stdlib either. No matter how beautiful and fast lxml is, it has one downside where it comes to installing it into the stdlib: it is based on large, complex 3rd party libraries, libxml2 and libxslt. Based on the sad example of BerkeleyDB, which was initially welcomed into the stdlib but more recently booted out for reasons having to do with the release cycle of the external dependency and other issues typical for large external dependencies, I think we should be very careful with including it in the standard library. Instead, let's hope Linux distros pick it up (and if anyone knows how to encourage that, let us know). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mar 5, 2009, at 12:32 PM, Guido van Rossum wrote: Instead, let's hope Linux distros pick it up (and if anyone knows how to encourage that, let us know). Gentoo: emerge lxml Ubuntu (and probably Debian): apt-get install python-lxml Guido, do you know where your time machine keys are? :) Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSbAaRHEjvBPtnXfVAQK3nQP/Uz6CF7zxIbTJWHyGyPBr1+pUTESzryNs SKnBwcyIjYw/+7whtdfp31jbgsv+FcZ9YmMx7haUzPS/lKaRClvfUlirXepDCQt/ Z44nxvjEbbpQPmvlmf9SAIgvk7AumWcigXth2cvMJedHz0fVA9jXA1f/bnGxdTA6 /DUrqxruwZo= =R5FW -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
2009/3/5 Guido van Rossum gu...@python.org: On Thu, Mar 5, 2009 at 2:39 AM, Stefan Behnel stefan...@behnel.de wrote: And, BTW, I wouldn't mind getting lxml into the stdlib either. No matter how beautiful and fast lxml is, it has one downside where it comes to installing it into the stdlib: it is based on large, complex 3rd party libraries, libxml2 and libxslt. And it depends on Cython, which is wonderful normally, but maybe difficult to deal with in language evolution since we wouldn't have direct control over the C sources. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
Vaibhav Mallya wrote: We do have HTMLParser, but that doesn't handle malformed pages well, and just isn't as nice as BeautifulSoup. Interesting, given that BeautifulSoup is built on HTMLParser ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Mar 4, 2009, at 9:56 AM, Chris Withers wrote: Vaibhav Mallya wrote: We do have HTMLParser, but that doesn't handle malformed pages well, and just isn't as nice as BeautifulSoup. Interesting, given that BeautifulSoup is built on HTMLParser ;-) I think html5lib would be a better candidate for an imrpoved HTML parser in the stdlib than BeautifulSoup. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
At 2:56 PM + 3/4/09, Chris Withers wrote: Vaibhav Mallya wrote: We do have HTMLParser, but that doesn't handle malformed pages well, and just isn't as nice as BeautifulSoup. Interesting, given that BeautifulSoup is built on HTMLParser ;-) In BeautifulSoup = 3.1, yes. Before that (= 3.07a), it was based on the more robust sgmllib.SGMLParser. The current BeautifulSoup can't handle 'foo a=bcb=cd', while the earlier SGMLParser versions can. I don't know quite how common that missing space is in the wild, but I've personally made HTML with that problem. Maybe this is the only problem with using HTMLParser instead of SGMLParser; I don't know. In the mean time, if I have a need for BeautifulSoup in Python3.x, I'll port sgmllib and use the older BeautifulSoup. -- TonyN.:' mailto:tonynel...@georgeanelson.com ' http://www.georgeanelson.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
James Y Knight schrieb: On Mar 4, 2009, at 9:56 AM, Chris Withers wrote: Vaibhav Mallya wrote: We do have HTMLParser, but that doesn't handle malformed pages well, and just isn't as nice as BeautifulSoup. Interesting, given that BeautifulSoup is built on HTMLParser ;-) I think html5lib would be a better candidate for an imrpoved HTML parser in the stdlib than BeautifulSoup. I second that. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Mar 4, 2009, at 12:32 PM, James Y Knight wrote: I think html5lib would be a better candidate for an imrpoved HTML parser in the stdlib than BeautifulSoup. While we're talking about alternatives, Ian Bicking appears to swear by lxml: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ Cheers, -- Ivan Krstić krs...@solarsail.hcs.harvard.edu | http://radian.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
