Re: dh_python and python policy analysis
Hi, This is version 1.05 of the draft, now with typo fixes, and some initial policy support for partial upgrades for pure python public modules that are trying to drop support for older versions of python. The idea is that error cases are minimized if we do not drop a version of python in a new stable that was the default version in oldstable; at the same time insisting that every pre python public module that says all in the versions supported field must support all shipping versions of python in stable (even if this takes extra effort in the form of splitting off private modules and including them conditionally based on python version from a wrapper module that does little but conditional imports. The current version, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ manoj Packaging with the new Python policy A package developers view Manoj Srivastava Developer The Debian Project Copyright (c) 2006 Manoj Srivastava Revision History Revision 1.0.5 4^th November 2006 Revision 1.0.4 12 Aug 2006 Revision 1.0.3 10 Aug 2006 Revision 1.0.2 8 Aug 2006 Revision 1.0.1 07 Aug 2006 Revision 1.0.0 31 Jul 2006 Specification of the new Python policy. This article grew as an attempt to correct a gap in the concrete specification of the new Python policy, and has grown to be close to a formal specification of the proposed new policy. -- Table of Contents 1. [1]Introduction 1.1. [2]Acknowledgements 2. [3]Goals of the new Python policy 3. [4]Nomenclature and definitions 3.1. [5]Python versions 3.1.1. [6]The Default Python version 3.2. [7]Categorization of Python software 4. [8]General Notes 4.1. [9]Naming python module packages 4.2. [10]Python versions supported by the source 4.3. [11]Byte compilation 4.4. [12]Linking extention modules 4.5. [13]Depends: 4.6. [14]Provides 4.7. [15]Build-Depends: 4.8. [16]Deprecating current in versions supported 5. [17]Recipe for developers 5.1. [18]Script 5.1.1. [19]Supported versions 5.2. [20]Private Pure Python Modules 5.2.1. [21]Byte compilation 5.2.2. [22]Supported versions 5.3. [23]Private Extension 5.3.1. [24]Supported versions 5.4. [25]Public pure Python Module 5.4.1. [26]Byte compiling 5.4.2. [27]Supported versions 5.4.3. [28]Depends: 5.5. [29]Public Extension 5.5.1. [30]Supported versions 5.5.2. [31]Provides 6. [32]Changing the default Python version 6.1. [33]Python rtupdate scripts 6.1.1. [34]rtupdate script invocation 1. Introduction While trying to update SELinux packages, I ran across problems in trying to determine if my packages were complying with the new python policy: any practical tips for packaging generally devolved to the statement Oh, just run dh_python. This is my attempt to offer more concrete tips for packaging. While this document started by reverse engineering dh_python, it has, thanks to help from various people more knowledgeable about Python than I, moved beyond that, and is closer to being a specification unfettered by the idiosyncrasies of current tools and implementations. So now this document is a draft formal specification of the proposed new Python packaging policy. While it draws upon earlier documents, notable [35]Debian Python Policy , and the [36]new policy Wiki, the [37]Debian Python FAQ, and the source code for dh_python, and debhelper scripts, it has essentially been written from scratch, with material reworded, reorganized, and rearranged, to the extent that it bears little resemblance to the original sources. Compiled Python modules are very dependent on the specific Python version they were compiled against, and may otherwise have restrictions on the versions of Python they are compatible with. Unless care is taken, introducing, or dropping, new versions of Python, or changing the default version, trigger long and often painful transitions where the archive is inconsistent, and the systems is ill-integrated for the duration. This new Python policy seeks to address these potential messy transitions, and minimize the pain.
Re: dh_python and python policy analysis
Manoj Srivastava [EMAIL PROTECTED] writes: Copyright (c) 2006 Manoj Srivastava Revision History Revision 1.0.5 4^th November 2006 Setember? -- O T A V I OS A L V A D O R - E-mail: [EMAIL PROTECTED] UIN: 5906116 GNU/Linux User: 239058 GPG ID: 49A5F855 Home Page: http://www.freedom.ind.br/otavio - Microsoft gives you Windows ... Linux gives you the whole house. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le samedi 26 août 2006 à 00:16 +0200, Mike Hommey a écrit : You can put them wherever you want as long as this complies with the FHS and that a set of .rtupdate/.rtinstall/.rtremove scripts take care of them. This is one of the only things that are currently standardized. Wasn't it possible to have only one implementation of these provided by python itself, the policy relying on those and only one location for python modules ? This was the original plan proposed back in January. Until the python maintainer forced its cancellation (#347758), then came up with a broken, incompatible implementation. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: dh_python and python policy analysis
On Sat, Aug 26, 2006 at 11:21:26AM +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Le samedi 26 août 2006 à 00:16 +0200, Mike Hommey a écrit : You can put them wherever you want as long as this complies with the FHS and that a set of .rtupdate/.rtinstall/.rtremove scripts take care of them. This is one of the only things that are currently standardized. Wasn't it possible to have only one implementation of these provided by python itself, the policy relying on those and only one location for python modules ? This was the original plan proposed back in January. Until the python maintainer forced its cancellation (#347758), then came up with a broken, incompatible implementation. Great Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Fri, Aug 25, 2006 at 11:34:51PM +0200, Josselin Mouette wrote: Le vendredi 25 août 2006 à 23:11 +0200, Mike Hommey a écrit : The problem with the python policy is that there is no policy as to where the modules are supposed to be installed. Depending on the tool you are using (python-support or python-central), the directory is different. Where is someone using none of these supposed to put them ? You can put them wherever you want as long as this complies with the FHS and that a set of .rtupdate/.rtinstall/.rtremove scripts take care of them. This is one of the only things that are currently standardized. So why not put that in policy, then? -- Lo-lan-do Home is where you have to wash the dishes. -- #debian-devel, Freenode, 2004-09-22 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le samedi 26 août 2006 à 08:43 +0200, Wouter Verhelst a écrit : You can put them wherever you want as long as this complies with the FHS and that a set of .rtupdate/.rtinstall/.rtremove scripts take care of them. This is one of the only things that are currently standardized. So why not put that in policy, then? It is: http://www.golden-gryphon.com/software/manoj-policy/x673.html -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: dh_python and python policy analysis
On Thu, Aug 24, 2006 at 11:46:25PM +0200, Josselin Mouette wrote: Let me rephrase it: the internals of python-support, and how it helps implementing the python policy, are developed in the python-support documentation. They don't need to be part of the policy Yes they do. and they have nothing to do with debhelper either. That's good, but not enough. This has now been going on for long enough that I conclude that the Python policy pushers really do intend to make using debhelper a Policy requirement for any package containing any Python code. I can't speak for others, but python-support provides pysupport-movemodules and pysupport-parseversions to separate the debhelper snippet from the actual abstraction code. That is still not what is required. Unless these tools become part of the dpkg-dev package, it should be documented in policy how they (are supposed to) do their job. Sure, you're free to seriously discourage people not to use these tools; but the assumption that it is possible to create a Debian package by just copying the right files to the right place and calling 'dpkg -b' on that has always been true. It would be wrong to change this assumption. (BTW, for a similar problematic that involves more than a hundred packages, nobody ever asked me how to make a package using GConf without using dh_gconf. Which means the GConf policy has never been written out but is currently defined by the dh_gconf behavior.) The mere fact that a given bug exists somewhere else, too, does not make it less of a bug. -- Lo-lan-do Home is where you have to wash the dishes. -- #debian-devel, Freenode, 2004-09-22 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le vendredi 25 août 2006 à 13:01 +0200, Wouter Verhelst a écrit : I can't speak for others, but python-support provides pysupport-movemodules and pysupport-parseversions to separate the debhelper snippet from the actual abstraction code. That is still not what is required. Unless these tools become part of the dpkg-dev package, it should be documented in policy how they (are supposed to) do their job. If you want to stick a policy label on the python-support documentation, that's fine with me, but this is the least of my concerns. The important point is for this documentation to exist. Also, my concern is to make the developer's life easier. I don't think writing gazillions of policies will help the developer. We have first to provide tools that implement this policy, after which we have plenty of time to write up formal documents. The whole point of free software is to re-use other people's tools and code. (BTW, for a similar problematic that involves more than a hundred packages, nobody ever asked me how to make a package using GConf without using dh_gconf. Which means the GConf policy has never been written out but is currently defined by the dh_gconf behavior.) The mere fact that a given bug exists somewhere else, too, does not make it less of a bug. The point is, it may be a bug, but do we have to fix a bug that nobody cares enough about to even *ask* about it? Granted, this is a much simpler case, but it shows that many developers don't care what a tool does if it works properly. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: dh_python and python policy analysis
On Fri, Aug 25, 2006 at 10:22:10PM +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Le vendredi 25 août 2006 à 13:01 +0200, Wouter Verhelst a écrit : I can't speak for others, but python-support provides pysupport-movemodules and pysupport-parseversions to separate the debhelper snippet from the actual abstraction code. That is still not what is required. Unless these tools become part of the dpkg-dev package, it should be documented in policy how they (are supposed to) do their job. If you want to stick a policy label on the python-support documentation, that's fine with me, but this is the least of my concerns. The important point is for this documentation to exist. Also, my concern is to make the developer's life easier. I don't think writing gazillions of policies will help the developer. We have first to provide tools that implement this policy, after which we have plenty of time to write up formal documents. The whole point of free software is to re-use other people's tools and code. The problem with the python policy is that there is no policy as to where the modules are supposed to be installed. Depending on the tool you are using (python-support or python-central), the directory is different. Where is someone using none of these supposed to put them ? Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le vendredi 25 août 2006 à 23:11 +0200, Mike Hommey a écrit : The problem with the python policy is that there is no policy as to where the modules are supposed to be installed. Depending on the tool you are using (python-support or python-central), the directory is different. Where is someone using none of these supposed to put them ? You can put them wherever you want as long as this complies with the FHS and that a set of .rtupdate/.rtinstall/.rtremove scripts take care of them. This is one of the only things that are currently standardized. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: dh_python and python policy analysis
On Fri, Aug 25, 2006 at 11:34:51PM +0200, Josselin Mouette [EMAIL PROTECTED] wrote: Le vendredi 25 août 2006 à 23:11 +0200, Mike Hommey a écrit : The problem with the python policy is that there is no policy as to where the modules are supposed to be installed. Depending on the tool you are using (python-support or python-central), the directory is different. Where is someone using none of these supposed to put them ? You can put them wherever you want as long as this complies with the FHS and that a set of .rtupdate/.rtinstall/.rtremove scripts take care of them. This is one of the only things that are currently standardized. Wasn't it possible to have only one implementation of these provided by python itself, the policy relying on those and only one location for python modules ? Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le mercredi 23 août 2006 à 15:39 +0300, Lars Wirzenius a écrit : The location is specific to the packaging tool and shouldn't be mentioned in the policy. Sure, that's fine: no need to mention it in policy. What was said earlier in the thread was that the locations should not be referenced in debian/rules, either, and that makes things difficult. This is the whole purpose of the dh_* snippets. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom
Re: dh_python and python policy analysis
to, 2006-08-24 kello 16:25 +0200, Josselin Mouette kirjoitti: Le mercredi 23 août 2006 à 15:39 +0300, Lars Wirzenius a écrit : The location is specific to the packaging tool and shouldn't be mentioned in the policy. Sure, that's fine: no need to mention it in policy. What was said earlier in the thread was that the locations should not be referenced in debian/rules, either, and that makes things difficult. This is the whole purpose of the dh_* snippets. Round and round we go. The people writing the dh_* snippets insist that the details of how they work, such as locations in which Python modules should actually be installed, can't be put into the Policy. The Policy editor, and those of use who don't want to use debhelper, insist that writing policy based on debhelper tools is not acceptable. This has now been going on for long enough that I conclude that the Python policy pushers really do intend to make using debhelper a Policy requirement for any package containing any Python code. -- On a clear disk, you seek forever. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Thu, Aug 24, 2006 at 05:56:07PM +0300, Lars Wirzenius wrote: installed, can't be put into the Policy. The Policy editor, and those of use who don't want to use debhelper, insist that writing policy based on debhelper tools is not acceptable. Not just those who don't want to use debhelper. Policy is, and has always been, written in terms of how dpkg works. Nothing more. It would be a very bad idea to change that, if only because somebody might have a nice idea for a build tool that would supersede debhelper. While it is, IMHO, perfectly acceptable to insist that people use some debhelper tool if they want to package something pythonesque, it is not acceptable to not document how they are *supposed* to work. -- Lo-lan-do Home is where you have to wash the dishes. -- #debian-devel, Freenode, 2004-09-22 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le jeudi 24 août 2006 à 17:56 +0300, Lars Wirzenius a écrit : Round and round we go. The people writing the dh_* snippets insist that the details of how they work, such as locations in which Python modules should actually be installed, can't be put into the Policy. The Policy editor, and those of use who don't want to use debhelper, insist that writing policy based on debhelper tools is not acceptable. Let me rephrase it: the internals of python-support, and how it helps implementing the python policy, are developed in the python-support documentation. They don't need to be part of the policy and they have nothing to do with debhelper either. This has now been going on for long enough that I conclude that the Python policy pushers really do intend to make using debhelper a Policy requirement for any package containing any Python code. I can't speak for others, but python-support provides pysupport-movemodules and pysupport-parseversions to separate the debhelper snippet from the actual abstraction code. (BTW, for a similar problematic that involves more than a hundred packages, nobody ever asked me how to make a package using GConf without using dh_gconf. Which means the GConf policy has never been written out but is currently defined by the dh_gconf behavior.) -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom signature.asc Description: Ceci est une partie de message numériquement signée
Re: dh_python and python policy analysis
On Thu, 24 Aug 2006 23:46:25 +0200, Josselin Mouette [EMAIL PROTECTED] said: Le jeudi 24 août 2006 à 17:56 +0300, Lars Wirzenius a écrit : Round and round we go. The people writing the dh_* snippets insist that the details of how they work, such as locations in which Python modules should actually be installed, can't be put into the Policy. The Policy editor, and those of use who don't want to use debhelper, insist that writing policy based on debhelper tools is not acceptable. Let me rephrase it: the internals of python-support, and how it helps implementing the python policy, are developed in the python-support documentation. They don't need to be part of the policy and they have nothing to do with debhelper either. This has now been going on for long enough that I conclude that the Python policy pushers really do intend to make using debhelper a Policy requirement for any package containing any Python code. I can't speak for others, but python-support provides pysupport-movemodules and pysupport-parseversions to separate the debhelper snippet from the actual abstraction code. I don't like adding unnecessary build dependencies for my packages. (BTW, for a similar problematic that involves more than a hundred packages, nobody ever asked me how to make a package using GConf without using dh_gconf. Which means the GConf policy has never been written out but is currently defined by the dh_gconf behavior.) Oh, all this means is that I havge not yet had any occassion to package Gnome stuff :) manoj -- There is nothing wrong with Southern California that a rise in the ocean level wouldn't cure. -- Ross MacDonald Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le samedi 12 août 2006 à 19:29 +0300, Lars Wirzenius a écrit : la, 2006-08-12 kello 18:10 +0200, Pierre Habouzit kirjoitti: /usr/share/pycentral /usr/share/python-support These location are tool specific and should not be referenced explicitely in the packaging scripts (debian/rules) agreed python-support seems to require me to put something into /usr/share/python-support (either the .py files, or a .dirs file for a package with private modules). How should I put them there without mentioning the location explicitly? The location is specific to the packaging tool and shouldn't be mentioned in the policy. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom
Re: dh_python and python policy analysis
ke, 2006-08-23 kello 10:46 +0200, Josselin Mouette kirjoitti: Le samedi 12 août 2006 à 19:29 +0300, Lars Wirzenius a écrit : la, 2006-08-12 kello 18:10 +0200, Pierre Habouzit kirjoitti: /usr/share/pycentral /usr/share/python-support These location are tool specific and should not be referenced explicitely in the packaging scripts (debian/rules) agreed python-support seems to require me to put something into /usr/share/python-support (either the .py files, or a .dirs file for a package with private modules). How should I put them there without mentioning the location explicitly? The location is specific to the packaging tool and shouldn't be mentioned in the policy. Sure, that's fine: no need to mention it in policy. What was said earlier in the thread was that the locations should not be referenced in debian/rules, either, and that makes things difficult. -- Latest nerd movie: Once were hackers -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Sun, Aug 13, 2006 at 07:56:09PM -0500, Manoj Srivastava wrote: OK, I see I have to dot the i's and cross the t's for this case here. So, here is the scenario: package python-foo packages a public pure python module. Package bar imports the module foo. Package baz is a package not yet written that would be written for Python2.6 that would also need module foo, but only when we actually get python2.6. Let us also have a package bar-baz that is written for python2.5. AFAICS you've correctly described the requirements for this scenario, but this is not the scenario that I've been trying to point out to you. The scenario where your policy draft fails to cover the needs of an application is this: Package python-foo packages a public pure python module. Package bar imports the module foo, and invokes /usr/bin/python2.3 as the interpreter. Current is 2.4, 2.3 is still in the archive (naturally, since bar uses it), other later versions may be available but are irrelevant; for our purposes, assume that foo is simple enough that it's compatible with all versions of python, past and present. Under 4.6 of the draft python policy, python-foo does not declare any Provides; thus: Package: python-foo Version: 1.0-1 Depends: python Package: bar Version: 1.0-1 Depends: python2.3, python-foo This is the only possible expression of these package relationships which is sanctioned by the policy draft. Now, introduce version 2.0 of python-foo. Because upstream considers python 2.3 obsolete, they have begun using language features in their module (internally, not as part of the module interface which remains unchanged) that are specific to python 2.4 and above. Now we have: Package: python-foo Version: 2.0-1 Depends: python (= 2.4) Provides: python2.4-foo But this package still satisfies the dependencies of package bar, even though bar needs python2.3 and python-foo is no longer compatible with python2.3. The package will be broken on partial upgrade, unless python-foo also adds a versioned conflict on bar! Thus, the problem arises when support for old versions of python is dropped by a module, rather than when new versions of python are introduced. Versioned conflicts are not scalable, frequently confuse the packaging system into removing packages instead of upgrading them, and will inevitably tend to be added after the fact resulting in unnecessarily bad user experiences in unstable (or even in testing and stable, if they escape notice long enough). This is why a package which needs module foo to be available for a specific version of python, instead of for the default version, should not be allowed to depend on python-foo. It is the same rationale as for requiring library package names to be changed with changes to library sonames. Now, whether python-foo actually declares Provides: python2.3-foo in the first place is something I think should be left entirely up to the maintainer; if the maintainer chooses not to declare the provides, then we should not allow bar to be packaged, because its rightful dependencies are python2.3, python2.3-foo and the latter dependency cannot be satisfied. I think it's perfectly ok for policy to discourage maintainers from diverging from the default version of python -- indeed, I think it's ideal to have only one version of python for a given stable release. But *if* a package is going to be in the archive which will require a non-default version of python, I think it's important that it be packaged in a way that doesn't require retconning with Conflicts to prevent broken package combinations on a user's system. Now we have two policy proposals, A, and B. A decrees that python-foo depends on python, ad has no provides. Policy B requires that python-foo also provide python2.3-foo and python2.4-foo. I don't actually think it's a good idea to require python-foo to provide python2.3-foo and python2.4-foo. I do think it should be *permitted*. I also recognize that expressing in the python policy when it is or isn't a good idea to declare the provides would be rather complicated... Now, looking at your example: The following transition events occur. No disagreements on 1) and 2). 3) Python2.3 is dropped. policy Apolicy B no upload. Upload python-foo, removing provides for 2.3 Why does python-foo need to drop the provides for 2.3? In what way does python2.3's removal from the archive mean that python-foo has stopped supporting python2.3? Now, if python-foo depends on an extension module and declares provides, then yes, it would need to be reuploaded whenever python2.3-quux ceases to be available. This will already be true for any modules that don't support all python versions. Also, 4b) foo can't be written for version 2,6, or will take time, and support for 2,6 is dropped (at least for
Re: dh_python and python policy analysis
On Sun, 13 Aug 2006 23:03:13 -0700, Steve Langasek [EMAIL PROTECTED] said: On Sun, Aug 13, 2006 at 07:56:09PM -0500, Manoj Srivastava wrote: OK, I see I have to dot the i's and cross the t's for this case here. So, here is the scenario: package python-foo packages a public pure python module. Package bar imports the module foo. Package baz is a package not yet written that would be written for Python2.6 that would also need module foo, but only when we actually get python2.6. Let us also have a package bar-baz that is written for python2.5. AFAICS you've correctly described the requirements for this scenario, but this is not the scenario that I've been trying to point out to you. Now, introduce version 2.0 of python-foo. Because upstream considers python 2.3 obsolete, they have begun using language features in their module (internally, not as part of the module interface which remains unchanged) that are specific to python 2.4 and above. This is similar to the case 4b in my scenario: a package changes from supporting all known versions of python to supporting a subset of that range. As in my case 4B, you don't take this action lightly, while there is python2.3 in the archive, you do a conditional import of the old version of the module in order to retain the compatibility, or you talk to your rdependsand tell them about the change. Now, looking at your example: The following transition events occur. No disagreements on 1) and 2). 3) Python2.3 is dropped. policy A policy B no upload. Upload python-foo, removing provides for 2.3 Why does python-foo need to drop the provides for 2.3? In what way does python2.3's removal from the archive mean that python-foo has stopped supporting python2.3? Because when python2.3 is removed, the site-packages directory would also be purged of all automatically compiled versions of the pure python modules. Indeed, I think this is already done; and minimizes the house cleaning packages have to perform. Also, 4b) foo can't be written for version 2,6, or will take time, and support for 2,6 is dropped (at least for the moment) Policy A policy B Package uploaded, with provides, Package uploaded, with provides Packages bar, bar-baz, and baz Package baz has to wait. (rdepends python-foo) informed of the provides. Need to upload the rdepends This is a case where I don't see any need to reupload foo under policy B: it already has all the provides it's going to have, since it didn't yet know about python2.6 and therefore couldn't have declared any provides for it, no? So python-foo Provides: python2.5-foo because bar-baz is written for python2.5 and needs it; bar is AFAICS written to use python, so only depends on python-foo; and baz has to wait. Well, if you are not going to upload, you can't be using python-support or python-central as the tools to automatically compile your module -- and unless you have changed the .versions file or XS-Python-Version variable, They'll try to compile, and may leave a miscompiled version around, depending on how subtle the 2.6 incompatibility is. === 1) Python 2.5 is added | no upload. python-foo | Upload python-foo, | recompiled for 2.5| adding provides for | | 2.5 | No transition for | package bar-baz must | package bar or| wait the upload. | bar-baz | -- 2) Default python | | version changes to | | 2.4| | -- 3) Python2.3 is dropped| | Upload python-foo, | | removing provides for | | 2.3 (since presumably | | the compiled files | | are gone -- 4) Python 2.6 is added.| | A) Source change fix | Package uploaded, with| Package uploaded, with | changed source. | changed source, and | | the provides. | package baz has to| package baz has t | wait | wait B) Do not support 2.6 | Package uploaded, with| Upload with changed | provides, rdepends | versions for byte | apprised, mini| compiling tools?? | transition| | package baz has to| package baz
Re: dh_python and python policy analysis
Hi, I have some more thoughts to offer on the example Steve presented: in essence, he is talking about a package that has become incompatible with the version that was in stable. On Sun, 13 Aug 2006 23:03:13 -0700, Steve Langasek [EMAIL PROTECTED] said: On Sun, Aug 13, 2006 at 07:56:09PM -0500, Manoj Srivastava wrote: Now, introduce version 2.0 of python-foo. Because upstream considers python 2.3 obsolete, they have begun using language features in their module (internally, not as part of the module interface which remains unchanged) that are specific to python 2.4 and above. This could happen to _any_ package written in any other language. Policy does not go out of its way to protect the reverse dependencies of a package that breaks compatibility between stable and testing -- were the package written in any other language. Why should Python be treated differently, if it makes for more uploads and makes Python transitions slower? What _do_ we do in other cases? There is precedence for creating a brand new package, called foo2 (X100/11, f;ex, apache, fvwm, and a load of others) so that new users can use the new functionality, while users of older package have time to transition. Maintainers can modify packages for Debian to retain compatibility. Or, worst case, we shrug and tell people that sorry, not in all cases can one maintain compatibility, so people should change the dependencies. I see no reason to go to heroic measures just for the sake of packages programmed in Python as opposed to, say, C -- given the drawbacks I have already mentioned in previous posts. manoj -- A nuclear war can ruin your whole day. Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Mon, 14 Aug 2006 09:36:29 -0500, Manoj Srivastava [EMAIL PROTECTED] said: Hate following up my own message. This is a biased recap of a discussion Steve ands I had on IRC, and ou should wait his response before taking my version of the discussion without a grain of salt. On Sun, 13 Aug 2006 23:03:13 -0700, Steve Langasek [EMAIL PROTECTED] said: On Sun, Aug 13, 2006 at 07:56:09PM -0500, Manoj Srivastava wrote: Now, introduce version 2.0 of python-foo. Because upstream considers python 2.3 obsolete, they have begun using language features in their module (internally, not as part of the module interface which remains unchanged) that are specific to python 2.4 and above. This could happen to _any_ package written in any other language. Policy does not go out of its way to protect the reverse dependencies of a package that breaks compatibility between stable and testing -- were the package written in any other language. Why should Python be treated differently, if it makes for more uploads and makes Python transitions slower? What _do_ we do in other cases? There is precedence for creating a brand new package, called foo2 (X100/11, f;ex, apache, fvwm, and a load of others) so that new users can use the new functionality, while users of older package have time to transition. Maintainers can modify packages for Debian to retain compatibility. Or, worst case, we shrug and tell people that sorry, not in all cases can one maintain compatibility, so people should change the dependencies. I see no reason to go to heroic measures just for the sake of packages programmed in Python as opposed to, say, C -- given the drawbacks I have already mentioned in previous posts. Additionally, consider what happens when we have a C library (Steve said that C libraries are a better fit than, say, flex). If libselinux comes out with a backwards incompatible change, and the so version changes, what do I have to do? I have to edit the control file, create libselinux2 instead of libselinux1; perhaps split off a new source package to have libselinux1 remain in the archive, and go from there. I see no need to behave differently if my package is coded in Python rather than in C. So create python-foo2, or python-foo1, in a fashion equivalent to what we do when the soname for a C lib changes. But until there is a backwards incompatible change, there is no need to do anything special (like changing the control file). Now, if the default version of Python in stable is dropped from testing, then there is an issue with partial upgrades from stable to testing ONLY for packages that have DELIBERATELY broken compatibility with the default version in stable. I would say we admonish people not to break compatibility with any version of Python shipping in the next release, and also not drop the default version in stable from testing. With these two policy directives, the partial upgrade issue is mitigated, and I think these are reasonable policy decisions. Even if upstream uses new features from the new version of the language (like decorators in version 2,4), the maintainer can still keep the old version around, create a wrapper, and conditionally load the old or the new version depending on the version of python used. I am willing to make life harder for people who introduce backwards incompatibilities in a pure python module when not doing so is simple workaround of conditional inclusion allows you to work with the old as well as the new python version. Debian maintainers are supposed to be competent enough in the languages used in the packages they maintainer to do a simple little extract into private module, and conditionally include one set of code or the other. On the issue of provides: I hate the notion that provides may be required, but hold on until someone asks for it bit. Either provides should be there, or not. If provides are required, then I support any private packages my users build as well as the so called official repo -- and since I have no way of knowing what private module packages out there are using, either provides become mandatory, or are not required. We are building a community her, not a club for DD's, so user created Python modules should not be neglected. It has been brought to my attention that a conservative maintainer might wish to assume that his module is incompatible with any older python versions that he hasn't explicitly tested it with, as well as any future versions of Python, because knowing that the code works correctly with an older version is a non-trivial problem. In this case, the conservative maintainer would never use XS-Python-Version: all, and the rules for pure python modules with a subset of versions of Python supported already have provisions for provide. So what's the conservative
Re: dh_python and python policy analysis
On Sat, 12 Aug 2006 16:57:52 -0600, Bruce Sass [EMAIL PROTECTED] said: On Sat August 12 2006 09:34, Matthias Klose wrote: First time I've seen the design goals laid out like this. Thanks, and sorry if this is out of place. No, not the whole design goal. Although the document is titled developer's view, the other goals should be mentioned as well. These are meant to work around processes in debian which are currently suboptimal, but unlikely to change. - The need to support more than one version of a python runtime or to support different implementations was seen. It takes a while until applications support a new version. What is needed now is a way to prevent all but the default modules and those selected by the admin from being built for all installed runtimes so Debian can re-step forward and properly support multiple runtimes. The new policies all-or-nothing attitude wrt modules is too much. It effectively forces one Python only and unnecessarily discourages Python hackers and developers because the cost of carrying all modules for all installed versions, instead of just those modules needed for the work being done in the different versions, can be quite high for someone with a comprehensive installation of even one Python version. Have you actually read the draft proposal? How did you come to the conclusion that only one run time is supported by that? manoj -- Whoever undertakes to set himself up as a judge of Truth and Knowledge is shipwrecked by the laughter of the gods.-- Albert Einstein Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote: 3.1.3. Provides Packages with public modules and extensions should be named, or should provide, python-foo. Pure Python public modules that support all Python versions need not have a Provides field. ... unless there is an application using a non-default python version using this module. or else you require the application depending on any indirect dependency of python-foo. Hmm. Two things: if application X requires my pure python public module (called, say, python-foo), and uses some specific version of python, why can't it depend on just python-foo Why do I have to provide pythonX.Y-foo? Because a dependency on python-foo expresses the request give me the foo module for the current version of python. There is no guarantee that the python-foo package installed is compatible with, or provides support for, the pythonX.Y you're using, except if this package declares a Provides: pythonX.Y-foo; so the depends/provides: pythonX.Y-foo needs to be there to ensure that the app and the modules it needs aren't allowed to get out of sync on a user's system (or in testing). Also, as a maintainer of python-foo, I can't know when such an application would be created, and we are trying to minimize reuploads of packages -- so either one provides all such pythonX.Y-foo at the get go, and reupload at every new python version or dropping of the old version -- or we upload every time some app is uploaded that may require yet abother X.y, and when we drop a version of Python. Such apps would ideally be few and far between, but after thinking about it for a while, I wasn't actually able to come up with a concrete case where having the Provides: declared ahead of time complicates transitions more than not having them would. For pure python modules, this still means inconvenient sourceful reuploads when new python implementations become available, since the Provides: can't be declared for pythonX.Y that we don't yet know about, but fortunately those reuploads would only need to be done on demand for modules that are actually used from scripts invoking a non-default python interpreter. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. [EMAIL PROTECTED] http://www.debian.org/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Sun, 13 Aug 2006 00:01:37 -0700, Steve Langasek [EMAIL PROTECTED] said: On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote: 3.1.3. Provides Packages with public modules and extensions should be named, or should provide, python-foo. Pure Python public modules that support all Python versions need not have a Provides field. ... unless there is an application using a non-default python version using this module. or else you require the application depending on any indirect dependency of python-foo. Hmm. Two things: if application X requires my pure python public module (called, say, python-foo), and uses some specific version of python, why can't it depend on just python-foo Why do I have to provide pythonX.Y-foo? Because a dependency on python-foo expresses the request give me the foo module for the current version of python. No, it does not. When a package bar depend on package baz, it means just that-- that it requires the package baz to work. With the policy draft that I have proposed, if it is a pure python module, with no intrinsic restrictions on the versions of python supported, other packages can just depend on the package, knowing that a policy compliant package would support all available versions. There is no guarantee that the python-foo package installed is compatible with, or provides support for, the pythonX.Y you're using, except if this package declares a Provides: pythonX.Y-foo; so the depends/provides: Rubissh. You are just making up these rules, and since it hurts, just don't make them up. If you want a pure python module that complies with the new policy, and does not provide pythonX.Y-foo; you know trhat you can just depend on python-foo, and things shall work. pythonX.Y-foo needs to be there to ensure that the app and the modules it needs aren't allowed to get out of sync on a user's system (or in testing). And why would they get out of sync? If they are compliant, then when a new python version is installed the module is compiled for it -- so no matter what version of python you use, there is a compiled .pyc file there. Also, as a maintainer of python-foo, I can't know when such an application would be created, and we are trying to minimize reuploads of packages -- so either one provides all such pythonX.Y-foo at the get go, and reupload at every new python version or dropping of the old version -- or we upload every time some app is uploaded that may require yet abother X.y, and when we drop a version of Python. Such apps would ideally be few and far between, but after thinking about it for a while, I wasn't actually able to come up with a concrete case where having the Provides: declared ahead of time complicates transitions more than not having them would. For pure python modules, this still means inconvenient sourceful reuploads when new python implementations become available, since the Provides: can't be declared for pythonX.Y that we don't yet know about, but fortunately those reuploads would only need to be done on demand for modules that are actually used from scripts invoking a non-default python interpreter. And if you just follow the new policy, no uploads are needed at all. The new policy wins. manoj -- America: born free and taxed to death. Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Sun, Aug 13, 2006 at 03:32:27AM -0500, Manoj Srivastava wrote: On Sun, 13 Aug 2006 00:01:37 -0700, Steve Langasek [EMAIL PROTECTED] said: On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote: 3.1.3. Provides Packages with public modules and extensions should be named, or should provide, python-foo. Pure Python public modules that support all Python versions need not have a Provides field. ... unless there is an application using a non-default python version using this module. or else you require the application depending on any indirect dependency of python-foo. Hmm. Two things: if application X requires my pure python public module (called, say, python-foo), and uses some specific version of python, why can't it depend on just python-foo Why do I have to provide pythonX.Y-foo? Because a dependency on python-foo expresses the request give me the foo module for the current version of python. No, it does not. When a package bar depend on package baz, it means just that-- that it requires the package baz to work. With the policy draft that I have proposed, if it is a pure python module, with no intrinsic restrictions on the versions of python supported, other packages can just depend on the package, knowing that a policy compliant package would support all available versions. Then what do you name a package that is a pure python module that *DOES* have intrinsic restrictions on the version of python supported? You are wrong to assume that a pure python module will automatically support all available versions of python -- if all python versions were completely backwards- and forwards-compatible with one another, there would be no reason in the first place to *have* multiple versions of them in the archive. The reality is that there *are* language differences with each implementation of python, and a pure python module may *not* work with any given implementation of python available in the archive, and we need a way to express such dependencies that guards users against package relationships being satisfied by broken combinations. There is no guarantee that the python-foo package installed is compatible with, or provides support for, the pythonX.Y you're using, except if this package declares a Provides: pythonX.Y-foo; so the depends/provides: Rubissh. You are just making up these rules, and since it hurts, just don't make them up. What hurts is your ignorance of design requirements that were discussed at length at the DebConf python BoF. If you want a pure python module that complies with the new policy, and does not provide pythonX.Y-foo; you know trhat you can just depend on python-foo, and things shall work. No, only packages that invoke /usr/bin/python as interpreter can do this. Packages that invoke /usr/bin/pythonX.Y can't possibly have any forward guarantee that python-foo will provide a working interface for the version of pythonX.Y they have installed. pythonX.Y-foo needs to be there to ensure that the app and the modules it needs aren't allowed to get out of sync on a user's system (or in testing). And why would they get out of sync. If they are compliant, then when a new python version is installed the module is compiled for it -- so no matter what version of python you use, there is a compiled .pyc file there. Python decorators are a language feature available only in python 2.4 and above. Given a module foo version 1.0 which does not use decorators, a module foo version 2.0 which does, and a package bar which invokes /usr/bin/python2.3 and depends on python2.3, python-foo, how do you protect against installing python-foo 2.0 and breaking bar? It doesn't matter a bit whether the .pyc is in the python2.3 search path when the module in question isn't compatible with that version of the language. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. [EMAIL PROTECTED] http://www.debian.org/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Sun, 13 Aug 2006 10:28:43 -0700, Steve Langasek [EMAIL PROTECTED] said: On Sun, Aug 13, 2006 at 03:32:27AM -0500, Manoj Srivastava wrote: On Sun, 13 Aug 2006 00:01:37 -0700, Steve Langasek [EMAIL PROTECTED] said: On Sat, Aug 12, 2006 at 12:10:07PM -0500, Manoj Srivastava wrote: 3.1.3. Provides Packages with public modules and extensions should be named, or should provide, python-foo. Pure Python public modules that support all Python versions need not have a Provides field. ... unless there is an application using a non-default python version using this module. or else you require the application depending on any indirect dependency of python-foo. Hmm. Two things: if application X requires my pure python public module (called, say, python-foo), and uses some specific version of python, why can't it depend on just python-foo Why do I have to provide pythonX.Y-foo? Because a dependency on python-foo expresses the request give me the foo module for the current version of python. No, it does not. When a package bar depend on package baz, it means just that-- that it requires the package baz to work. With the policy draft that I have proposed, if it is a pure python module, with no intrinsic restrictions on the versions of python supported, other packages can just depend on the package, knowing that a policy compliant package would support all available versions. Then what do you name a package that is a pure python module that *DOES* have intrinsic restrictions on the version of python supported? WShy do you need a special name? Had you read the actual proposed draft, you'd realize that we do cater to that possibility. You are wrong to assume that a pure python module will automatically support all available versions of python -- if all python versions were completely backwards- and forwards-compatible with one another, there would be no reason in the first place to *have* multiple versions of them in the archive. The reality is that there *are* language differences with each implementation of python, and a pure python module may *not* work with any given implementation of python available in the archive, and we need a way to express such dependencies that guards users against package relationships being satisfied by broken combinations. In these two paragraphs you are doing three things. a) Belabouring the obvious b) jumping to conclusions about what I assume, c) demonstrating that you have not read the proposed draft. There is no guarantee that the python-foo package installed is compatible with, or provides support for, the pythonX.Y you're using, except if this package declares a Provides: pythonX.Y-foo; so the depends/provides: Rubissh. You are just making up these rules, and since it hurts, just don't make them up. What hurts is your ignorance of design requirements that were discussed at length at the DebConf python BoF. A little knowledge is dangerous. Go and educate yourself on what is being discussed in this thread -- or, if you are too almighty busy to familiarize yourself with the things you weigh in on, let meattempt to summarize, eliding any oooh shiny sections: If you are packaging a public pure python module (not extension or private modules), there are two cases: either you have intrinsic versions dependencies in your module, or not. If you have version dependencies, your Depends and Provides rules are differennt, and you need to read the draft to see what they are. If there are no such internal dependencies, then you need not provide pythonX.Y-foo virtual packages, since you should arrange for your packages to be compiled for any new version that is uploaded. As to the BOF thing, I'll bite: Why one earth did the bof come up with design decisiosn that require every single goldarned python module package to be reuploaded every time a new version of python is added or removed? Why did the BOF choose to ignore the experience of the emacs listp community, that also has byte compilation issues, and subtle differences in byte compiler? Were the people in the bof aware that the python interpreter, like emacs itself, can optionally byte compile source on the fly, and lal that is lost is some speed? Frankly, if the BOF came up with a design that requires every single public pure python module to be uploaded every single time a new version of python is added or removed, than you should be happy I am ignoring their delibrations. Byte compiling interpred language files is not rocvket science. It is also not a new problem. manoj beginning to realize why the state pof python in Debian is such a mess -- To err is human, to moo bovine. Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To
Re: dh_python and python policy analysis
Le dim 13 août 2006 22:17, Manoj Srivastava a écrit : As to the BOF thing, I'll bite: Why one earth did the bof come up with design decisiosn that require every single goldarned python module package to be reuploaded every time a new version of python is added or removed? actually,it's not truly needed, it's needed iff a package do needs the new provides or not. and those reuploads are kind of binNMUs, the real problem here is that tehre is no arch:all binNMU and maybe that's here the problem that need fixing. -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgp9uI8A3k8ar.pgp Description: PGP signature
Re: dh_python and python policy analysis
Hi, OK, I see I have to dot the i's and cross the t's for this case here. So, here is the scenario: package python-foo packages a public pure python module. Package bar imports the module foo. Package baz is a package not yet written that would be written for Python2.6 that would also need module foo, but only when we actually get python2.6. Let us also have a package bar-baz that is written for python2.5. Also, ket us assume the module foo would work for versions 2.3, 2.4, 2.5 -- but in the yet unreleased version 2.6, stuff changes, and module foo would not be compatible as written. OK? State of Python at the start of the thought experiment: current is 2.3, available is also 2.4, and let us pretend no one has heard of 2.5 or 2.6 yet. With me so far? Now we have two policy proposals, A, and B. A decrees that python-foo depends on python, ad has no provides. Policy B requires that python-foo also provide python2.3-foo and python2.4-foo. The following transition events occur. 1) Python 2.5 is added. policy Apolicy B no upload. python-foo recompiledUpload python-foo, adding for 2.5 provides for 2.5 No transition for package bar package bar-baz must wait or bar-baz the upload. 2) Default python version changes to 2,4 3) Python2.3 is dropped. policy Apolicy B no upload. Upload python-foo, removing provides for 2.3 4) Python 2.6 is added. Here there are two cases. Either module foo can't be written at all for version 2.6, or it the same functionality can be provided with a code change, perhaps hidden behind a version conditional. How often have people seen a regression in Python that something that was doable for version N can't be done at all in version N + 1? 4a) foo can be coded for version 2.6 Policy A policy B Package uploaded, with the changed Package uploaded, with the source. package baz has to wait changed source, and with the provides. Package baz has to wait. 4b) foo can't be written for version 2,6, or will take time, and support for 2,6 is dropped (at least for the moment) Policy A policy B Package uploaded, with provides, Package uploaded, with provides Packages bar, bar-baz, and baz Package baz has to wait. (rdepends python-foo) informed of the provides. Need to upload the rdepends Now, most pure python packages will never see option 4 at all; and those that do, a number will be case 4a. Even for the case of 4b, there is time to do the transition for packages bar and bar-baz -- until 2.6 becomes the default, there is no critical bug in bar or bar-baz. Now take this to every single pure python module package in Debian, multiply with the upload by default for every single addition or removal of python packages, and you can see that adding more work in the corner case 4b is worth not having to upload packages multiple times by default. manoj -- I used to be Snow White, but I drifted. Mae West Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Sun, 2006-08-13 at 19:56 -0500, Manoj Srivastava wrote: Here there are two cases. Either module foo can't be written at all for version 2.6, or it the same functionality can be provided with This is a little simplistic. The parser changes fairly routinely in python versions. This means that a version conditional is not sufficient to provide compatability with older pythons - the module will not parse. The usual thing done for cross version support is to write in the older version of python, or in extreme cases (i.e. where performance really hurts) have two separate modules _foo_2_5 and _foo_2_6 and conditionally do 'from _foo_2_5 import *' etc. -Rob -- GPG key available at: http://www.robertcollins.net/keys.txt. signature.asc Description: This is a digitally signed message part
Re: dh_python and python policy analysis
On Sun, 13 Aug 2006 23:37:15 +0200, Pierre Habouzit [EMAIL PROTECTED] said: Le dim 13 août 2006 22:17, Manoj Srivastava a écrit : As to the BOF thing, I'll bite: Why one earth did the bof come up with design decisiosn that require every single goldarned python module package to be reuploaded every time a new version of python is added or removed? actually,it's not truly needed, it's needed iff a package do needs the new provides or not. and those reuploads are kind of binNMUs, the real problem here is that tehre is no arch:all binNMU and maybe that's here the problem that need fixing. They are sourceful NMU's: since the packaged source comprises of the .dsc, and that includes the full provides line, as well as the information in the source Packages file, etc, also needs to changes. The draft does not require any more uploads in the default case than absolutely required. I think the corner case that something that was feasible in the old version but not in th new one is a low probability event, really, so making that case a trifle inconvenient to ease the routine transitions of adding/removing python versions is a good tradeoff. manoj -- Novinson's Revolutionary Discovery: When comes the revolution, things will be different -- not better, just different. Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Mon, 14 Aug 2006 11:21:07 +1000, Robert Collins [EMAIL PROTECTED] said: On Sun, 2006-08-13 at 19:56 -0500, Manoj Srivastava wrote: Here there are two cases. Either module foo can't be written at all for version 2.6, or it the same functionality can be provided with This is a little simplistic. The parser changes fairly routinely in python versions. This means that a version conditional is not sufficient to provide compatability with older pythons - the module will not parse. The usual thing done for cross version support is to write in the older version of python, or in extreme cases (i.e. where performance really hurts) have two separate modules _foo_2_5 and _foo_2_6 and conditionally do 'from _foo_2_5 import *' etc. Either conditional code or conditional inclusion are variations on the common theme I alluded to in my admittedly, and designedly, simplified example. manoj -- There's too much beauty upon this earth for lonely men to bear. Richard Le Gallienne Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
la, 2006-08-12 kello 18:10 +0200, Pierre Habouzit kirjoitti: /usr/share/pycentral /usr/share/python-support These location are tool specific and should not be referenced explicitely in the packaging scripts (debian/rules) agreed python-support seems to require me to put something into /usr/share/python-support (either the .py files, or a .dirs file for a package with private modules). How should I put them there without mentioning the location explicitly? (Note that offering dh_anything is not a sufficient answer. The whole point of Manoj's document is to get the Python policy away from mandating debhelper, or any other helper package.) -- I am a werehuman. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Sat August 12 2006 09:34, Matthias Klose wrote: First time I've seen the design goals laid out like this. Thanks, and sorry if this is out of place. No, not the whole design goal. Although the document is titled developer's view, the other goals should be mentioned as well. These are meant to work around processes in debian which are currently suboptimal, but unlikely to change. - The need to support more than one version of a python runtime or to support different implementations was seen. It takes a while until applications support a new version. What is needed now is a way to prevent all but the default modules and those selected by the admin from being built for all installed runtimes so Debian can re-step forward and properly support multiple runtimes. The new policies all-or-nothing attitude wrt modules is too much. It effectively forces one Python only and unnecessarily discourages Python hackers and developers because the cost of carrying all modules for all installed versions, instead of just those modules needed for the work being done in the different versions, can be quite high for someone with a comprehensive installation of even one Python version. - The old schema of using pythonX.Y-foo packages let's land packages in the NEW queue, when support for another python runtime is added to the package. That certainly is a process, which could be addressed by FTP master (do not process a pythonX.Y+1-foo package manually, if pythonX.Y-foo is already in the archive). If pythonX.Y+1-foo has a -1 or -0.x Debian version then it is a NEW package... Having pythonX.Y-foo mentioned in the control file would disallow binary NMU's in situations where a python runtime is dropped or added (the control needs to be regenerated). A solution would be to define an own target to regenerate the control file, which is not called during the normal package build. Such source package would not be binNMUable, but could be the target of automated uploads. ...as are all packages built against a new runtime (see next paragraph), and if an X.Y runtime is dropped so are any X.Y-foo packages. So, what is the objective of this design goal? Pushing untested packages into the archive appears to be the result. Python routinely spits out warnings about code breaking changes happenning at the minor version level, surely that requires better handling than blindly binNMUing packages (what else can mass binNMU's be?)... the package's metainfo looks good, but nobody familiar with the code has actually tried the combination (build, intalled, and at least some testing) to see how it works, with any arch? That is un-Debian-like, even for unstable, and as a 10+ year Debian user I think I am within my rights to make such a judgement. - Putting extension modules for more than one python version into a package eases transition of these packages to the testing distribution, provided that the package supports to default python version in testing and the default python version in unstable. The schema used before with python-foo depending on pythondefault ver-foo required an extra upload of every package containing an extension, adding new dependencies on new shared libraries in unstable, but not yet in testing. All packages having a python ( X.Y) dependency had to be moved to testing at once. This needed to be fixed, but more time should have been taken in the planning because the entirety of the solution is even more suboptimal than the previous (incomplete) policy, imo. The thing that seems really telling in all this is that team maintenance promises to increase the quality of packages, yet Debian's Python team is explicitly using it to potentially decrease quality with mass binNMUs to speed up transitions to new runtime environments. I really hope someone shows how I've just made a fool of myself but I don't have a lot of confidence that will happen because speed and quality are usually a trade-off. - Bruce -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le sam 12 août 2006 17:34, Matthias Klose a écrit : dh_pysupport doesn't use this information, but requires the developer to explicitely pass the directory containing the extension module. that's not completely true, it only searches in /usr/lib/$pkg, /usr/share/$pkg, /usr/lib/games/$pkg and /usr/share/games/$pkg. If he finds a .so in there, he uses objdump to find out if the .so is python related or not, and acts accordingly. It just dont need current/current_ext at all, and uses different implementations choices. that's all. -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpfoLiO7xepj.pgp Description: PGP signature
Re: dh_python and python policy analysis
Manoj Srivastava writes: policy document. The current version, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ unreachable, comments for the posted text follow 1.1. Categorization of Python software Program/script This consists of software directly called by an end user of external program, and is independently interpreted by the Python interpreter. Usually starts with the magic bytes #!, with the interpreter being /usr/bin/python* or /usr/bin/env python*. Modules This is code included in python programs/scripts, and not invoked directly (serving as library modules in compiled languages). Modules can be categorized under two orthogonal criteria: firstly, based on the whether or not they are implemented purely in python, like so: Pure Python Module These are python source code, to be interpreted by the Python interpreter just like program/script code is, and may work across many versions of Python. Extension Module Extensions are C code compiled and linked against a specific version of the libpython library, and so can only be used by one version of Python. There should be no reason to link the extension against the python library. Usually many extensions which are developed upstream on Windows do link by default to libpython. Other extensions linking against libpython are those with build infrastructure maybe predating distutils. python-semanage is an example (and should not link using -z defs). Another thing to mention here is a Python package, a directory containing an __init__.py file plus modules and extensions. Another way of categorizing modules is based on whether or not they are available for use by third party scripts/modules. Public Public modules are available for use in other Python scripts or modules using the import directive. They are installed in one of the directories: /usr/lib/pythonX.Y This directory is reserved for official python modules. No other package apart from upstream official Python modules should install modules in this directory. /usr/lib/pythonX.Y/site-packages This is where most add-on modules live. Often, packages do not directly install modules here, but instead use utility packages like python-central and python-support to byte compile and install modules as needed. /var/lib/python-support/pythonX.Y Packages using python-support actually have their packages linked in from this directory, but no package should directly install modules there directly. See the documentation for python-support for details. maybe shorten that to all directories in sys.path; not sure if an explicit list of directories is needed. Packages may install public Python modules in directories specific to Python packaging utilities -- which specify the directories under which such modules should be droppped, and the the structure of these directories is defined by the utilities themselves. Please note that these directories are not in the path for Python, and are not available for modules to be imported from. At the time of writing, such uility specific directories include: ^^ /usr/share/pycentral /usr/share/python-support These location are tool specific and should not be referenced explicitely in the packaging scripts (debian/rules) 2. Goals of the new Python policy The new policy is designed to reduce the load on people packaging python modules when one of the following events occur, and, by the dint of doing so, ease the transition that occur as new Python versions are introduced, old ones removed, and as the default version of Python changes, with minimal impact on the target system. In case of the following events: No, not the whole design goal. Although the document is titled developer's view, the other goals should be mentioned as well. These are meant to work around processes in debian which are currently suboptimal, but unlikely to change. - The need to support more than one version of a python runtime or to support different implementations was seen. It takes a while until applications support a new version. - The old schema of using pythonX.Y-foo packages let's land
Re: dh_python and python policy analysis
Le sam 12 août 2006 17:34, Matthias Klose a écrit : Manoj Srivastava writes: policy document. The current version, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ unreachable, comments for the posted text follow doh, that works for me ?! Extension Module Extensions are C code compiled and linked against a specific version of the libpython library, and so can only be used by one version of Python. There should be no reason to link the extension against the python library. Usually many extensions which are developed upstream on Windows do link by default to libpython. Other extensions linking against libpython are those with build infrastructure maybe predating distutils. python-semanage is an example (and should not link using -z defs). Another thing to mention here is a Python package, a directory containing an __init__.py file plus modules and extensions. yes, that's indeed a thing to add. Public […] maybe shorten that to all directories in sys.path; not sure if an explicit list of directories is needed. I've no real opinion on that. Packages may install public Python modules in directories specific to Python packaging utilities -- which specify the directories under which such modules should be droppped, and the the structure of these directories is defined by the utilities themselves. Please note that these directories are not in the path for Python, and are not available for modules to be imported from. At the time of writing, such uility specific directories include: ^^ /usr/share/pycentral /usr/share/python-support These location are tool specific and should not be referenced explicitely in the packaging scripts (debian/rules) agreed No, not the whole design goal. Although the document is titled developer's view, the other goals should be mentioned as well. These are meant to work around processes in debian which are currently suboptimal, but unlikely to change. […] The alternative of dropping the python-foo package and just keeping the pythonX.Y-foo packages was not followed anymore (ftp master intervention and rewriting of control files). These are design decisions made for the distribution. There are concerns about some upstream developers that the design of dropping the pythonX.Y namespace for packages may not be rebust enough. Another consequence of the current design: the default python version *has* to be installed, other supported versions can be installed additionally, not as a replacement. agreed, this also has to be mentionned. This can be a single version, or one or more of a list of non-overlapping ranges. The lowest range may optionally omit a low end, and the highest range may optionally omit an upper end. In other words, the overall range may be open ended. The ranges are often matched to the set of all known Python version that have existed, and the supported set is the intersection of the known versions of python and the range specification. XS-Python-Version can have the values current and current_ext as well (plus the list of ranges), which will expand to current, if the package does not have any extensions and can be used with another python default version without a new upload. It's replaced by the version number of the current default version in the Python:Versions substitution variable. current_ext normally only needs to be used for packages having a private extension module. dh_pysupport doesn't use this information, but requires the developer to explicitely pass the directory containing the extension module. IMHO current and current_ext are choices of implementations of pycentral, which your following explanations truly underline. [snip] 3.1.2. Depends: The rules for calculating the dependencies a package has are simple. 1. If a script invokes /usr/bin/pythonX.Y, than the package must depend on pythonX.Y. This is because no amount of automatic byte compiling would ever get rid of the requirement that /usr/bin/pythonX.Y has to be present for the script to function. I think, that is too strict. The current behaviour is depending on the dh_python implementation scanning all files for that interpreter line. Consider a package with scripts in /usr/bin: foo, foo2.3, foo2.4, calling python, python2.3 and python2.4, which would lead to a dependency on all supported python versions. The scripts work for the default python version, for the non-default python versions only if the corresponding pythonX.Y package is installed. then the packager can choose to make those script use /usr/bin/python. I've seen very few of those packages, and if those truly exist, I'm confident we can solve the problem easily. 3.1.3. Provides Packages with public modules and
Re: dh_python and python policy analysis
On Sat, 12 Aug 2006 17:34:06 +0200, Matthias Klose [EMAIL PROTECTED] said: Manoj Srivastava writes: policy document. The current version, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ Unfortunately, you are commenting on an old version of the document, so some comments refer to issue that have already been addressed. Extension Module Extensions are C code compiled and linked against a specific version of the libpython library, and so can only be used by one version of Python. There should be no reason to link the extension against the python library. Usually many extensions which are developed upstream on Windows do link by default to libpython. Other extensions linking against libpython are those with build infrastructure maybe predating distutils. python-semanage is an example (and should not link using -z defs). I'll add these in general notes, or perhaps in the extension specific sections (since compilation rules do not belong in the classification section). Another thing to mention here is a Python package, a directory containing an __init__.py file plus modules and extensions. This is mentioned in general notes; I did not think it was a useful fifth classification, since we do not actually treat it very differently. Public Public modules are available for use in other Python scripts or modules using the import directive. They are installed in one of the directories: maybe shorten that to all directories in sys.path; not sure if an explicit list of directories is needed. Not all paths in sys.path are oermitted in packages (like the path ending in .zip). Packages may install public Python modules in directories specific to Python packaging utilities -- which specify the directories under which such modules should be droppped, and the the structure of these directories is defined by the utilities themselves. Please note that these directories are not in the path for Python, and are not available for modules to be imported from. At the time of writing, such uility specific directories include: ^^ /usr/share/pycentral /usr/share/python-support These location are tool specific and should not be referenced explicitely in the packaging scripts (debian/rules) You have just made all my selinux python packages violate this line :). I use update-python-modules in postinst and prerm, but I do not use any debhelper modules, and I should not be required to do so. I don't think that implicitly mandating the use of the helper scripts dh_(pysupport|pycentral|python) should belong in policy. 2. Goals of the new Python policy No, not the whole design goal. Although the document is titled developer's view, the other goals should be mentioned as well. These are meant to work around processes in debian which are currently suboptimal, but unlikely to change. [SNIP lots of good stuff] I'll add these to the document. New python version introduced * Most pure Python modules with no restrictions on the version of Python supported, and those pure Python modules that only have a lower bound on the versions of python supported (for example, 2.3-, or all), would require no 2.3- - = 2.3, not all. the range notation cannot express things like 2.2, = 2.4. The use case for the latter is the jython package (now removed from testing) still at an implementation level of the corresponding cpython version, with i.e. 2.3 not a supported python version anymore. So in the following text set of versions, instead of range of versions should be used. OK, done. The new policy also reduces the numbers of packages in the archive, by supporting multiple versions of Python in the same binary package (at the cost of increased size of that one package, but it should still result in space saving.) Maybe mention the two cases, where the package size increases: - extension modules - pure modules where different versions of the upstream package are shipped and are directly installed into /usr/lib/pythonX.Y/ OK. 3. Recipe for developers 3.1.1. Python versions supported by the source The XS-Python-Version field in debian/control specifies the versions of Python supported by the package [30][1]. While this is a requirement only if using the utility package python-central (python-support, for example, prefers debian/pyversions), setting this is appreciated in any case, according to the [31]new policy wiki[32][2]. This is used to track packages during Python transitions. ... and test rebuilds. OK. This can be a single version, or one or more of a list of non-overlapping ranges. The lowest range may optionally omit a low end, and the highest range may optionally omit an upper end. In other words, the overall range
Re: dh_python and python policy analysis
Le mer 9 août 2006 01:33, Manoj Srivastava a écrit : Hi, Another day, another draft. Here is the latest update for my take on the new Python policy document. The current version, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ very nice. I'm going to review it comparing it with the initial python policy by J.Wreschnig so that I'm sure we have not left anything apart, or contradicted vital points. Cheers, -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpPzCbUAbQy2.pgp Description: PGP signature
Re: dh_python and python policy analysis
Le mar 8 août 2006 00:18, Pierre Habouzit a écrit : § 2.3.3, 2.4.2, 2.5.3, 2.6.2: *here* the python$version alternative is correct, because /extensions/ can be used with a '/usr/bin/python' as soon as the python current version is in their supported range. so take the previous algorithm, and add to (2): if current python version isn't in that range, add an alternative to the pythonX.Y corresponding to the range lower bound. Meaning that in my test cases, instead of *SHOULD NOT HAPPEN* you will get: python (= 2.4) | python2.4 and in fact, wrt Depends, the algorithm for pure modules or extensions, private or public is the same. I forgot to explicitely mention that when extensions are in the package, then you have to restrict your 'python' range to the range of the python for which extensions have been built. That seems obvious, but it has to be stated in the policy very clearly. That means that if you ship a .so for e.g. 2.3, 2.4, then your python depends will be: python (= 2.3), python ( 2.5) even if the pyversions is 2.1- about debian/pyversions, unlike his brother XS-P-V it does not supports all/current. for python support you have to use sth like 2.1-. current explicitely says that the package is only built for the current python version, and not for any other supported in debian. But I don't like that value for the following reasons: * it says for what the package is built, whereas other values explain for which range of python versions the package is build-able, so semanticaly it's not homogenous ; * it prevents the packager to explain with which python versions the package is compatible, as saying 'current' suggests that the package is now compatible with the current python version, and will always in the future, wich may be wrong when (if that happen) some python 3.0 that is not 100% backward compatible should exists ; * it also give an information we already have as a package that is built for the current python version should depend upon python-dev and not python-all-dev ; * current has not a constant meaning, as it depends of the state of the package python-defaults, and not only of the state of the archive when the package was uploaded. This is IMHO the biggest flaw of that field value. so IMHO the current value should be documented as an internal pycentral value, and should not be recommended to be used for other ways of python packaging. OTOH, I thing pysupport should also support all as it's prettier than 2.1- and more explicit to the packager, and then it could go into the policy as a recomended value in that case. -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpkEcXWLwifH.pgp Description: PGP signature
Re: dh_python and python policy analysis
Hi, Another day, another draft. Here is the latest update for my take on the new Python policy document. The current version, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ I am including a text version below. manoj Packaging with the new Python policy A package developers view Manoj Srivastava Copyright (c) 2006 Manoj Srivastava Revision History Revision 1.0.4 25 Jul 2006 Obstacles to conformance with the new python policy. While the new Python policy, specifically the [1]Packaged Modules chapter, contains the elements that must be present in the debian/control filename, it is not very explicit about how the values are to be substituted. The Debian Wiki falls back on calling dh_python, which is not helpful in understanding the actual logic to be followed. This article is an attempt to correct this gap in documentation. -- Table of Contents 1. [2]Introduction 1.1. [3]Categorization of Python software 2. [4]Goals of the new Python policy 3. [5]Recipe for developers 3.1. [6]General Notes 3.1.1. [7]Python versions supported by the source 3.1.2. [8]Depends: 3.1.3. [9]Provides 3.1.4. [10]Build-Depends: 3.2. [11]Deprecating current in versions supported 3.3. [12]Script 3.3.1. [13]Supported versions 3.4. [14]Private Pure Python Modules 3.4.1. [15]Byte compilation 3.4.2. [16]Supported versions 3.5. [17]Private Extension 3.5.1. [18]Supported versions 3.6. [19]Public pure Python Module 3.6.1. [20]Byte compiling 3.6.2. [21]Supported versions 3.6.3. [22]Provides: 3.7. [23]Public Extension 3.7.1. [24]Supported versions 3.7.2. [25]Provides 4. [26]Changing the default Python version 4.1. [27]Python rtupdate scripts 4.1.1. [28]rtupdate script invocation 1. Introduction While trying to update SELinux packages, I ran across problems in trying to determine if my packages were complying with the new python policy: any practical tips for packaging generally devolved to the statement Oh, just run dh_python. This is my attempt to offer more concrete tips for packaging. While this document started by reverse engineering dh_python, it has, thanks to help from various people more knowledgeable about Python than I, moved beyond that, and is closer to being a specification unfettered by the idiosyncrasies of current tools and implementations. -- 1.1. Categorization of Python software Program/script This consists of software directly called by an end user of external program, and is independently interpreted by the Python interpreter. Usually starts with the magic bytes #!, with the interpreter being /usr/bin/python* or /usr/bin/env python*. Modules This is code included in python programs/scripts, and not invoked directly (serving as library modules in compiled languages). Modules can be categorized under two orthogonal criteria: firstly, based on the whether or not they are implemented purely in python, like so: Pure Python Module These are python source code, to be interpreted by the Python interpreter just like program/script code is, and may work across many versions of Python. Extension Module Extensions are C code compiled and linked against a specific version of the libpython library, and so can only be used by one version of Python. Another way of categorizing modules is based on whether or not they are available for use by third party scripts/modules. Public Public modules are available for use in other Python scripts or modules using the import directive. They are installed in one of the directories: /usr/lib/pythonX.Y This directory is reserved for official python modules. No other package apart from upstream official Python modules should install modules in this directory. /usr/lib/pythonX.Y/site-packages
Re: dh_python and python policy analysis
Hi, Here is round two of my take on python policy. I have incorporate the correction offered by various people, and read the documents for python-central and python-support, and incorporated my understanding of those into this document. So, this is my take on the new python policy, based on the analysis of the new Python policy draft and dh_python, and is supposed to be a rough specification of what the python policy is supposed to be (based on current dh_python behaviour). The current analysis, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ This can still use a lot of polishing. I am including a text version below. manoj Packaging with the new Python policy A package developers view Manoj Srivastava Copyright (c) 2006 Manoj Srivastava Revision History Revision 1.0 25 Jul 2006 Obstacles to conformance with the new python policy. While the new Python policy, specifically the [1]Packaged Modules chapter, contains the elements that must be present in the debian/control filename, it is not very explicit about how the values are to be substituted. The Debian Wiki falls back on calling dh_python, which is not helpful in understanding the actual logic to be followed. This article is an attempt to correct this gap in documentation. -- Table of Contents 1. [2]Introduction 1.1. [3]Categorization of Python software 2. [4]Recipe for developers 2.1. [5]General Notes 2.1.1. [6]Python versions supported by the source 2.1.2. [7]Supported versions in the binary package 2.1.3. [8]Depends: 2.1.4. [9]Provides 2.1.5. [10]Build-Depends: 2.2. [11]Script 2.2.1. [12]Supported versions 2.2.2. [13]Depends: 2.3. [14]Private Pure Python Modules 2.3.1. [15]Byte compilation 2.3.2. [16]Supported versions 2.3.3. [17]Depends: 2.4. [18]Private Extension 2.4.1. [19]Supported versions 2.4.2. [20]Depends: 2.5. [21]Public pure Python Module 2.5.1. [22]Byte compiling 2.5.2. [23]Supported versions 2.5.3. [24]Depends: 2.5.4. [25]Provides: 2.6. [26]Public Extension 2.6.1. [27]Supported versions 2.6.2. [28]Depends: 2.6.3. [29]Provides 1. Introduction While trying to update SELinux packages, I ran across problems in trying to determine if my packages were complying with the new python policy: any practical tips for packaging generally devolved to the statement Oh, just run dh_python. This is my attempt to offer more concrete tips for packaging, by reverse engineering dh_python for the specifications and tips. -- 1.1. Categorization of Python software Program/script This consists of software directly called by an end user of external program, and is independently interpreted by the Python interpreter. Usually starts with the magic bytes #!, with the interpreter being /usr/bin/python* or /usr/bin/env python*. Modules This is code included in python programs/scripts, and not invoked directly (serving as library modules in compiled languages). Modules can be categorized under two orthogonal criteria: firstly, based on the whether or not they are implemented purely in python, like so: Pure Python Module These are python source code, to be interpreted by the Python interpreter just like program/script code is, and may work across many versions of Python. Extension Module Extensions are C code compiled and linked against a specific version of the libpython library, and so can only be used by one version of Python. Another way of categorizing modules is based on whether or not they are available for use by third party scripts/modules. Public Public modules are available for use in other Python scripts or modules using the import directive. They are installed in one of the directories /usr/lib/pythonX.Y/site-packages
Re: dh_python and python policy analysis
my changes proposals follow. {+ +} are part I've actually modified.
Le lun 7 août 2006 21:42, Manoj Srivastava a écrit :
Hi,
Here is round two of my take on python policy. I have
incorporate the correction offered by various people, and read the
documents for python-central and python-support, and incorporated my
understanding of those into this document.
So, this is my take on the new python policy, based on the
analysis of the new Python policy draft and dh_python, and is
supposed to be a rough specification of what the python policy is
supposed to be (based on current dh_python behaviour). The current
analysis, and future updates, are to be found at
http://www.golden-gryphon.com/software/manoj-policy/
This can still use a lot of polishing.
I am including a text version below.
I'd add a § that explains the goals of the policy, that are unclear to
many people. Those are the following:
the new policy aims to reduce the pressure on packagers when
the /usr/bin/python version changes. In that case, it tries to:
* magically migrate packages that do not need *ANY* rebuild (pure
modules, private or public, they mostly only need new byte
compilation)
* packages that have already an extension built for the new python
package. E.G. current packages with public extensions for python2.3
and python2.4 does not need anything to work when python2.4 will
become default.
It aims also to reduce the pressure when a new pythonX.Y version will be
introduced. e.g., when python2.5 will arrive:
* most pure modules that have 2.3-, all, or alike pyversions will just
work (only need some bc which pycentral/pysupport will do)
* only need a binNMU for public extensions:
o packages using $(shell pyversions -s) to know for which python
versions to build for, and B-D upon python-all-dev will just work.
o same for cdbs packages, and distutils ones.
== here you should talk about pyversion (the /usr/bin/pyversion) and
its -d and -s options.
* only need a binNMU for private modules (will work if the package uses
pyversions -s to determine the python version to build for).
It endly also aims to reduce the pressure when an old pythonX.Y is
dropped, as only the packages *directly* depending on that pythonX.Y
version will need an manual upgrade (meaning the packages using
#! /usr/bin/pythonX.Y). the rest is again a matter of automated gestion
of the byte compiling, or binNMU to remove the extensions built for
that version of python.
One could also argue that the new policy greatly reduce the number of
packages in the archive (often 1 instead of 3 for the old policy) to
the cost of having many .so for different pythonv versions in the same
package, wich is a bit ugly. but this saves a lot of entries for
dpkg/apt, and also wins a lot of space for pure modules that are almost
always identical for different python version (meaning that currently
most of the pythonX.Y-foo for pure modules are just identical packages,
except for the /usr/lib/pythonX.Y path...)
The only packages that will need manual upgrades, are packages that
needed a range of python versions that did not contained the (at the
upload time) current python version, and a couple of other cases, wich
should reduce the packages that need a real (like in non automatable
way) action of the packager (like a source full upload). that will
mostly happen for packages that will begin to support
the /usr/bin/python in debian. That should makes the number of packages
that need manual things go from sth like 700 to sth like 50, wich makes
transitions mostly painless (wrt as a current full one).
§ 1.1, definition of Public modules:
This part is well worded IMHO. I'd instead say:
--8-
Public modules are available for use in other Python scripts or
modules using the import directive. They are installed in one of the
directories:
{+
o /usr/lib/pythonX.Y (for official python modules, nobody else than
upstream official python modules should live in there)
o /usr/lib/pythonX.Y/site-packages
Packages using python-support also see their public modules be linked
from /usr/lib/python-support/pythonX.Y, but nobody should directly
install anything here by itself.
+}
Packages may install public Python modules in directories specific
to Python packaging utilities -- which specify the directories under
which such modules should be droppped, and the the structure of
these directories is defined by the utilities themselves. Please
note that these directories are not in the path for Python, and are
not available for modules to be imported from. At the time of
writing, such uility specific directories include:
/usr/share/pycentral
/usr/share/python-support
Re: dh_python and python policy analysis
On Mon, Jul 31, 2006, Manoj Srivastava wrote: 2.1. [5]XS-Python-Version: 2.2. [6]XB-Python-Version: Your document keeps mentionning these, even as requirements, but XB- isn't required for packages using python-support, and XS can be replaced by debian/pyversions. Is your document targetted at inclusion in the dev-ref? PS: I really don't see the point in cross-posting to debian-devel@, please either post to one or the other, thanks. -- Loïc Minier [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Roberto C. Sanchez [EMAIL PROTECTED] wrote: Not sure if I missed it, but you seem to claim a copyright but not give an explicit license. I imagine you meant to put it under GPL or a free version of the GFDL. Could you please clarify and also add it to the document? I couldn't care less whether this thing has a license or not, but if it gets one, I'm sure Manoj will *not* choose any variant of the GNU Free Documentation License. And nobody should do that, or encourage people to use this flawed license. Note that we have voted by a GR that the GFDL is acceptable under certain conditions - but not that it is a good license. It isn't. Regards, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX/TeXLive)
Re: dh_python and python policy analysis
Le lundi 31 juillet 2006 à 21:10 -0500, Manoj Srivastava a écrit : Public modules are available for use in other Python scripts or modules using the import directive. They are installed in one of the directories /var/lib/python-support/pythonX.Y /usr/share/python-support Note that these two contain the same modules. The /usr/share directory isn't read by python, only the generated tree in /var is. The new python policy places certain requirements for packages that contain Python bits. 2.1. XS-Python-Version: 2.2. XB-Python-Version: These two are by no means requirements. XS-Python-Version is only a way to tell the packaging tools which versions to use, but you can also use debian/pyversions which is the recommended way as it doesn't pollute control files. XB-Python-Version is a way to generate metadata but it isn't necessary either. The same applies to all you've written about these fields. 2.3. Depends: Packaged modules available for the default Python version (or many versions including the default) must depend on python (= X.Y). If they require other modules to work, they must depend on the corresponding python-foo. They must not depend on any pythonX.Y-foo. For the packages to be consistent, the package should depend on all pythonX.Y-foo for the versions listed in Provides:. However, no packaging tool is currently able to generate this information. 2.4. Provides Packages with public modules and extensions should be named, or should provide, python-foo, if the package contains an extension for more than one python version. Also, for every version of python supported the package should provide pythonX.Y-foo packages. In fact, it should not provide this unless it has correct dependencies on all pythonX.Y-bar - but everyone is doing this wrong. 3.1.1.1. XS-Python-Version: This is a list of python versions supported by the package. This field can be a single version, or a set of ranges. This should be set to the list of python versions that the script can support, or all. If a script invokes /usr/bin/pythonX.Y, then XS-Python-Version should be set to X.Y. If dh_python isn't able to parse these headers (as it used to do in the old policy), I consider it a bug. 3.1.5. Public Extension Public extensions should be packaged with a name of python-foo, where foo is the name of the module. Such a package should support the current Debian Python version, and more if possible. Maybe a word on how public extensions and public python modules interact would be nice. Generally speaking, I don't find this document useful to the package maintainer. It focuses mostly on python-central's internals, which don't need to be fully understood by the maintainer, and which aren't useful if you don't use python-central. -- .''`. Josselin Mouette/\./\ : :' : [EMAIL PROTECTED] `. `'[EMAIL PROTECTED] `- Debian GNU/Linux -- The power of freedom
Re: dh_python and python policy analysis
Frank Küster wrote: Roberto C. Sanchez [EMAIL PROTECTED] wrote: Not sure if I missed it, but you seem to claim a copyright but not give an explicit license. I imagine you meant to put it under GPL or a free version of the GFDL. Could you please clarify and also add it to the document? I couldn't care less whether this thing has a license or not, but if it gets one, I'm sure Manoj will *not* choose any variant of the GNU Free Documentation License. And nobody should do that, or encourage people to use this flawed license. Note that we have voted by a GR that the GFDL is acceptable under certain conditions - but not that it is a good license. It isn't. Regards, Frank Thanks for the clarification. I was unaware of that. I just wanted to make certain that there were not any questions about it (the ability to derive from/modify Manoj's work). -Roberto -- Roberto C. Sanchez http://familiasanchez.net/~roberto signature.asc Description: OpenPGP digital signature
Re: dh_python and python policy analysis
On Tue, 01 Aug 2006 09:55:39 +0200, Josselin Mouette [EMAIL PROTECTED] said:
Le lundi 31 juillet 2006 à 21:10 -0500, Manoj Srivastava a écrit :
Public modules are available for use in other Python scripts or
modules using the import directive. They are installed in one of
the directories
/var/lib/python-support/pythonX.Y /usr/share/python-support
Note that these two contain the same modules. The /usr/share
directory isn't read by python, only the generated tree in /var is.
Thanks, I'll explicitly make a note of that in that section.
The new python policy places certain requirements for packages that
contain Python bits.
2.1. XS-Python-Version: 2.2. XB-Python-Version:
These two are by no means requirements. XS-Python-Version is only a
way to tell the packaging tools which versions to use, but you can
also use debian/pyversions which is the recommended way as it
doesn't pollute control files.
Hmm. I'll remove mention of it, then. How exactly one
invokes packaging tools should not be in this document; which is
trying to be a specification more than a packaging guide.
XB-Python-Version is a way to generate metadata but it isn't
necessary either. The same applies to all you've written about these
fields.
The draft Python policy states:
,[ Section 2.3 ]
| Your control file should also have a line:
| XB-Python-Version: ${python:Versions}
| The python:Versions is substituted by the supported Python versions of
| the binary package, based on XS-Python-Version. (If you are not using
| dh_python you will need to handle this substitution yourself.) The
| format of the field XB-Python-Version is the same as the
| XS-Python-Version field for packages not containing
| extensions. Packages with extensions must list the versions
| explicitely.
`
http://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html
Is the draft policy incorrect in this case? (The directive is
a should.)
2.3. Depends:
Packaged modules available for the default Python version (or many
versions including the default) must depend on python (= X.Y). If
they require other modules to work, they must depend on the
corresponding python-foo. They must not depend on any
pythonX.Y-foo.
For the packages to be consistent, the package should depend on all
pythonX.Y-foo for the versions listed in Provides:.
I'll add this note, thanks.
However, no packaging tool is currently able to generate this
information.
Well, that's all right. First we decide what is the right
thing to do, then we provide tools. Packaging tools are there to
assist us, after all, not to limit us.
2.4. Provides
Packages with public modules and extensions should be named, or
should provide, python-foo, if the package contains an extension
for more than one python version. Also, for every version of python
supported the package should provide pythonX.Y-foo packages.
In fact, it should not provide this unless it has correct
dependencies on all pythonX.Y-bar - but everyone is doing this
wrong.
Thanks, I'll try an go through to document to ensure I have
the specifications done correctly.
3.1.5. Public Extension
Public extensions should be packaged with a name of python-foo,
where foo is the name of the module. Such a package should support
the current Debian Python version, and more if possible.
Maybe a word on how public extensions and public python modules
interact would be nice.
I'd appreciate any suggestions.
Generally speaking, I don't find this document useful to the package
maintainer. It focuses mostly on python-central's internals, which
don't need to be fully understood by the maintainer, and which
aren't useful if you don't use python-central.
It is curious that you say that, since I have not yet looked
at pycentral, what you see here is derived ojnly from reading the
draft policy in detail and then reverse engineering what dh_python
does.
The motivation for this exercise was for me to be able to
understand what the desired requirements of the new python policy are
well enough to comply with them -- I prefer not using packaging tools
if I do not understand what they do and can't do it independently.
At this point, could you please point out the salient points
of divergence between python-central and python-support? From what I
am given to understand, these take public pure Python modules and
byte compile them for every avaialable version on the taarget
machine, and recompile as needed when new versions of python become
available.
Pointers to any packaging guyides using either tool would also
be appreciated.
manoj
--
If you can count your money, you don't have a billion dollars. Paul
Getty
Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
--
To
Re: dh_python and python policy analysis
On Tue, 1 Aug 2006 09:35:56 +0200, Loïc Minier [EMAIL PROTECTED] said: On Mon, Jul 31, 2006, Manoj Srivastava wrote: 2.1. [5]XS-Python-Version: 2.2. [6]XB-Python-Version: Your document keeps mentionning these, even as requirements, but XB- isn't required for packages using python-support, and XS can be replaced by debian/pyversions. I guess I do not understand enough about python-support, in that case. I was basing it off section 2.3 of the draft policy[1], which has inclusions of XB-Python-Version as a should directive. Could you point me to documentation on python-support, what it does, how to use it, and how it differs from python-central? Is your document targetted at inclusion in the dev-ref? Targeted for dev-ref, no. If python policy is ever going to be made into official Debian technical policy, then some kind of detailed specification PS: I really don't see the point in cross-posting to debian-devel@, please either post to one or the other, thanks. I think that this is of general enough interest to involve -devel; but I do not want to miss out Python folks who have given up on -devel, and this is of interest to Python packaging. So, in my opinion, cross posting for this is jusified as long as the discussion stays on topic. manoj [1] http://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html -- Who cares if it doesn't do anything? It was made with our new Triple-Iso-Bifurcated-Krypton-Gate-MOS process ... Manoj Srivastava [EMAIL PROTECTED] http://www.debian.org/%7Esrivasta/ 1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
On Tue, Aug 01, 2006, Manoj Srivastava wrote: Could you point me to documentation on python-support, what it does, how to use it, and how it differs from python-central? Well, python-support is documented at the expected /usr/share/doc/python-support and in the dh_pysupport man page. Perhaps the wiki page http://wiki.debian.org/DebianPython/NewPolicy is the best place where you can see how the two tools differ? -- Loïc Minier [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: dh_python and python policy analysis
Le mardi 01 août 2006 à 09:45 -0500, Manoj Srivastava a écrit :
Public extensions should be packaged with a name of python-foo,
where foo is the name of the module. Such a package should support
the current Debian Python version, and more if possible.
Maybe a word on how public extensions and public python modules
interact would be nice.
I'd appreciate any suggestions.
When there are both public extensions and public modules, both packaging
tools will handle them fine. With python-support, they are put
in /usr/{lib,share}/python-support/$package and will be symlinked
from /var/lib/python-support. The key point is to have them at the same
place, otherwise sometimes they won't work.
When there are only public extensions, packaging tools aren't needed at
all.
Generally speaking, I don't find this document useful to the package
maintainer. It focuses mostly on python-central's internals, which
don't need to be fully understood by the maintainer, and which
aren't useful if you don't use python-central.
It is curious that you say that, since I have not yet looked
at pycentral, what you see here is derived ojnly from reading the
draft policy in detail and then reverse engineering what dh_python
does.
This is because python-central's internals use the control file.
At this point, could you please point out the salient points
of divergence between python-central and python-support? From what I
am given to understand, these take public pure Python modules and
byte compile them for every avaialable version on the taarget
machine, and recompile as needed when new versions of python become
available.
Both tools have the same needs but they do it differently. This is
because python-central relies on special fields in the control file
while python-support relies on its own files and directories.
1. Information given by the maintainer about supported versions in
the source package: python-central uses the XS-Python-Version
field, while python-support uses debian/pyversions. For best
compatibility, both tools are able to use each other's data.
2. Place to store public modules: /usr/share/python-central
vs. /usr/share/python-support/$package.
3. Place to store public extensions: python-central keeps them in
place, in /usr/lib/pythonX.Y/site-packages, while python-support
moves them to /usr/lib/python-support/$package/pythonX.Y.
4. Information stored about supported versions in the binary
package: python-central uses XB-Python-Version - which is
mandatory in this case - while python-support uses either the
list of public extensions versions
in /usr/lib/python-support/$package or
the /usr/share/python-support/$package/.version file.
5. Information about which private extensions to bytecompile:
python-central will use all files ending in .py belonging to the
package, while python-support lists directories
in /usr/share/python-support/$package.dirs.
Pointers to any packaging guyides using either tool would also
be appreciated.
Python-support's README provides basic information on how to make a
package using it.
There's also a wiki page: http://wiki.debian.org/DebianPythonFAQ
--
.''`. Josselin Mouette/\./\
: :' : [EMAIL PROTECTED]
`. `'[EMAIL PROTECTED]
`- Debian GNU/Linux -- The power of freedom
signature.asc
Description: Ceci est une partie de message numériquement signée
dh_python and python policy analysis
Hi, I have finished my initial analysis of Python policy and dh_python, and created a rough specification of what the python policy is supposed to be (based on current dh_python behaviour). The current analysis, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ The document is a draft, since I have not been involved in Python development, it may have flaws, and I am hoping that people more conversant with Python development would point them out to me. The document could also stand some polishing; and since it was written piecemeal, continuity leaves much to be desired as yet. I am including a text version below. manoj Packaging with the new Python policy A package developers view Manoj Srivastava Copyright (c) 2006 Manoj Srivastava Revision History Revision 1.0 25 Jul 2006 Obstacles to conformance with the new python policy. While the new Python policy, specifically the [1]Packaged Modules chapter, contains the elements that must be present in the debian/control filename, it is not very explicit about how the values are to be substituted. The Debian Wiki falls back on calling dh_python, which is not helpful in understanding the actual logic to be followed. This article is an attempt to correct this gap in documentation. -- Table of Contents 1. [2]Introduction 1.1. [3]Categorization of Python software 2. [4]Requirements for packages (new policy) 2.1. [5]XS-Python-Version: 2.2. [6]XB-Python-Version: 2.3. [7]Depends: 2.4. [8]Provides 3. [9]Recipe for developers 3.1. [10]Based on type of python modules being packaged 3.1.1. [11]Script 3.1.2. [12]Private Pure Python Modules 3.1.3. [13]Private Extension 3.1.4. [14]Public pure Python Module 3.1.5. [15]Public Extension 1. Introduction While trying to update SELinux packages, I ran across problems in trying to determine if my packages were complying with the new python policy: any practical tips for packaging generally devolved to the statement Oh, just run dh_python. This is my attempt to offer more concrete tips for packaging, by reverse engineering dh_python for the specifications and tips. -- 1.1. Categorization of Python software Program/script This consists of software directly called by an end user of external program, and is independently interpreted by the Python interpreter. Usually starts with the magic bytes #!, with the interpreter being /usr/bin/python* or /usr/bin/env python*. Modules This is code included in python programs/scripts, and not invoked directly (serving as library modules in compiled languages). Modules can be categorized under two orthogonal criteria: firstly, based on the whether or not they are implemented purely in python, like so: Pure Python Module These are python source code, to be interpreted by the Python interpreter just like program/script code is, and may work across many versions of Python. Extension Module Extensions are C code compiled and linked against a specific version of the libpython library, and so can only be used by one version of Python. Another way of categorizing modules is based on whether or not they are available for use by third party scripts/modules. Public Public modules are available for use in other Python scripts or modules using the import directive. They are installed in one of the directories /usr/lib/pythonX.Y/site-packages /usr/lib/pythonX.Y /var/lib/python-support/pythonX.Y /usr/share/pycentral /usr/share/python-support Private Private modules are generally only accessible to a specific program or suite of programs included in the same package. They are installed in special directories, for example: /usr/lib/package /usr/share/package /usr/lib/games/package /usr/share/games/package -- 2. Requirements for packages (new policy) The new python policy places certain requirements for packages that contain Python bits. -- 2.1. XS-Python-Version: The XS-Python
Re: dh_python and python policy analysis
Manoj Srivastava wrote: Hi, I have finished my initial analysis of Python policy and dh_python, and created a rough specification of what the python policy is supposed to be (based on current dh_python behaviour). The current analysis, and future updates, are to be found at http://www.golden-gryphon.com/software/manoj-policy/ The document is a draft, since I have not been involved in Python development, it may have flaws, and I am hoping that people more conversant with Python development would point them out to me. The document could also stand some polishing; and since it was written piecemeal, continuity leaves much to be desired as yet. I am including a text version below. manoj Packaging with the new Python policy A package developers view Manoj Srivastava Copyright (c) 2006 Manoj Srivastava Revision History Revision 1.0 25 Jul 2006 Not sure if I missed it, but you seem to claim a copyright but not give an explicit license. I imagine you meant to put it under GPL or a free version of the GFDL. Could you please clarify and also add it to the document? Regards, -Roberto -- Roberto C. Sanchez http://familiasanchez.net/~roberto signature.asc Description: OpenPGP digital signature
