FHS location for Python libraries as locally-compiled bytecode (was: debhelper 7 and python-central)
Josselin Mouette <[EMAIL PROTECTED]> writes: > The python-central approach puts in the same directories files > managed by dpkg and files managed by python-central, which means > they cannot be sorted out easily, and if you run in a bug like > #409390 or #480741, you’re hosed. > > If anything ever gets wrong with python-support, all you need to do > is to run update-python-modules -f and the /var/lib/python-support > hierarchy will be entirely regenerated. I find this a convincing argument for separating the source and compiled versions (and thank you, I hadn't thought of this). I don't see that it leads to storing the compiled programs to live under /var/, rather than /usr/ which to my mind is more appropriate for compiled versions of programs installed by the package manager, which don't change except through explicit sysadmin action to change them. > As for the FHS argument, I don’t feel strongly for putting these > files in /var/lib, it just seemed like the most obvious location as > this is data that can be regenerated at any time. It can be changed > very easily if there is consensus that another place is better. FHS 2.3 §4 allows for: /usr/lib : Alternate format libraries (optional) Purpose /usr/lib performs the same role as /usr/lib for an alternate binary format, except that the symbolic links /usr/lib/sendmail and /usr/lib/X11 are not required. [26] "Python source code" versus "compiled Python bytecode" certainly qualifies as "an alternative binary format", so this seems the most applicable section of the FHS. Would '/usr/libcompiled/' or '/usr/libbytecode/' be an appropriate hierarchy to place locally-compiled bytecode for package-installed software? > What I do feel strongly for, is putting them in a directory that > remains separate from /usr/lib/python2.X. Thanks for your convincing argument, I'm now in support of this much. -- \ "When I was little, my grandfather used to make me stand in a | `\ closet for five minutes without moving. He said it was elevator | _o__) practice." -- Steven Wright | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: debhelper 7 and python-central
Pierre Habouzit <[EMAIL PROTECTED]> writes: > On Tue, May 20, 2008 at 10:59:05AM +, Ben Finney wrote: > > What makes you think 'foo.pyc' is /var material? > > Oh yes that seems obvious to me. In fact, I'd say it should be > /var/cache material because: > + it's not mandatory to have it, python works fine without .pyc, just > (way) slower (which makes it /something/*cache* material per se). It "works fine without it" only in the sense that it has to re-generate it before the program can run. Much like any compiled form of a program. > + it can be regenerated any time when a python version change (so that > we can gain new optimizations in how bytecode is built e.g.) which > makes it rather /var material rather than /usr. This property still makes it change only when the packaging system upgrades or installs software. If it only changes at that time, it still seems more appropriate for /usr/ than /var/. -- \ "My mother was like a sister to me, only we didn't have sex | `\ quite so often." -- Emo Philips | _o__) | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: debhelper 7 and python-central
On Tue, May 20, 2008 at 01:41:20PM +0200, Josselin Mouette wrote: > Le lundi 19 mai 2008 à 19:22 +0100, Floris Bruynooghe a écrit : > > > The latter approach is less fragile overall, > > > > How is that? As far as I can see both python-central and > > python-support maintain directories for each python version and > > symlink the .py files to a shared location. Just on that basis I > > can't see why one is better then the other. Am I missing something > > here? > > The python-central approach puts in the same directories files managed > by dpkg and files managed by python-central, which means they cannot be > sorted out easily, and if you run in a bug like #409390 or #480741, > you’re hosed. > > If anything ever gets wrong with python-support, all you need to do is > to run update-python-modules -f and the /var/lib/python-support > hierarchy will be entirely regenerated. Ack, that's a good reason to keep the tool-managed directory separate from /usr/lib/pythonX.Y/. > As for the FHS argument, I don’t feel strongly for putting these files > in /var/lib, it just seemed like the most obvious location as this is > data that can be regenerated at any time. It can be changed very easily > if there is consensus that another place is better. What I do feel > strongly for, is putting them in a directory that remains separate > from /usr/lib/python2.X. I still do feel strongly about /var/lib, despite Pierre's good argumentation (I agree with him that a alternate shadow path directory for .pyc files would fix this). The problem is that not just the .pyc files end up on /var, but also the .py files (in symlink form). This makes it required to have /var on sys.path and that is what makes me feel really bad. I expect tools (not daemons and more exotic stuff that need runtime state) to work when /var is unmounted. Therefore I'd be much happier if python-support's managed directory was somewhere on /usr/. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: debhelper 7 and python-central
On Tue, May 20, 2008 at 10:59:05AM +, Ben Finney wrote: > Pierre Habouzit <[EMAIL PROTECTED]> writes: > > > On Mon, May 19, 2008 at 11:20:10PM +, Ben Finney wrote: > > > Floris Bruynooghe <[EMAIL PROTECTED]> writes: > > > > "/var/lib : Variable state information [...] State information > > > > is data that programs modify while they run, and that pertains > > > > to one specific host." [2] > > > > > > Agreed. Complied-one-time-on-install Python library code is not > > > "variable state information"; rather, it is an unchanging (modulo > > > package-system changes) part of the system, so belongs in /usr > > > somewhere. > > > > pysupport puts a farmlink in /var/lib so that .pyc files that are > > /var material are in /var/lib. > > What makes you think 'foo.pyc' is /var material? Oh yes that seems obvious to me. In fact, I'd say it should be /var/cache material because: + it's not mandatory to have it, python works fine without .pyc, just (way) slower (which makes it /something/*cache* material per se). + it can be regenerated any time when a python version change (so that we can gain new optimizations in how bytecode is built e.g.) which makes it rather /var material rather than /usr. > > Note that the real issue here for real is that python isn't capable of > > using an alternate shadow path hierarchy to store .pyc files like e.g. > > fontconfig does. That's why pycentral/pysupport have to do really > > scabrous things in the first place. > > Agreed, and thank you for a wonderful use of the word "scabrous" :-) Heh, it's a Gallicism in fact, I mean it was a word to word translation of a quite current French expression ;) -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpbGy3AOjrUk.pgp Description: PGP signature
Re: debhelper 7 and python-central
Le lundi 19 mai 2008 à 19:22 +0100, Floris Bruynooghe a écrit : > > The latter approach is less fragile overall, > > How is that? As far as I can see both python-central and > python-support maintain directories for each python version and > symlink the .py files to a shared location. Just on that basis I > can't see why one is better then the other. Am I missing something > here? The python-central approach puts in the same directories files managed by dpkg and files managed by python-central, which means they cannot be sorted out easily, and if you run in a bug like #409390 or #480741, you’re hosed. If anything ever gets wrong with python-support, all you need to do is to run update-python-modules -f and the /var/lib/python-support hierarchy will be entirely regenerated. Because files are mixed in the same directories, python-central needs to do horrible things in the preinst scripts just to guarantee the files remain available during the upgrades. With a separate directory structure, all that is needed is to delay the cleanup (which actually resulted in reducing the code size for introducing this feature in python-support). As for the FHS argument, I don’t feel strongly for putting these files in /var/lib, it just seemed like the most obvious location as this is data that can be regenerated at any time. It can be changed very easily if there is consensus that another place is better. What I do feel strongly for, is putting them in a directory that remains separate from /usr/lib/python2.X. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: debhelper 7 and python-central
Le mardi 20 mai 2008 à 00:45 +0200, Pierre Habouzit a écrit : > This is _wrong_ to put XB-PV: 2.4, 2.5 for an arch:all package, this > should be XB-PV: all or something similar. Indeed, dh_pysupport puts wrong values in XB-Python-Version, so this is a bug. A very minor one, since this information is not used. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: debhelper 7 and python-central
Pierre Habouzit <[EMAIL PROTECTED]> writes: > On Mon, May 19, 2008 at 11:20:10PM +, Ben Finney wrote: > > Floris Bruynooghe <[EMAIL PROTECTED]> writes: > > > "/var/lib : Variable state information [...] State information > > > is data that programs modify while they run, and that pertains > > > to one specific host." [2] > > > > Agreed. Complied-one-time-on-install Python library code is not > > "variable state information"; rather, it is an unchanging (modulo > > package-system changes) part of the system, so belongs in /usr > > somewhere. > > pysupport puts a farmlink in /var/lib so that .pyc files that are > /var material are in /var/lib. What makes you think 'foo.pyc' is /var material? It's compiled once when the package is installed, and shouldn't change thereafter. That isn't "data that programs modify while they run", it's the binary form of the program itself. > The .py files live in /usr/share. That's fine. > Note that the real issue here for real is that python isn't capable of > using an alternate shadow path hierarchy to store .pyc files like e.g. > fontconfig does. That's why pycentral/pysupport have to do really > scabrous things in the first place. Agreed, and thank you for a wonderful use of the word "scabrous" :-) -- \"If you continue running Windows, your system may become | `\ unstable." -- Microsoft, Windows 95 BSOD message | _o__) | Ben Finney -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: debhelper 7 and python-central
On Mon, May 19, 2008 at 11:20:10PM +, Ben Finney wrote: > Floris Bruynooghe <[EMAIL PROTECTED]> writes: > > > On Mon, May 19, 2008 at 12:49:11PM +0200, Josselin Mouette wrote: > > > Python-central links modules at their original places while > > > python-support puts them in /var/lib to follow the FHS. > > > > "/var/lib : Variable state information [...] State information is data > > that programs modify while they run, and that pertains to one specific > > host." [2] > > Agreed. Complied-one-time-on-install Python library code is not > "variable state information"; rather, it is an unchanging (modulo > package-system changes) part of the system, so belongs in /usr > somewhere. pysupport puts a farmlink in /var/lib so that .pyc files that are /var material are in /var/lib. The .py files live in /usr/share. pysupport is supporting FHS to the letter. On the other side, pycentral puts things where they were installed, and pyc along them. That's why you find _totally_ arch indep .py files live in /usr/lib/python2.5, because they must go along with .pyc files and puting them in /usr/share where they belong would break because of .pyc files. Note that the real issue here for real is that python isn't capable of using an alternate shadow path hierarchy to store .pyc files like e.g. fontconfig does. That's why pycentral/pysupport have to do really scabrous things in the first place. pysupport is forward compatible with such a possible change in python, it would _just need_ to stop doing its symlink farm and keep byte compiled stuff in /var/lib like it does right now. -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpl8FyV9yXUK.pgp Description: PGP signature
Re: debhelper 7 and python-central
On Tue, May 20, 2008 at 07:55:44AM +, Emilio Pozuelo Monfort wrote: > Pierre Habouzit wrote: > > This is _wrong_ to put XB-PV: 2.4, 2.5 for an arch:all package, this > > should be XB-PV: all or something similar. > > Even if you package doesn't work with some supported Python versions? (e.g. it > uses the finally clause) Then you then don't have set XS-P-V to all, but something like >= 2.4 e.g., or 2.4- in the pysupport syntax. And the result SHOULD be XB-PV: 2.4- too. So that when python 2.6 arrives, your package doesn't need to be rebuilt or whatever. IOW for arch:all packages, XB-PV should be the same as XS-PV, not the intersection betweel XS-PV and available python version at the time of the build. And again, it's what dh_pysupport does, and it's the proper thing to do. -- ·O· Pierre Habouzit ··O[EMAIL PROTECTED] OOOhttp://www.madism.org pgpKCg3S1qjsk.pgp Description: PGP signature
Re: debhelper 7 and python-central
Pierre Habouzit wrote: > This is _wrong_ to put XB-PV: 2.4, 2.5 for an arch:all package, this > should be XB-PV: all or something similar. Even if you package doesn't work with some supported Python versions? (e.g. it uses the finally clause) Emilio signature.asc Description: OpenPGP digital signature
