Re: python-central 0.4
On Fri, Oct 04, 2002 at 02:05:32AM +1000, Donovan Baarda wrote: > The reason I said site-packages is the _right_ way, is it makes mailman > modules usable from other applications. However, I agree that we shouldn't > _require_ application specific modules to be in site-packages, particularly > when upstream don't package that way. As you pointed out there are plenty of > good reasons not to put stuff in site-packages. just as an example, i am working on packaging spyce [1], and it does not put any of its module files in a directory like mailman does. thus all of the scripts using those module file would have to be changed from: import spyceModule to something like: from spyce import spyceModule which is not a fun change to have to make. > Is there any cases you can think of where 'dpkg -L | grep *.py' > will not be sufficient to identify the files to be "registered"? why doesn't python-central handle the (re)compilation of python modules. it can just get the list using "dpkg -L | grep *.py". that way, package postinst scripts dont have to worry about the details and we wont have to run dpkg-reconfigure, which could, for example, ask debconf questions again. [1] http://spyce.sourceforge.net/ -- gram pgpexbbG3Un9n.pgp Description: PGP signature
Re: python-central 0.4
On Thu, Oct 03, 2002 at 05:21:34PM +0200, Matthias Klose wrote:
> Donovan Baarda writes:
> > > I wouldn't call moving some files the packaging hell, and I have yet to
> > > understand why /usr/lib/mailman is so much saner or better than
> > > /usr/lib/python/site-packages/mailman.
> > > I looked into the mailman package. It should not be that much work to
> > > adapt it to the python-central framework (moving the Mailman module
> > > into .../site-packages/, removing the paths.py hack).
> > > And when the mailman Bug#162761 is fixed, I could even provide a patch ;)
> >
> > I guess this is an issue of "upstream compatiblity". Sure, packages could be
> > restructured to fit the scheme, but the other alternative is to masssage the
> > scheme to fit the upstream packages...
> >
> > I think that using /usr/lib/python/site-packages/mailman properly is the
> > _right_ way to do it, but I can see the argument for /usr/lib/mailman,
> > particularly when upstream does it that way.
>
> IMO this is absolutely wrong.
I was agreeing with you :-)
The reason I said site-packages is the _right_ way, is it makes mailman
modules usable from other applications. However, I agree that we shouldn't
_require_ application specific modules to be in site-packages, particularly
when upstream don't package that way. As you pointed out there are plenty of
good reasons not to put stuff in site-packages.
[...]
> So python-central _has_ to handle the case of python files laying
> around somewhere (i.e. /usr/lib/foo/{bar,baz}.py), recompiling these
> files on python upgrades.
>
> Please design python-central for all kind of packages, do not require
> the packages to be adapted to python-central.
That's another vote for support of python application packages with modules
outside /usr/lib/python/site-packages :-)
> > > Conclusion: the python-central framework as it is now offers enough for
> > > everyone. If people don't want to use it, they are on their own.
> >
> > I think we could add support for the mailman case with minimal hassles...
> >
> > The biggest problem I can forsee is this introduces another option that will
> > confuse developers... But this can be rectified with "strongly suggests" in
> > the python policy.
>
> There should be no confusion telling python-central the locations of
> files to be registered.
The confusion I was refering to was the "is a python-foo package
optional/encoraged/required for my pythonX.Y-foo packages" type questions
that have cropped up in responce to the current policy. I think this option
will raise "should I put my modules in /usr/lib/foo or
/usr/lib/python/site-packages/foo" type questions unless we clarify the
circumstances for each in the policy.
Is there any cases you can think of where 'dpkg -L | grep *.py'
will not be sufficient to identify the files to be "registered"?
> > > Hmm, reading the Debian policy, the only difference is that Depends only
> > > are considered on configuring a package, Conflicts are considered on
> > > installing. I dont know whats better, I think either will do.
> > > I will adapt the man pages to the Python Policy.
> >
> > Give this, I would prefer the "Depends" rather than "Conflicts", as it more
> > closely matches what a python version incompatability means... you can
> > install the files, but you can't configure them to work.
>
> please send a patch for the policy.
The "Depends" option is what is in the current policy. The python-central
man page was not matching the current policy.
> Bastian Kleineidam writes:
> > python-central (0.4) unstable; urgency=low
> >
> > * renamed register-python-package to python-register, this way
> > its prefixed with "python" (and its shorter ;)
>
> as long as python-central handles libraries/modules only, it should be
> called register-python-library. Don't confuse the packager ;-)
It'll get there eventualy :-)
--
--
ABO: finger [EMAIL PROTECTED] for more info, including pgp key
--
Re: python-central 0.4
Donovan Baarda writes:
> > I wouldn't call moving some files the packaging hell, and I have yet to
> > understand why /usr/lib/mailman is so much saner or better than
> > /usr/lib/python/site-packages/mailman.
> > I looked into the mailman package. It should not be that much work to
> > adapt it to the python-central framework (moving the Mailman module
> > into .../site-packages/, removing the paths.py hack).
> > And when the mailman Bug#162761 is fixed, I could even provide a patch ;)
>
> I guess this is an issue of "upstream compatiblity". Sure, packages could be
> restructured to fit the scheme, but the other alternative is to masssage the
> scheme to fit the upstream packages...
>
> I think that using /usr/lib/python/site-packages/mailman properly is the
> _right_ way to do it, but I can see the argument for /usr/lib/mailman,
> particularly when upstream does it that way.
IMO this is absolutely wrong.
Bastian Kleineidam writes:
> Hi,
>
> to put the code where my mouth was I tried to make a patch for mailman
> using the python-register utility.
> It was just moving files around - so I could fire up python and
> do an 'import Mailman'.
I originally mentioned mailman as an example for an application. Sure
mailman does consist of a library, which may be worth using in
ordinary python programs. However there _are_ applications with
modules not belonging in the /usr/lib/python tree: Assume a program
foo importing baz.py and bar.py. You cannot simply put these files in
/usr/lib/python:
- these may not be library modules.
- they pollute the namespace (assuming you have just another
application foo2, having its own bar module.
Even putting them in a separate directory does not help, because
these directories are all added to sys.path.
So python-central _has_ to handle the case of python files laying
around somewhere (i.e. /usr/lib/foo/{bar,baz}.py), recompiling these
files on python upgrades.
Please design python-central for all kind of packages, do not require
the packages to be adapted to python-central.
> > Conclusion: the python-central framework as it is now offers enough for
> > everyone. If people don't want to use it, they are on their own.
>
> I think we could add support for the mailman case with minimal hassles...
>
> The biggest problem I can forsee is this introduces another option that will
> confuse developers... But this can be rectified with "strongly suggests" in
> the python policy.
There should be no confusion telling python-central the locations of
files to be registered.
> > Hmm, reading the Debian policy, the only difference is that Depends only
> > are considered on configuring a package, Conflicts are considered on
> > installing. I dont know whats better, I think either will do.
> > I will adapt the man pages to the Python Policy.
>
> Give this, I would prefer the "Depends" rather than "Conflicts", as it more
> closely matches what a python version incompatability means... you can
> install the files, but you can't configure them to work.
please send a patch for the policy.
Bastian Kleineidam writes:
> python-central (0.4) unstable; urgency=low
>
> * renamed register-python-package to python-register, this way
> its prefixed with "python" (and its shorter ;)
as long as python-central handles libraries/modules only, it should be
called register-python-library. Don't confuse the packager ;-)
Matthias, away 'til Sunday.
Re: python-central 0.4
On Tue, Oct 01, 2002 at 12:23:38AM +1200, Carey Evans wrote: > Donovan Baarda wrote: [...] > It might also be nice to have separate files that list the directories > or source files to compile for each package, and have python-central > call compileall itself, but I guess this is a separate issue. you mean like; dpkg -L | grep '.py$' :-) -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key --
Re: python-central 0.4
Donovan Baarda wrote: But this is redundant... if the package is installed, the corresponding *.py files _should_ be unpacked. How is testing for the existance of a file any better indication than testing for installation of the package? True... I think I'll try to blame my flawed argument on lack of caffeine in the bloodstream. [...] This is an interesting issue I hadn't considered. However, I can't see how seperate compilation scripts do a better job of avoiding this problem than dpkg-reconfigure. Since, in the absence of debconf, all dpkg-reconfigure does is run /var/lib/dpkg/info/package.postinst configure /I guess editing the postinst would be fine as well. /However, it still seem to me that having recompilation separate from installing documentation, menus, MIME types and whatever else might be worthwhile. It might also be nice to have separate files that list the directories or source files to compile for each package, and have python-central call compileall itself, but I guess this is a separate issue. [...] So the postinst scripts get passed the version upgraded from? (I should know this :-) I should have checked too. See section 6 of Policy for details, especially what can happen if an error occurs, but the postinst is the one script that _doesn't_ get called with the other version number. The old version's prerm and postrm, and the new version's preinst, could all check whether an incompatible upgrade was going to or had occurred. In this case, I guess they'd set a flag and delete the old *.py[co] files.
Re: python-central 0.4
On Mon, Sep 30, 2002 at 08:19:49AM +1000, Donovan Baarda wrote: | On Mon, Sep 30, 2002 at 08:13:23AM +1200, Carey Evans wrote: | > It also makes it easier for users to modify if a Python package's | > dependencies are incorrect, and it stops compiling under a newer version | > of Python, making the whole dpkg run fail. | | This is an interesting issue I hadn't considered. However, I can't see how | seperate compilation scripts do a better job of avoiding this problem than | dpkg-reconfigure. If there's a malfunctioning script in /somewhere that I can insert exit 0 ; or import sys ; sys.exit( 0 ) at the top, then as a user/admin I can sidestep the installation error. If the logic is tucked away in the bowels of some other program, it isn't as trivial to hack around. (I don't have a position on this issue, though, I'm just sorta lurking on the discussions) -D -- But As for me and my household, we will serve the Lord. Joshua 24:15 http://dman.ddts.net/~dman/ pgp0JFaZZdMpe.pgp Description: PGP signature
Re: python-central 0.4
On Sun, Sep 29, 2002 at 05:51:30PM +0200, Bastian Kleineidam wrote: > On Sun, Sep 29, 2002 at 11:29:02PM +1000, Donovan Baarda wrote: > > I've just had a look at this and it looks good. It perfectly meets the > > requirement of allowing pure python module packages to support multiple > > pythonX.Y python packages simultaniously. > > > > The only problem is we are still missing something. We still can't easily > > support pure python _program_ packages that work with multiple versions of > > (the default) python. > What is wrong with > Depends: python2.2 | python2.3, python (>= 2.2), python (<< 2.4) > This supports the 2.2 and upcoming 2.3 default python package. Yes it does. However, under python central this means you would have all the modules compiled and symlinked for both python2.2 and python2.3 if they were both installed. This is great if you want to use these modules from both pythonX.Y versions, but in the case of something like mailman, you only want to use them from the default python. Having the other pythonX.Y support is not required. > > Programs like mailman put their modules in their own /usr/lib/mailman > > directory. One way that mailman _could_ support multiple pythonX.Y packages > > it to have the packager move _all_ the modules into a > > /usr/lib/python/site-packages directory and then use python central. > > However, this is overkill for mailman because it doesn't need to support > > python2.1 and python2.2 simultanously, just the default python. It is also > > hell for the packager. > I wouldn't call moving some files the packaging hell, and I have yet to > understand why /usr/lib/mailman is so much saner or better than > /usr/lib/python/site-packages/mailman. > I looked into the mailman package. It should not be that much work to > adapt it to the python-central framework (moving the Mailman module > into .../site-packages/, removing the paths.py hack). > And when the mailman Bug#162761 is fixed, I could even provide a patch ;) I guess this is an issue of "upstream compatiblity". Sure, packages could be restructured to fit the scheme, but the other alternative is to masssage the scheme to fit the upstream packages... I think that using /usr/lib/python/site-packages/mailman properly is the _right_ way to do it, but I can see the argument for /usr/lib/mailman, particularly when upstream does it that way. > Conclusion: the python-central framework as it is now offers enough for > everyone. If people don't want to use it, they are on their own. I think we could add support for the mailman case with minimal hassles... The biggest problem I can forsee is this introduces another option that will confuse developers... But this can be rectified with "strongly suggests" in the python policy. > > Another thing I noticed, the man pages are now recommending using; > > > >Depends: python (>=2.1) > >Conflicts: python (>= 2.4) > > > > Doesn't the policy have; > > > >Depends: python (>=2.1), python (<<2.4) > > > > I think these mean the same thing, but has the policy changed for some > > reason? I actually think that "Conflicts" is less ambiguous... > > Hmm, reading the Debian policy, the only difference is that Depends only > are considered on configuring a package, Conflicts are considered on > installing. I dont know whats better, I think either will do. > I will adapt the man pages to the Python Policy. Give this, I would prefer the "Depends" rather than "Conflicts", as it more closely matches what a python version incompatability means... you can install the files, but you can't configure them to work. > > Pending feedback/ideas whatever, I can code something up to do this. I > > probably favor using python-apt (if it can actualy do it) or grep-dctrl to > > do the package lookup stuff, but can do it manually if this is a problem. > Dont code anything like that. I'd rather fix those hacky'ish packages > which try to append something to sys.path. I think this needs feedback from the people actually maintaining the packages. If they think restructuring into /var/lib/python/site-packages is a good idea, then fine. However, if they would prefer to structure their files the same as upstream does... -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key --
Re: python-central 0.4
On Mon, Sep 30, 2002 at 08:13:23AM +1200, Carey Evans wrote: > Donovan Baarda wrote: > > >finding all packages that depend on "python" is non-trivial using only > >dpkg. > >Something like dpkg-awk, grep-dctrl, or python-apt make it much easier, but > >do we want to depend on them? The old "registry" idea would have made this > >a > >little easier, but I still prefer using the dpkg database to find this kind > >of info. > > I prefer myself the way that packages like emacsen-common, mime-support, > menu and I'm sure others do this: by having every package using Python > and wanting to be recompiled install a script in (for example) > /usr/share/python-central/packages/, which can be called > whenever .py files need recompiling. Alternatively, it could just be a > configuration file with paths to the files to recompile. python-central originally did have a "repository" where info about installed packages was put that could then be used to recompile/whatever. However, it was removed because there was nothing that this "repository" did that couldn't be done by using the existing dpkg database, so it just included redundant information that could have potentialy got out of sync. I don't think that having a seperate compile script for each package at /usr/share/python-central/packages/ is better than just running "dpkg-reconfigure ". The only slightly tricky thing is finding which packages need to be reconfigured, but all this info _is_ in the dpkg database dependancies. > Separate files that are part of the package would do the best job of > existing only when the corresponding .py files are also unpacked. But this is redundant... if the package is installed, the corresponding *.py files _should_ be unpacked. How is testing for the existance of a file any better indication than testing for installation of the package? > It also makes it easier for users to modify if a Python package's > dependencies are incorrect, and it stops compiling under a newer version > of Python, making the whole dpkg run fail. This is an interesting issue I hadn't considered. However, I can't see how seperate compilation scripts do a better job of avoiding this problem than dpkg-reconfigure. > >We only need to reconfigure/recompile when the default version of python > >upgrades to a different version of python, not minor package upgrades. Is > >there any way we can detect this and avoid un-necisary recompiles? > > The python package itself knows from the arguments to its postinst, and > could pass this on to python-central. So the postinst scripts get passed the version upgraded from? (I should know this :-) -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key --
Re: python-central 0.4
Donovan Baarda wrote: finding all packages that depend on "python" is non-trivial using only dpkg. Something like dpkg-awk, grep-dctrl, or python-apt make it much easier, but do we want to depend on them? The old "registry" idea would have made this a little easier, but I still prefer using the dpkg database to find this kind of info. I prefer myself the way that packages like emacsen-common, mime-support, menu and I'm sure others do this: by having every package using Python and wanting to be recompiled install a script in (for example) /usr/share/python-central/packages/, which can be called whenever .py files need recompiling. Alternatively, it could just be a configuration file with paths to the files to recompile. Separate files that are part of the package would do the best job of existing only when the corresponding .py files are also unpacked. It also makes it easier for users to modify if a Python package's dependencies are incorrect, and it stops compiling under a newer version of Python, making the whole dpkg run fail. We only need to reconfigure/recompile when the default version of python upgrades to a different version of python, not minor package upgrades. Is there any way we can detect this and avoid un-necisary recompiles? The python package itself knows from the arguments to its postinst, and could pass this on to python-central. -- Carey
Re: python-central 0.4
On Sun, Sep 29, 2002 at 11:29:02PM +1000, Donovan Baarda wrote: > I've just had a look at this and it looks good. It perfectly meets the > requirement of allowing pure python module packages to support multiple > pythonX.Y python packages simultaniously. > > The only problem is we are still missing something. We still can't easily > support pure python _program_ packages that work with multiple versions of > (the default) python. What is wrong with Depends: python2.2 | python2.3, python (>= 2.2), python (<< 2.4) This supports the 2.2 and upcoming 2.3 default python package. > Programs like mailman put their modules in their own /usr/lib/mailman > directory. One way that mailman _could_ support multiple pythonX.Y packages > it to have the packager move _all_ the modules into a > /usr/lib/python/site-packages directory and then use python central. > However, this is overkill for mailman because it doesn't need to support > python2.1 and python2.2 simultanously, just the default python. It is also > hell for the packager. I wouldn't call moving some files the packaging hell, and I have yet to understand why /usr/lib/mailman is so much saner or better than /usr/lib/python/site-packages/mailman. I looked into the mailman package. It should not be that much work to adapt it to the python-central framework (moving the Mailman module into .../site-packages/, removing the paths.py hack). And when the mailman Bug#162761 is fixed, I could even provide a patch ;) Conclusion: the python-central framework as it is now offers enough for everyone. If people don't want to use it, they are on their own. > Another thing I noticed, the man pages are now recommending using; > >Depends: python (>=2.1) >Conflicts: python (>= 2.4) > > Doesn't the policy have; > >Depends: python (>=2.1), python (<<2.4) > > I think these mean the same thing, but has the policy changed for some > reason? I actually think that "Conflicts" is less ambiguous... Hmm, reading the Debian policy, the only difference is that Depends only are considered on configuring a package, Conflicts are considered on installing. I dont know whats better, I think either will do. I will adapt the man pages to the Python Policy. > Pending feedback/ideas whatever, I can code something up to do this. I > probably favor using python-apt (if it can actualy do it) or grep-dctrl to > do the package lookup stuff, but can do it manually if this is a problem. Dont code anything like that. I'd rather fix those hacky'ish packages which try to append something to sys.path. Cheers, Bastian -- Bastian Kleineidam reflexionsniveau AT web.de calvin AT users.sf.net calvin AT debian.org pgpG7b6H6swnL.pgp Description: PGP signature
Re: python-central 0.4
On Sun, Sep 29, 2002 at 12:26:30AM +0200, Bastian Kleineidam wrote: > Hi, > > uploaded the new version 0.4 at > http://people.debian.org/~calvin/python-central/ > > python-central (0.4) unstable; urgency=low > > * renamed register-python-package to python-register, this way > its prefixed with "python" (and its shorter ;) > * add options to enable/suppress the compiling of .py[co] files > * add a new command "python-compile" to compile single python > files > * Added a separate section in python-register.1 for python programs; > also give an example. > > -- Bastian Kleineidam <[EMAIL PROTECTED]> Fri, 13 Sep 2002 22:17:45 +0200 I've just had a look at this and it looks good. It perfectly meets the requirement of allowing pure python module packages to support multiple pythonX.Y python packages simultaniously. The only problem is we are still missing something. We still can't easily support pure python _program_ packages that work with multiple versions of (the default) python. Programs like mailman have large mailman specific modules that are used only by the mailman program. The mailman program doesn't need to support multiple pythonX.Y packages simultaniously, just multiple versions of the default python. This is in many ways a simpler requirement than what python-central solves, but we still haven't got it covered. Programs like mailman put their modules in their own /usr/lib/mailman directory. One way that mailman _could_ support multiple pythonX.Y packages it to have the packager move _all_ the modules into a /usr/lib/python/site-packages directory and then use python central. However, this is overkill for mailman because it doesn't need to support python2.1 and python2.2 simultanously, just the default python. It is also hell for the packager. The only problem programs like mailman have is if the default python upgrades after they are installed, they need to be recompiled. The best way to handle this is to have the default python packages run dpkg-reconfigure on all installed packages that depend on python. This could be added to python central. Currently it auto-detects whether a python module or pythonX.Y package is being installed by looking at the package name, and behaves accordingly. All that needs to be added is to have python-central check for the package name "python", and have it do the dpkg-reconfigure for all packages depending on python. We then have the default python packages "register" themselves using python-central. Now some random thoughts; finding all packages that depend on "python" is non-trivial using only dpkg. Something like dpkg-awk, grep-dctrl, or python-apt make it much easier, but do we want to depend on them? The old "registry" idea would have made this a little easier, but I still prefer using the dpkg database to find this kind of info. Does dpkg-reconfigure actually run the postinst script? Does it do it in a way (right parameters etc) that would make exising python packages re-compile properly? Is a dpkg-reconfigure overkill? We only need to recompile the *.py's. Do some packages really need to go through a dpkg-reconfigure anyway? What if they are using debconf options for things like "compile modules"? We only need to reconfigure/recompile when the default version of python upgrades to a different version of python, not minor package upgrades. Is there any way we can detect this and avoid un-necisary recompiles? Should python-central be providing compile scripts for use by all python packages? Should we be making use of these scripts manditory in the Python Policy? If it was then we should be safe using these scripts to recompile packages instead of using dpkg-reconfigure. But would it be any better? Is compiling in postinst scripts so trivial as to not need support scripts? Another thing I noticed, the man pages are now recommending using; Depends: python (>=2.1) Conflicts: python (>= 2.4) Doesn't the policy have; Depends: python (>=2.1), python (<<2.4) I think these mean the same thing, but has the policy changed for some reason? I actually think that "Conflicts" is less ambiguous... Pending feedback/ideas whatever, I can code something up to do this. I probably favor using python-apt (if it can actualy do it) or grep-dctrl to do the package lookup stuff, but can do it manually if this is a problem. -- -- ABO: finger [EMAIL PROTECTED] for more info, including pgp key --
python-central 0.4
Hi, uploaded the new version 0.4 at http://people.debian.org/~calvin/python-central/ python-central (0.4) unstable; urgency=low * renamed register-python-package to python-register, this way its prefixed with "python" (and its shorter ;) * add options to enable/suppress the compiling of .py[co] files * add a new command "python-compile" to compile single python files * Added a separate section in python-register.1 for python programs; also give an example. -- Bastian Kleineidam <[EMAIL PROTECTED]> Fri, 13 Sep 2002 22:17:45 +0200 cu, Bastian -- Bastian Kleineidam reflexionsniveau AT web.de calvin AT users.sf.net calvin AT debian.org pgph75unJxJ81.pgp Description: PGP signature
