Re: On robustness of maintainer scripts
* Jakub Wilk jw...@debian.org, 2013-05-10, 00:41: prerm = [...] Real-world failure: #706758 Also #708831. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130521120144.ga4...@jwilk.net
Re: On robustness of maintainer scripts
Am 13.05.2013 19:21, schrieb Steve Langasek: On Sat, May 11, 2013 at 12:29:07AM +0200, Piotr Ożarowski wrote: [Jakub Wilk, 2013-05-10] postinst [...] Proposed solution: 1) Wait until #671711 is fixed. 3) Make pycompile a shell script that does only two simple things: - write options it was called with to a file, say, /var/lib/python/pyX.Ycompile.todo; - use dpkg-trigger to trigger pythonX.Y. why not generate maintainer scripts without pycompile at all? I wanted to delegate as much as possible to interpreter packages, but your idea with temp. files is even better - maintainer scripts can look like this: | touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹ | touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹ namespace packages aren't an issue anymore in python3, and for python2 it would be better to ship these files into separate binary packages. these aren't that many, and it would be much better for the robustness. nothing to do anymore about these in maintainer scripts. | dpkg -L package | grep \.py$ | while read file | do |echo ${file} /var/lib/python/pyX.Ycompile.todo | echo ${file} /var/lib/python/pyX.Zcompile.todo | done The disadvantage is that the more logic is included directly in the maintainer scripts, the harder it is to fix any bugs with that logic because every package that includes the buggy behavior needs to be fixed. Even if that's only a simple rebuild with an updated version of dh_python2, it's quite costly to do that over all the affected packages in the archive. So from a robustness standpoint, it would be preferable if this logic could be encapsulated, if not in pycompile (for the reasons described), then at least in a trigger. If it needs to be encapsulated, then it should be in the pythonx.y packages shipping the interpreters, even if it does mean to duplicate these scripts up to two times. My hope was that most of the code for pycompile would end up in py_compile.py, and the pycompile script would rely on that. But that didn't happen. Depending on the complexity of the resulting pycompile, it could be inlined in maintainer scripts, but it doesn't have to be. For my part, I'm having a hard time understanding why any of the proposed changes here are warranted. Yes, there are corner cases where the current maintainer scripts will fail, but the real-world failures included in Jakub's original mail seem to reduce to bug #680930 (which I think it was reasonable to fix by changing the interpreter package's linkage) and bug #671711 (which is an obviously unacceptable bug in dpkg that needs to be fixed). The other bugs, such as bug #706758, seem to map to these directly (i.e., it's fixable by applying the same change for bug #680930 to the python2.6 package in wheezy - currently this fix has only been applied for python2.7); and I don't think that any problem that requires a command like this in a minimal reproducer case: # dpkg --force-depends -r libssl0.9.8 ... should seriously be considered a high priority, and be allowed to dictate the design of the interpreter handling. Relying on packages to be usable when unpacked-but-not-configured is fragile. But I am not convinced that the proposed cures are not worse than the disease. Certainly if we want to continue using pythonX.Y from maintainer scripts the way we do currently, great care would need to be taken to ensure they're usable when unpacked - which means a committment to either not add new library dependencies to the interpreter or ensure new library deps are listed in pre-depends instead of depends. Yet I think this should be entertained as an alternative to increasing the amount of code duplicated across hundreds of maintainer scripts. python2.7-minimal shouldn't get any more dependencies on libraries. Now the version in unstable depends on libc6, libgcc1, zlib1g. The libgcc1 dependency comes from a linker bug when built with -flto and is dropped once linked with ld 2.23.x. zlib is needed in the interpreter for the zip file support. From what I can see, it would be hard to drop. libc6 is interesting in that the update to 2.17 causes #708831, which I now worked around by making libc6 a pre-dependency. I don't think this is specific for python2.7-minimal, and will be exposed by other packages as well, as long as some essential packages like coreutils and perl are rebuilt against the new libc6. Afaicr, adding trigger support in some helpers did introduce its own set of upgrade issues, but this might be safe, if no helpers use triggers to generate and update symlink farms. I don't share the opinion on the severity of #709198, and how it should be fixed, if we are going this route of keeping the python interpreter working during upgrades. The dependency of python2.7-minimal on libpython2.7-minimal looks safe to me, I don't think it needs to be raised to a pre-dependency. Matthias -- To
Re: On robustness of maintainer scripts
* Matthias Klose d...@debian.org, 2013-05-21, 17:55: The dependency of python2.7-minimal on libpython2.7-minimal looks safe to me, At least in theory, it's not safe. If python2.7-minimal is unpacked, but libpython2.7-minimal is not (yet) unpacked, then all pyclean can do is: Could not find platform independent libraries prefix Consider setting $PYTHONHOME to prefix[:exec_prefix] ImportError: No module named site dpkg: error while cleaning up: subprocess installed post-installation script returned error exit status 1 -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130521173556.ga5...@jwilk.net
Re: On robustness of maintainer scripts
* Matthias Klose d...@debian.org, 2013-05-21, 17:55: namespace packages aren't an issue anymore in python3, and for python2 it would be better to ship these files into separate binary packages. these aren't that many, and it would be much better for the robustness. nothing to do anymore about these in maintainer scripts. We have about ~60 packages which ship files in /usr/share/python/ns/, which is quite a bit. That said, most of them don't need __init__.py at all, because the namespace would be created at runtime via *-nspkg.pth files. The only namespaces for which appropriate *-nspkg.pth don't exist are: * flaskext (python-flaksext.wtf): flaskext/__init__.py is already shipped by python-flask, so not a problem. * r2 (python-radare2): no other users of the namespace AFAICS, so it could ship __init__.py directly in .deb. * dummy (python-van.pydeb) * zope.app (python-van.pydeb) - these two look like a bug in dh_python2. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130521180705.ga8...@jwilk.net
Re: On robustness of maintainer scripts
On Tue, May 21, 2013 at 05:55:58PM +0200, Matthias Klose wrote: | dpkg -L package | grep \.py$ | while read file | do | echo ${file} /var/lib/python/pyX.Ycompile.todo | echo ${file} /var/lib/python/pyX.Zcompile.todo | done The disadvantage is that the more logic is included directly in the maintainer scripts, the harder it is to fix any bugs with that logic because every package that includes the buggy behavior needs to be fixed. Even if that's only a simple rebuild with an updated version of dh_python2, it's quite costly to do that over all the affected packages in the archive. So from a robustness standpoint, it would be preferable if this logic could be encapsulated, if not in pycompile (for the reasons described), then at least in a trigger. If it needs to be encapsulated, then it should be in the pythonx.y packages shipping the interpreters, even if it does mean to duplicate these scripts up to two times. That makes sense to me. Would that code be able to live in a trigger? (Bearing in mind that a trigger can't doesn't get told *where* the new files are) # dpkg --force-depends -r libssl0.9.8 ... should seriously be considered a high priority, and be allowed to dictate the design of the interpreter handling. Relying on packages to be usable when unpacked-but-not-configured is fragile. But I am not convinced that the proposed cures are not worse than the disease. Certainly if we want to continue using pythonX.Y from maintainer scripts the way we do currently, great care would need to be taken to ensure they're usable when unpacked - which means a committment to either not add new library dependencies to the interpreter or ensure new library deps are listed in pre-depends instead of depends. Yet I think this should be entertained as an alternative to increasing the amount of code duplicated across hundreds of maintainer scripts. python2.7-minimal shouldn't get any more dependencies on libraries. Now the version in unstable depends on libc6, libgcc1, zlib1g. The libgcc1 dependency comes from a linker bug when built with -flto and is dropped once linked with ld 2.23.x. zlib is needed in the interpreter for the zip file support. From what I can see, it would be hard to drop. libc6 is interesting in that the update to 2.17 causes #708831, which I now worked around by making libc6 a pre-dependency. I don't think this is specific for python2.7-minimal, and will be exposed by other packages as well, as long as some essential packages like coreutils and perl are rebuilt against the new libc6. Essential packages are *required* to deal with library dependencies as pre-dependencies. So this is specific to python, in the sense that python is being called from maintainer scripts before the package has been configured. I think the pre-depends is the right change here; but as Aurelien points out, pre-dependencies are supposed to be discussed on debian-devel because they can have system-wide repercussions. Can you please kick off a thread to discuss this there? I don't share the opinion on the severity of #709198, and how it should be fixed, if we are going this route of keeping the python interpreter working during upgrades. Having the debconf package break during upgrades because it tries to call python and fails looks like a pretty serious bug to me. I'm not sure about what the right way to *fix* it is, but we definitely don't want users' upgrades to blow up... -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: On robustness of maintainer scripts
[Jakub Wilk, 2013-05-12] * Piotr Ożarowski pi...@debian.org, 2013-05-11, 00:29: why not generate maintainer scripts without pycompile at all? - There are hundreds of packages that use py{compile,clean} in the maintainer script. The script needs to continue to exist at least until jessie is released. true, but we will get rid of it in jessie+1, not in jessie - Fixing the problem in py{compile,clean} means that even the packages that haven't been rebuilt can benefit from the fix. that's why I want to keep it simple in maintainer scripts. dh_python2 knows for which versions given directory has to be bytecompiled, so why not hardcode these directories in maintainer scripts (and use dpkg -L package | grep dir to fill todo files). - If you take all corner cases into cosideration, the required gets pretty complicated. IMO too complicated to risk putting the code directly into maintainer scripts. if we get rid of exclude patterns (and not a single package in unstable provides a file in /usr/share/python/bcep/ directory, so it's not really needed apparently) there are not that many features in pycompile... Maintainer scripts can simply create missing namespaces and then let interpreter packages know which files they should bytecompile (by creating todo files for X.Y or default interpreter). pycompileX.Y (provided by interpreter package and invoked by dpkg trigger) can read such files and bytecompile all files listed there (rt* scripts can do similar thing) [...] Here you need to check with versions are supported, and which of them are installed^Wunpacked. Oh, and the call to dpkg-trigger is missing. /o\ shebang is missing too! ... you know it was just an example to demonstrate the idea and not the final code, right? :) -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130513093816.gd24...@p1otr.com
Re: On robustness of maintainer scripts
On Sat, May 11, 2013 at 12:29:07AM +0200, Piotr Ożarowski wrote: [Jakub Wilk, 2013-05-10] postinst [...] Proposed solution: 1) Wait until #671711 is fixed. 3) Make pycompile a shell script that does only two simple things: - write options it was called with to a file, say, /var/lib/python/pyX.Ycompile.todo; - use dpkg-trigger to trigger pythonX.Y. why not generate maintainer scripts without pycompile at all? I wanted to delegate as much as possible to interpreter packages, but your idea with temp. files is even better - maintainer scripts can look like this: | touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹ | touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹ | dpkg -L package | grep \.py$ | while read file | do | echo ${file} /var/lib/python/pyX.Ycompile.todo | echo ${file} /var/lib/python/pyX.Zcompile.todo | done The disadvantage is that the more logic is included directly in the maintainer scripts, the harder it is to fix any bugs with that logic because every package that includes the buggy behavior needs to be fixed. Even if that's only a simple rebuild with an updated version of dh_python2, it's quite costly to do that over all the affected packages in the archive. So from a robustness standpoint, it would be preferable if this logic could be encapsulated, if not in pycompile (for the reasons described), then at least in a trigger. For my part, I'm having a hard time understanding why any of the proposed changes here are warranted. Yes, there are corner cases where the current maintainer scripts will fail, but the real-world failures included in Jakub's original mail seem to reduce to bug #680930 (which I think it was reasonable to fix by changing the interpreter package's linkage) and bug #671711 (which is an obviously unacceptable bug in dpkg that needs to be fixed). The other bugs, such as bug #706758, seem to map to these directly (i.e., it's fixable by applying the same change for bug #680930 to the python2.6 package in wheezy - currently this fix has only been applied for python2.7); and I don't think that any problem that requires a command like this in a minimal reproducer case: # dpkg --force-depends -r libssl0.9.8 ... should seriously be considered a high priority, and be allowed to dictate the design of the interpreter handling. Relying on packages to be usable when unpacked-but-not-configured is fragile. But I am not convinced that the proposed cures are not worse than the disease. Certainly if we want to continue using pythonX.Y from maintainer scripts the way we do currently, great care would need to be taken to ensure they're usable when unpacked - which means a committment to either not add new library dependencies to the interpreter or ensure new library deps are listed in pre-depends instead of depends. Yet I think this should be entertained as an alternative to increasing the amount of code duplicated across hundreds of maintainer scripts. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: Digital signature
Re: On robustness of maintainer scripts
* Jakub Wilk jw...@debian.org, 2013-05-10, 00:41: postinst [...] Test-case (in a squeeze chroot, with wheezy added to sources.lists): # dpkg --unpack cpuset_1.5.6-2_all.deb # apt-get install -t squeeze python2.6 # apt-get install -t wheezy python gcc-4.4 # dpkg --force-depends -r libssl0.9.8 # dpkg -i python-pkg-resources_0.6.24-1_all.deb Blargh, I suck at copyingpasting: g/cpuset/d prerm = [...] Test case (in wheezy chroot): # apt-get install -q python-pkg-resources # dpkg --force-depends -r libssl0.9.8 # dpkg -r python-ipaddr s/wheezy/squeeze/ s/ipaddr/pkgresources/ A different test case (in squeeze chroot): # apt-get install python-pkg-resources # dpkg --unpack --auto-deconfigure python-minimal_2.7.3-4_all.deb # dpkg -r python-pkg-resources -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130513223139.ga8...@jwilk.net
Re: On robustness of maintainer scripts
* Steve Langasek vor...@debian.org, 2013-05-13, 12:21: why not generate maintainer scripts without pycompile at all? I wanted to delegate as much as possible to interpreter packages, but your idea with temp. files is even better - maintainer scripts can look like this: | touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹ | touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹ | dpkg -L package | grep \.py$ | while read file | do | echo ${file} /var/lib/python/pyX.Ycompile.todo | echo ${file} /var/lib/python/pyX.Zcompile.todo | done The disadvantage is that the more logic is included directly in the maintainer scripts, the harder it is to fix any bugs with that logic because every package that includes the buggy behavior needs to be fixed. Even if that's only a simple rebuild with an updated version of dh_python2, it's quite costly to do that over all the affected packages in the archive. Hear, hear. We had a Python helper that put too much logic directly into maintainer scripts once: it was python-central in lenny. Don't let such a disaster happen again. Thanks. :) -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130513223338.gb8...@jwilk.net
RE : On robustness of maintainer scripts
Hello Jackub prerm = ... Real-world failure: #706758 should I reassign this bug to python ? If I understand well, all this, python should be fixed then a rebuild of all python packages should be done. right ? which bug of python should I track to know when a rebuild of the package should fix the issue ? Fred -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/a2a20ec3b8560d408356cac2fc148e5358e90...@sun-dag1.synchrotron-soleil.fr
Re: On robustness of maintainer scripts
* Piotr Ożarowski pi...@debian.org, 2013-05-11, 00:29: postinst [...] Proposed solution: 1) Wait until #671711 is fixed. 3) Make pycompile a shell script that does only two simple things: - write options it was called with to a file, say, /var/lib/python/pyX.Ycompile.todo; - use dpkg-trigger to trigger pythonX.Y. why not generate maintainer scripts without pycompile at all? - There are hundreds of packages that use py{compile,clean} in the maintainer script. The script needs to continue to exist at least until jessie is released. - Fixing the problem in py{compile,clean} means that even the packages that haven't been rebuilt can benefit from the fix. - If you take all corner cases into cosideration, the required gets pretty complicated. IMO too complicated to risk putting the code directly into maintainer scripts. I wanted to delegate as much as possible to interpreter packages, but your idea with temp. files is even better - maintainer scripts can look like this: | touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹ | touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹ | dpkg -L package | grep \.py$ | while read file | do | echo ${file} /var/lib/python/pyX.Ycompile.todo | echo ${file} /var/lib/python/pyX.Zcompile.todo | done Here you need to check with versions are supported, and which of them are installed^Wunpacked. Oh, and the call to dpkg-trigger is missing. /o\ [¹] namespace has to be created here as other packages might want to use given library in maintainer scipts, even without .pyc files ACK (test -f ... || is probably missing here) Yes, it is missing. :) prerm = [...] Proposed solution: Get rid of /usr/bin/pyclean, so that the fallback code is activated. \o/ Just kidding. :) Not only that would almost certainly have unintended consequences, but also the fallback code doesn't look complete: it doesn't take care of namespaces. Actual proposed solution: Rewrite /usr/bin/pyclean in shell. again, removing /usr/bin/pyclean is not that bad idea. I like it and will try to provide example scripts for interpreter packages (and then generate prerm like in postinst example) You need to remove namespaces, remove files from *.todo... I wouldn't dare to put it directly to prerm. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130512205130.ga2...@jwilk.net
Re: On robustness of maintainer scripts
* PICCA Frédéric-Emmanuel frederic-emmanuel.pi...@synchrotron-soleil.fr, 2013-05-12, 08:22: prerm = ... Real-world failure: #706758 should I reassign this bug to python ? I think so. (But maybe there's flaw a in my reasoning. Please shout you if this is the case. :P) -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130512205508.gb2...@jwilk.net
Re: On robustness of maintainer scripts
[Jakub Wilk, 2013-05-10] postinst [...] Proposed solution: 1) Wait until #671711 is fixed. 3) Make pycompile a shell script that does only two simple things: - write options it was called with to a file, say, /var/lib/python/pyX.Ycompile.todo; - use dpkg-trigger to trigger pythonX.Y. why not generate maintainer scripts without pycompile at all? I wanted to delegate as much as possible to interpreter packages, but your idea with temp. files is even better - maintainer scripts can look like this: | touch /usr/lib/pythonX.Y/dist-packages/namespace/__init__.py¹ | touch /usr/lib/pythonX.Z/dist-packages/namespace/__init__.py¹ | dpkg -L package | grep \.py$ | while read file | do | echo ${file} /var/lib/python/pyX.Ycompile.todo | echo ${file} /var/lib/python/pyX.Zcompile.todo | done [¹] namespace has to be created here as other packages might want to use given library in maintainer scipts, even without .pyc files (test -f ... || is probably missing here) (with some extra code to handle private directories, including rt* scripts, or any other logic from pycompile script, probably without exclude patterns or other features that are not really used) 2) Add code to pythonX.Y's postinst that'll process stuff from /var/lib/python/pyX.Ycompile.todo. Note that you don't need to move any complicated logic to the interpreter packages. All that the postinst would have to do is to call some script shipped in python-minimal. prerm = [...] Proposed solution: Get rid of /usr/bin/pyclean, so that the fallback code is activated. \o/ Just kidding. :) Not only that would almost certainly have unintended consequences, but also the fallback code doesn't look complete: it doesn't take care of namespaces. Actual proposed solution: Rewrite /usr/bin/pyclean in shell. again, removing /usr/bin/pyclean is not that bad idea. I like it and will try to provide example scripts for interpreter packages (and then generate prerm like in postinst example) -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130510222907.gb24...@p1otr.com
Re: On robustness of maintainer scripts
[Piotr Ożarowski, 2013-05-11] | dpkg -L package | grep \.py$ | while read file | do | echo ${file} /var/lib/python/pyX.Ycompile.todo | echo ${file} /var/lib/python/pyX.Zcompile.todo | done with some grep .../pythonX.Y/dist-packages of course -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130510223413.gc24...@p1otr.com