Re: Handling s390 libc ABI change in Debian
On 14/07/14 22:36, Aurelien Jarno wrote: Hi all, glibc 2.19 has changed the libc ABI on s390, more specifically the setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle some cases, but it doesn't work when a jmp_buf variable is embedded into a structure, as it changes the size of the structure. The result is that mixing programs or libraries built with 2.18 with ones built with 2.19 do not work anymore, usually they end up with a segmentation fault. Some persons from this list have experienced that with perl. We first thought it was limited to a few packages (even if all perl is already more than that), but as time goes more and more issues are found. libpng and gauche are also affected, the issue with mono is also likely due to this ABI change. According to upstream [3], the problem is that Debian doesn't do a mass rebuild, which is the strategy chosen by Red Hat to handle^Wworkaround this issue. This means some programs might segfault during the upgrade, or on partially upgraded systems. Now we have to chose a strategy for Debian. I see multiple options: 1) Ignore the issue and just rebuild (binNMU) the packages that seems affected when we discover them. This means partial upgrades will likely be broken, and that we might discover some broken packages only after the jessie release. 2) Rebuild (binNMU) all packages. This means partial upgrades will likely be broken. 3) Bump the soname of affected packages and rebuild their reverse dependencies. It is the solution that is currently being implemented for perl. It clearly won't scale if more broken packages (and even for libpng) are discovered as it requires a source upload and a transition handled by the release team. It also means breaking the ABI compatibility with other distributions. 4) Bump the libc soname to libc.so.6.1 and do a libc transition. This is probably what upstream should have done instead of breaking the ABI. This is a huge work though, and this also means breaking the ABI compatibility with other distributions. 5) Revert the ABI change. This is likely just postponing the problem as the change is required to support future hardware. This also means breaking the ABI compatibility with other distributions. 6) simply drop the s390x port and tell users to either use an other distribution or use Debian on other hardware. Status update: So in the end upstream has agreed to revert the change that broke the ABI, and that's what we will do as well. The biggest thing we need to revert with it is the perl ABI, but since we're going to update to perl 5.20 in a couple of weeks, we'll do the two at the same time to avoid having two perl transitions (one everywhere and another one on s390x). Emilio -- To UNSUBSCRIBE, email to debian-glibc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/53d6b4cb.4080...@debian.org
Re: Handling s390 libc ABI change in Debian
On Tue, Jul 15, 2014 at 07:18:39AM +0200, Aurelien Jarno wrote: I can follow up with a list affected packages, but we are slowly discovering them one by one, so it might takes time. So far we have: * Mixing modules/libraries built with pre-2.19 and 2.19 libc - perl - libpng * Using libc 2.19 without rebuilding anything: - gauche - mono I think it's pretty important for perl to keep working as much as is required for the system to upgrade itself. I'd be a bit less concerned (aside already broken binary compatibility) if the base system keeps working during the upgrade. We could conceivably document the breakage in the release upgrade notes, as long as updates can complete and suggest a reboot afterwards. It's a huge work for Debian, maybe not for other distribution, as it basically means we have to rebootstrap everything. This includes manual bootstrapping of self-dependent languages (haskell, gnat, ...) and manual handling of some dependencies loop. In addition it's something which hasn't been done since the libc5 transition, so we might discover some unexpected issues. Will it necessarily explode if both libcs are loaded into the same address space or only if the broken functionality is used? Would setjmp/longjmp only break if used across libc6/6.1 boundaries? Passing around an incompatible pthread struct seems bad, though. If this would work, a re-bootstrap would not necessarily be needed, I think? Kind regards and thanks Philipp Kern signature.asc Description: Digital signature
Re: Handling s390 libc ABI change in Debian
On Tue, Jul 15, 2014 at 1:18 AM, Aurelien Jarno aure...@debian.org wrote: On Mon, Jul 14, 2014 at 11:14:42PM -0400, Carlos O'Donell wrote: On Mon, Jul 14, 2014 at 4:36 PM, Aurelien Jarno aure...@debian.org wrote: glibc 2.19 has changed the libc ABI on s390, more specifically the setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle some cases, but it doesn't work when a jmp_buf variable is embedded into a structure, as it changes the size of the structure. The result is that mixing programs or libraries built with 2.18 with ones built with 2.19 do not work anymore, usually they end up with a segmentation fault. Some persons from this list have experienced that with perl. That is not true. This is an over generalization of the problem. You can use libraries built with 2.18 and 2.19 and they work just fine. I agree I probably a bit over exaggerated here, but the problem is real, breakages do happen, and some persons on this mailing list have already experienced them. The extent of the problem in correct language is listed here: https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes This seems to minimize the problem, listing only perl. In practice we have seen much more breakages, part of them being due to the change of the __pthread_unwind_buf_t struct. That is a change that nobody reported. You're the first to mention it and that does make it more serious. We have discussed this upstream and I agree that we need more versioning of the interfaces there to support the change fully. We first thought it was limited to a few packages (even if all perl is already more than that), but as time goes more and more issues are found. libpng and gauche are also affected, the issue with mono is also likely due to this ABI change. That is new information, and it is important for distributions to relay this information back upstream where the decision for a SO bump can be made. I can follow up with a list affected packages, but we are slowly discovering them one by one, so it might takes time. So far we have: * Mixing modules/libraries built with pre-2.19 and 2.19 libc - perl - libpng You can never support a mixed-ABI environment with versioning. You must update all of those packages at once. The best we could do is warn the user of the incompatibility at runtime and refuse to load the module via dlopen, or refuse to start the application at startup. * Using libc 2.19 without rebuilding anything: - gauche - mono This we believe to be pthread issues. According to upstream [3], the problem is that Debian doesn't do a mass rebuild, which is the strategy chosen by Red Hat to handle^Wworkaround this issue. This means some programs might segfault during the upgrade, or on partially upgraded systems. I apologize if you took what I wrote to mean that. I did not mean it was Debian's problem, but rather that Debian suffered the most because they don't do rebuilds. The two are orthogonal. You face a situation that is unique to the framework used to build the distribution. Please engage upstream to champion a SO name bump for libc for I think that would be the correct solution. That said as it is not something trivial and thus not done often, it's an opportunity to push for more ABI changes if some others are envisaged in the future. The problems are worse. I just tried to simulate this on x86-64 and there are serious problems. In most libraries you can load multiple different copies and it won't conflict. Here libc.so.6 and libc.so.7 or libc.so.6.1 all conflict in the same namespace and worse control aspects of the implementation like TLS. It doesn't work to bump the SONAME. We would have to implement a coordination framework amongst all the SONAME bumped libc's for all of the basic functionality that had to keep working. That would force future libcs to stay compatible internally with other libcs and that would be very difficult to maintain. I am starting to think that a tooling option to fail to load mixed-ABI objects is the only option, with user rebuilds happening after that. Now we have to chose a strategy for Debian. I see multiple options: 1) Ignore the issue and just rebuild (binNMU) the packages that seems affected when we discover them. This means partial upgrades will likely be broken, and that we might discover some broken packages only after the jessie release. 2) Rebuild (binNMU) all packages. This means partial upgrades will likely be broken. 3) Bump the soname of affected packages and rebuild their reverse dependencies. It is the solution that is currently being implemented for perl. It clearly won't scale if more broken packages (and even for libpng) are discovered as it requires a source upload and a transition handled by the release team. It also means breaking the ABI compatibility with other distributions. 4) Bump the libc soname to libc.so.6.1 and do a libc
Re: Handling s390 libc ABI change in Debian
On Tue, Jul 15, 2014 at 09:21:30AM +0200, Philipp Kern wrote: On Tue, Jul 15, 2014 at 07:18:39AM +0200, Aurelien Jarno wrote: I can follow up with a list affected packages, but we are slowly discovering them one by one, so it might takes time. So far we have: * Mixing modules/libraries built with pre-2.19 and 2.19 libc - perl - libpng * Using libc 2.19 without rebuilding anything: - gauche - mono I think it's pretty important for perl to keep working as much as is required for the system to upgrade itself. I'd be a bit less concerned (aside already broken binary compatibility) if the base system keeps working during the upgrade. It might not be easy to ensure the upgrade process works correctly. For example in mono case, as soon as a new libc is installed, mono stops working, and installing/upgrading a mono package would fail as mono is called in the postinst (this is bug#751171). We have to avoid this by using strict dependencies to make sure the packages are installed in the right order, but we can't guarantee to detect and handle all cases. That means some upgrades might break. -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net -- To UNSUBSCRIBE, email to debian-glibc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140715075003.gd32...@hall.aurel32.net
Re: Handling s390 libc ABI change in Debian
On Tue, Jul 15, 2014 at 03:49:04AM -0400, Carlos O'Donell wrote: On Tue, Jul 15, 2014 at 1:18 AM, Aurelien Jarno aure...@debian.org wrote: On Mon, Jul 14, 2014 at 11:14:42PM -0400, Carlos O'Donell wrote: On Mon, Jul 14, 2014 at 4:36 PM, Aurelien Jarno aure...@debian.org wrote: glibc 2.19 has changed the libc ABI on s390, more specifically the setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle some cases, but it doesn't work when a jmp_buf variable is embedded into a structure, as it changes the size of the structure. The result is that mixing programs or libraries built with 2.18 with ones built with 2.19 do not work anymore, usually they end up with a segmentation fault. Some persons from this list have experienced that with perl. That is not true. This is an over generalization of the problem. You can use libraries built with 2.18 and 2.19 and they work just fine. I agree I probably a bit over exaggerated here, but the problem is real, breakages do happen, and some persons on this mailing list have already experienced them. The extent of the problem in correct language is listed here: https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes This seems to minimize the problem, listing only perl. In practice we have seen much more breakages, part of them being due to the change of the __pthread_unwind_buf_t struct. That is a change that nobody reported. You're the first to mention it and that does make it more serious. We have discussed this upstream and I agree that we need more versioning of the interfaces there to support the change fully. We first thought it was limited to a few packages (even if all perl is already more than that), but as time goes more and more issues are found. libpng and gauche are also affected, the issue with mono is also likely due to this ABI change. That is new information, and it is important for distributions to relay this information back upstream where the decision for a SO bump can be made. I can follow up with a list affected packages, but we are slowly discovering them one by one, so it might takes time. So far we have: * Mixing modules/libraries built with pre-2.19 and 2.19 libc - perl - libpng You can never support a mixed-ABI environment with versioning. You must update all of those packages at once. The best we could do is warn the user of the incompatibility at runtime and refuse to load the module via dlopen, or refuse to start the application at startup. For perl we handled that using dependencies in the package manager, and we can probably add some more checks for user modules. That said that do not scale if we discover more and more affected packages. This is my fear so far. * Using libc 2.19 without rebuilding anything: - gauche - mono This we believe to be pthread issues. Yes, this is the pthread issue. It's a huge work for Debian, maybe not for other distribution, as it basically means we have to rebootstrap everything. This includes manual bootstrapping of self-dependent languages (haskell, gnat, ...) and manual handling of some dependencies loop. In addition it's something which hasn't been done since the libc5 transition, so we might discover some unexpected issues. Why do you have to do that? Is it just like for rpm where the packaging system encodes the SONAME as a dependency? We would also need a manual bootstrap in Fedora because of this issue. I assumed that both libc can't be used simultaneously, so that's basically like bootstrapping a new architecture, except the manual bootstrapping of self-dependent languages can be done more easily. Cheers, Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net -- To UNSUBSCRIBE, email to debian-glibc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140715083054.gk1...@hall.aurel32.net
Handling s390 libc ABI change in Debian
Hi all, glibc 2.19 has changed the libc ABI on s390, more specifically the setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle some cases, but it doesn't work when a jmp_buf variable is embedded into a structure, as it changes the size of the structure. The result is that mixing programs or libraries built with 2.18 with ones built with 2.19 do not work anymore, usually they end up with a segmentation fault. Some persons from this list have experienced that with perl. We first thought it was limited to a few packages (even if all perl is already more than that), but as time goes more and more issues are found. libpng and gauche are also affected, the issue with mono is also likely due to this ABI change. According to upstream [3], the problem is that Debian doesn't do a mass rebuild, which is the strategy chosen by Red Hat to handle^Wworkaround this issue. This means some programs might segfault during the upgrade, or on partially upgraded systems. Now we have to chose a strategy for Debian. I see multiple options: 1) Ignore the issue and just rebuild (binNMU) the packages that seems affected when we discover them. This means partial upgrades will likely be broken, and that we might discover some broken packages only after the jessie release. 2) Rebuild (binNMU) all packages. This means partial upgrades will likely be broken. 3) Bump the soname of affected packages and rebuild their reverse dependencies. It is the solution that is currently being implemented for perl. It clearly won't scale if more broken packages (and even for libpng) are discovered as it requires a source upload and a transition handled by the release team. It also means breaking the ABI compatibility with other distributions. 4) Bump the libc soname to libc.so.6.1 and do a libc transition. This is probably what upstream should have done instead of breaking the ABI. This is a huge work though, and this also means breaking the ABI compatibility with other distributions. 5) Revert the ABI change. This is likely just postponing the problem as the change is required to support future hardware. This also means breaking the ABI compatibility with other distributions. 6) simply drop the s390x port and tell users to either use an other distribution or use Debian on other hardware. Any opinion? Any other ideas how to handle that? Regards, Aurelien [1] http://sourceware.org/git/?p=glibc.git;a=commitdiff;h=ee4ec1d7f9bdbdfc87117133478cfb2f6653e65c [2] https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes [3] https://sourceware.org/ml/libc-alpha/2014-07/msg00316.html -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: Digital signature
Re: Handling s390 libc ABI change in Debian
On Mon, Jul 14, 2014 at 4:36 PM, Aurelien Jarno aure...@debian.org wrote: glibc 2.19 has changed the libc ABI on s390, more specifically the setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle some cases, but it doesn't work when a jmp_buf variable is embedded into a structure, as it changes the size of the structure. The result is that mixing programs or libraries built with 2.18 with ones built with 2.19 do not work anymore, usually they end up with a segmentation fault. Some persons from this list have experienced that with perl. That is not true. This is an over generalization of the problem. You can use libraries built with 2.18 and 2.19 and they work just fine. The extent of the problem in correct language is listed here: https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes We first thought it was limited to a few packages (even if all perl is already more than that), but as time goes more and more issues are found. libpng and gauche are also affected, the issue with mono is also likely due to this ABI change. That is new information, and it is important for distributions to relay this information back upstream where the decision for a SO bump can be made. According to upstream [3], the problem is that Debian doesn't do a mass rebuild, which is the strategy chosen by Red Hat to handle^Wworkaround this issue. This means some programs might segfault during the upgrade, or on partially upgraded systems. I apologize if you took what I wrote to mean that. I did not mean it was Debian's problem, but rather that Debian suffered the most because they don't do rebuilds. The two are orthogonal. You face a situation that is unique to the framework used to build the distribution. Please engage upstream to champion a SO name bump for libc for Now we have to chose a strategy for Debian. I see multiple options: 1) Ignore the issue and just rebuild (binNMU) the packages that seems affected when we discover them. This means partial upgrades will likely be broken, and that we might discover some broken packages only after the jessie release. 2) Rebuild (binNMU) all packages. This means partial upgrades will likely be broken. 3) Bump the soname of affected packages and rebuild their reverse dependencies. It is the solution that is currently being implemented for perl. It clearly won't scale if more broken packages (and even for libpng) are discovered as it requires a source upload and a transition handled by the release team. It also means breaking the ABI compatibility with other distributions. 4) Bump the libc soname to libc.so.6.1 and do a libc transition. This is probably what upstream should have done instead of breaking the ABI. This is a huge work though, and this also means breaking the ABI compatibility with other distributions. 5) Revert the ABI change. This is likely just postponing the problem as the change is required to support future hardware. This also means breaking the ABI compatibility with other distributions. 6) simply drop the s390x port and tell users to either use an other distribution or use Debian on other hardware. Any opinion? Any other ideas how to handle that? Option (6) is the nuclear option, and clearly a little excessive given the situation. If user's install from an installer they get a perfectly working system. Punishing those users because partial upgrades don't work seems excessive. Option (5) postpones the problem until newer s390 hardware arrives. Option (4) is likely what upstream should have done. Why do you think it's a huge amount of work? If all s390-supporting distributions agree then we just change the SO name? Option (3), (2) or (1) all look like reasonable fire and forget options that require users of s390 to simply move forward with their installs to full new versions of Debian. It's an option for RHEL because we don't allow partial-upgrades and we are working with users to notify them of the breakage during a major RHEL upgrade e.g. RHEL7 to RHEL8. In my opinion it's not too late to do option (4). Cheers, Carlos. -- To UNSUBSCRIBE, email to debian-glibc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/CAE2sS1hgdMH4F42iFw-_MrOk1q2+2=qhm1hjuqfktmsfwyq...@mail.gmail.com
Re: Handling s390 libc ABI change in Debian
On Mon, Jul 14, 2014 at 11:14:42PM -0400, Carlos O'Donell wrote: On Mon, Jul 14, 2014 at 4:36 PM, Aurelien Jarno aure...@debian.org wrote: glibc 2.19 has changed the libc ABI on s390, more specifically the setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle some cases, but it doesn't work when a jmp_buf variable is embedded into a structure, as it changes the size of the structure. The result is that mixing programs or libraries built with 2.18 with ones built with 2.19 do not work anymore, usually they end up with a segmentation fault. Some persons from this list have experienced that with perl. That is not true. This is an over generalization of the problem. You can use libraries built with 2.18 and 2.19 and they work just fine. I agree I probably a bit over exaggerated here, but the problem is real, breakages do happen, and some persons on this mailing list have already experienced them. The extent of the problem in correct language is listed here: https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes This seems to minimize the problem, listing only perl. In practice we have seen much more breakages, part of them being due to the change of the __pthread_unwind_buf_t struct. We first thought it was limited to a few packages (even if all perl is already more than that), but as time goes more and more issues are found. libpng and gauche are also affected, the issue with mono is also likely due to this ABI change. That is new information, and it is important for distributions to relay this information back upstream where the decision for a SO bump can be made. I can follow up with a list affected packages, but we are slowly discovering them one by one, so it might takes time. So far we have: * Mixing modules/libraries built with pre-2.19 and 2.19 libc - perl - libpng * Using libc 2.19 without rebuilding anything: - gauche - mono According to upstream [3], the problem is that Debian doesn't do a mass rebuild, which is the strategy chosen by Red Hat to handle^Wworkaround this issue. This means some programs might segfault during the upgrade, or on partially upgraded systems. I apologize if you took what I wrote to mean that. I did not mean it was Debian's problem, but rather that Debian suffered the most because they don't do rebuilds. The two are orthogonal. You face a situation that is unique to the framework used to build the distribution. Please engage upstream to champion a SO name bump for libc for I think that would be the correct solution. That said as it is not something trivial and thus not done often, it's an opportunity to push for more ABI changes if some others are envisaged in the future. Now we have to chose a strategy for Debian. I see multiple options: 1) Ignore the issue and just rebuild (binNMU) the packages that seems affected when we discover them. This means partial upgrades will likely be broken, and that we might discover some broken packages only after the jessie release. 2) Rebuild (binNMU) all packages. This means partial upgrades will likely be broken. 3) Bump the soname of affected packages and rebuild their reverse dependencies. It is the solution that is currently being implemented for perl. It clearly won't scale if more broken packages (and even for libpng) are discovered as it requires a source upload and a transition handled by the release team. It also means breaking the ABI compatibility with other distributions. 4) Bump the libc soname to libc.so.6.1 and do a libc transition. This is probably what upstream should have done instead of breaking the ABI. This is a huge work though, and this also means breaking the ABI compatibility with other distributions. 5) Revert the ABI change. This is likely just postponing the problem as the change is required to support future hardware. This also means breaking the ABI compatibility with other distributions. 6) simply drop the s390x port and tell users to either use an other distribution or use Debian on other hardware. Any opinion? Any other ideas how to handle that? Option (6) is the nuclear option, and clearly a little excessive given the situation. If user's install from an installer they get a perfectly working system. Punishing those users because partial upgrades don't work seems excessive. If we are not able to solve this problem by ensuring all the packages in the next release are using a consistent ABI, I think it is an option to consider. It's probably better being honest with the users telling them we fail than shipping a half broken release. Consider that we have limited human resources to maintain this port. Option (5) postpones the problem until newer s390 hardware arrives. Option (4) is likely what upstream should have done. Why do you think it's a huge amount of work? If all s390-supporting distributions agree then we just change the SO name? It's a
Re: Handling s390 libc ABI change in Debian
On Tue, Jul 15, 2014 at 1:18 PM, Aurelien Jarno wrote: It's a huge work for Debian, maybe not for other distribution, as it basically means we have to rebootstrap everything. This includes manual bootstrapping of self-dependent languages (haskell, gnat, ...) and manual handling of some dependencies loop. In addition it's something which hasn't been done since the libc5 transition, so we might discover some unexpected issues. Helmut Grohne and others have been working on automated rebootstrap of the Debian archive. The rebootstrap_s390x_gcc49_nobiarch jenkins job seems to be in good shape currently. That is by no means a full archive bootstrap, which requires more things. https://wiki.debian.org/HelmutGrohne/rebootstrap https://jenkins.debian.net/view/rebootstrap/ https://wiki.debian.org/DebianBootstrap https://wiki.debian.org/Sprints/2014/BootstrapSprint -- bye, pabs https://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-glibc-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/CAKTje6GUdNfY4ixidAFae6YheF41jo=yY7uYmne0TXLWPiHY=w...@mail.gmail.com
