Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote: The other thing we needed to do was completely remove the use of or building of binpkgs during the update_seed stage. Portage has no capability to check binpkg linking to ensure the binpkg was properly usable. Can somebody actually please implement this, to run before the binpkg merge phase? -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/10/2013 10:03 PM, Robin H. Johnson wrote: On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote: The other thing we needed to do was completely remove the use of or building of binpkgs during the update_seed stage. Portage has no capability to check binpkg linking to ensure the binpkg was properly usable. Can somebody actually please implement this, to run before the binpkg merge phase? Please be more specific, this is currently implemented... - -Zero -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJR3ihMAAoJEKXdFCfdEflKH4sP/1dswIQytHVOXtI6mqzntqjy 9rBl2tkBft2gdQp7Vru3d4s0HiicpeLImSyvIM8H2FVmbEVutMwOOUWp2tllOUNh bb98BnrAoSMz4GEPtcS2d0vmZ3HdDj99mrnkOOWcqvHRPMn/J0KogAyrCVRa0+Pb bNEo2dU5g4Jegs8gl4l1ilH0hIyb7uqurwjd5G8NayJIs0U5gIpTHZzPvRJo0B98 tnzEsoSTY0JekgSGC+8xgR4ijmR0AicF5nCciKUjFf7OjEDitK6sF6g3m92zM8hU fjPLTW0vjR63KbtOJNgw992DpQoUfRj2jYn4ErjDB1jKz5+xvLy0+rpwZPYA4hgf QNck4OnLRqK6rMy45Tzu58gfUsH1GIdmdH0JJ0x2rBLu89GHB4HSR+UlE5ugxmvj uzs2ZJ8MPIT7mYnrqJ8XbiMzDmkvspkixbi6YpbiNCi76R08ljNXFjT0Ju5nI29u W+6p/we4FMPLo47W5zSAS4Csa0reIayaIpJTbeyfHVe88DcS6jO+JAs416mod9ec 1SYogk+SFrmFC0PyW9uQeP0gE12U20bw0NCI0R5+PmXIs54jrowIJgjB8hPH8Cpz df7+ZNGuDaOqQnJT5j1EwzfAPEGVHmHtIaMu3bRMv0EJZj3WMa1rABtFFX+65S1x dz87xG1hys5ZZmz8IzOQ =+Lzs -END PGP SIGNATURE-
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Wed, Jul 10, 2013 at 8:36 PM, Rick Zero_Chaos Farina zeroch...@gentoo.org wrote: On 07/10/2013 10:03 PM, Robin H. Johnson wrote: On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote: The other thing we needed to do was completely remove the use of or building of binpkgs during the update_seed stage. Portage has no capability to check binpkg linking to ensure the binpkg was properly usable. Can somebody actually please implement this, to run before the binpkg merge phase? Please be more specific, this is currently implemented... He's talking about portage, not catalyst.
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Wed, Jul 10, 2013 at 11:36:44PM -0400, Rick Zero_Chaos Farina wrote: On 07/10/2013 10:03 PM, Robin H. Johnson wrote: On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote: The other thing we needed to do was completely remove the use of or building of binpkgs during the update_seed stage. Portage has no capability to check binpkg linking to ensure the binpkg was properly usable. Can somebody actually please implement this, to run before the binpkg merge phase? Please be more specific, this is currently implemented... I was partially responding to dolsen's comment that it wasn't capable, as well as a discussion I had with somebody on the ChromeOS team a few months ago. In the ChromeOS case, if a library was removed off the system, but Portage still thought it was there, and you went to install a binpkg that had an ELF dependency on the removed library, you'd have broken binaries on the system. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On 07/10/2013 08:43 PM, Robin H. Johnson wrote: On Wed, Jul 10, 2013 at 11:36:44PM -0400, Rick Zero_Chaos Farina wrote: On 07/10/2013 10:03 PM, Robin H. Johnson wrote: On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote: The other thing we needed to do was completely remove the use of or building of binpkgs during the update_seed stage. Portage has no capability to check binpkg linking to ensure the binpkg was properly usable. Can somebody actually please implement this, to run before the binpkg merge phase? Please be more specific, this is currently implemented... I was partially responding to dolsen's comment that it wasn't capable, as well as a discussion I had with somebody on the ChromeOS team a few months ago. In the ChromeOS case, if a library was removed off the system, but Portage still thought it was there, and you went to install a binpkg that had an ELF dependency on the removed library, you'd have broken binaries on the system. Ideally, this would be handled entirely by EAPI 5 slot-operator dependencies. However, it seems like we'll need some form of sub-slot pass-through [1] and/or sub-slot dictionaries [2] before we'll be able to achieve complete adoption of slot-operators. Meanwhile, adding a preinst sanity check seems like a good idea. The LinkageMap class that's used for preserve-libs already has all of the info that we need about installed libraries, so it's just a matter of checking the binary package's NEEDED.ELF.2 entries against that. [1] https://bugs.gentoo.org/show_bug.cgi?id=449094 [2] https://bugs.gentoo.org/show_bug.cgi?id=462138 -- Thanks, Zac
[gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Fri, Jul 5, 2013 at 7:18 PM, Rick Zero_Chaos Farina zeroch...@gentoo.org wrote: When we then move onto stage 2, it uses just the packages built during stage1 (/tmp/stage1root becomes /). This means, if seed stage has mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed. To combat this kind of failure we are currently running emerge --update - --deep --newuse --complete-graph --rebuild-if-new-ver gcc which could just be emerge --update gcc if eapi 5 subslots were used properly. The best solution to this is, and has always been, just updating gcc's deps during update_seed. Or am I misremembering something? As far as I know, you don't need to waste time rebuilding the seed stage's gcc, since gcc is rebuilt in stage2 and then everything is built by it in stage3.
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Sat, 2013-07-06 at 00:08 -0700, Matt Turner wrote: On Fri, Jul 5, 2013 at 7:18 PM, Rick Zero_Chaos Farina zeroch...@gentoo.org wrote: When we then move onto stage 2, it uses just the packages built during stage1 (/tmp/stage1root becomes /). This means, if seed stage has mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed. To combat this kind of failure we are currently running emerge --update - --deep --newuse --complete-graph --rebuild-if-new-ver gcc which could just be emerge --update gcc if eapi 5 subslots were used properly. The best solution to this is, and has always been, just updating gcc's deps during update_seed. Or am I misremembering something? As far as I know, you don't need to waste time rebuilding the seed stage's gcc, since gcc is rebuilt in stage2 and then everything is built by it in stage3. Yes, it does need to be rebuilt if key deps are updated. The gcc produced in the stage1 is broken, so won't run to rebuild itself during the stage2 run. Also the --rebuild-if-new-ver gcc only rebuilds gcc if it's deps are updated to new versions (likely breaking lib linkage), it is not unconditional. It is not as accurate as subslots, so may rebuild more often than possibly needed. But it is better than having unreliable stage builds, not knowing when they may break. -- Brian Dolbec dol...@gentoo.org
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/06/2013 03:08 AM, Matt Turner wrote: On Fri, Jul 5, 2013 at 7:18 PM, Rick Zero_Chaos Farina zeroch...@gentoo.org wrote: When we then move onto stage 2, it uses just the packages built during stage1 (/tmp/stage1root becomes /). This means, if seed stage has mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed. To combat this kind of failure we are currently running emerge --update - --deep --newuse --complete-graph --rebuild-if-new-ver gcc which could just be emerge --update gcc if eapi 5 subslots were used properly. The best solution to this is, and has always been, just updating gcc's deps during update_seed. Or am I misremembering something? You are misremembering that we are using preserve_libs to save our butts when mpc is updated and gcc is still linked to the old mpc. I feel very uncomfortable as the recommendation of preserve-libs is to remerge as soon as possible not build a whole system like this. Is there an actual failure here? Not that I've seen yet, but it's an awkward way to build in my opinion. - -Zero As far as I know, you don't need to waste time rebuilding the seed stage's gcc, since gcc is rebuilt in stage2 and then everything is built by it in stage3. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJR2DeYAAoJEKXdFCfdEflKDwwP/1Pr2eB9GjSKncEabiB9WkEb eziSaccKcsmjdXq9Svg0dfTM9m9rgroK0iM15IWLHhbAoU/5beVPv4bOYVYejYkP NBMWp2+MoIE7VRhziFToj7tHxTnXsUg1l3dMFWECewWpMewo9lZw1eYTn/iaUGaI tkfNi7iQV9PvfknUgtQZ8lfgSUjUz2CdtjCyjaoMpO+vls+gVvH74vGETIMMrHWr CN7iMfx7F6FGpc1+FxZ0CJ1zKSifY/1R+dLABass8xaLRMPNqTIpm8b37xD2tHOA f2pfzCIgkwLEo8moJrmkl21CqC2CjqZ0HPqd3dd/wSTg2x1ccslNHVOUf8+mZu1I 4zwJUwS7e2w8rxcq/UTu9x3J18D2doFjLg1mLUtWgmptn9Tydr/GYL+yYJei0yK+ MiADpdK+UI5frUo1B8bZ+Gs0N5IIh2pcGkjdupz4HXRAeD+2VN5G0HBpTZ8I4vNI rK9wRQN1iyxb4sn0Wr0n+GwSlxyTao6yuUSJwT5nfD1k9gSGI9Zh7tERlD/D9ceN Vfvv0No+ikh47TDjm3hSmz2fdbTT5vxKecnXT72EpYQwIZFoVMo7tnRVwxd4gBbM Fx4LDUaSn3ommMBZF/jRt5zsn+cGMB/7qHkNo1DPIbKgXgExEQWRz7Lxhy8Hm4er YYVhPl4Zhc0/BCBwBpK/ =zakW -END PGP SIGNATURE-
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Sat, Jul 6, 2013 at 12:46 AM, Brian Dolbec dol...@gentoo.org wrote: Yes, it does need to be rebuilt if key deps are updated. The gcc produced in the stage1 is broken, so won't run to rebuild itself during the stage2 run. But we keep the old libs around via preserve_libs, and once stage1 is done we effectively throw the seed stage out.
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Sat, Jul 6, 2013 at 8:28 AM, Rick Zero_Chaos Farina zeroch...@gentoo.org wrote: You are misremembering that we are using preserve_libs to save our butts when mpc is updated and gcc is still linked to the old mpc. I feel very uncomfortable as the recommendation of preserve-libs is to remerge as soon as possible not build a whole system like this. Is there an actual failure here? Not that I've seen yet, but it's an awkward way to build in my opinion. Keeping the old libs seems perfectly fine, since it's in a seed stage that we don't care about after stage1 is complete. An unnecessary build of gcc may not matter much on a relatively fast amd64, but it's going to be a pain on a bunch of slower architectures. And on mips/multilib it'll be even worse since we get to build the libraries for three ABIs.
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Sat, 2013-07-06 at 09:36 -0700, Matt Turner wrote: On Sat, Jul 6, 2013 at 12:46 AM, Brian Dolbec dol...@gentoo.org wrote: Yes, it does need to be rebuilt if key deps are updated. The gcc produced in the stage1 is broken, so won't run to rebuild itself during the stage2 run. But we keep the old libs around via preserve_libs, and once stage1 is done we effectively throw the seed stage out. It could be that I missed that combination of changes that could occur during my testing. But I am quite sure that it failed for me. You could always do some re-runs of those tests using an older seed stage built against libmpc.so.1.
Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On 07/06/2013 12:39 PM, Matt Turner wrote: On Sat, Jul 6, 2013 at 8:28 AM, Rick Zero_Chaos Farina zeroch...@gentoo.org wrote: You are misremembering that we are using preserve_libs to save our butts when mpc is updated and gcc is still linked to the old mpc. I feel very uncomfortable as the recommendation of preserve-libs is to remerge as soon as possible not build a whole system like this. Is there an actual failure here? Not that I've seen yet, but it's an awkward way to build in my opinion. Keeping the old libs seems perfectly fine, since it's in a seed stage that we don't care about after stage1 is complete. An unnecessary build of gcc may not matter much on a relatively fast amd64, but it's going to be a pain on a bunch of slower architectures. And on mips/multilib it'll be even worse since we get to build the libraries for three ABIs. Thanks Matt, I was going to make that point. ~3 days to recompile world on the lemote yeelong. -- Anthony G. Basile, Ph. D. Chair of Information Technology D'Youville College Buffalo, NY 14201 (716) 829-8197
[gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Fri, 2013-07-05 at 22:18 -0400, Rick Zero_Chaos Farina wrote: On 07/05/2013 09:41 PM, Ryan Hill wrote: On Fri, 05 Jul 2013 15:47:08 -0400 Rick \Zero_Chaos\ Farina zeroch...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paweł was nice enough to write a patch for us to get toolchain.eclass up to EAPI 5. I believe it still needs some pieces like prefix support and I haven't reviewed it in depth but it looks good so far (and much simpler than I thought (oops)). I'm planning on moving up an EAPI at a time, bumping it whenever we could use new features or people start hucking fruit. I would be forever in your debt if toolchain were on eapi 5 and had proper subslot deps. What use case are you encountering? I hadn't planned on using subslots, but I'm open to good reasons/bribery. Livecd builds work like this: stage3 tarball is unpacked, then toolchain (minimum package set) is built with ROOT=/tmp/stage1root and is linked to the original (seed stage) while being built. When we then move onto stage 2, it uses just the packages built during stage1 (/tmp/stage1root becomes /). This means, if seed stage has mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed. To combat this kind of failure we are currently running emerge --update --deep --newuse --complete-graph --rebuild-if-new-ver gcc which could just be emerge --update gcc if eapi 5 subslots were used properly. Please forgive me if any of these details aren't perfect, I'm using my best recollection of the most recent issue which happened when mpc was bumped a few months ago. I am available on irc quite a bit if you would like to discuss, however, if you want to keep it on the ML we really need to move this to -dev. Thanks, Zero Continuing this on -dev mail list. The other thing we needed to do was completely remove the use of or building of binpkgs during the update_seed stage. Portage has no capability to check binpkg linking to ensure the binpkg was properly usable. EAPI 5 subslots also put that info into the DEPENDS so portage then considers that information. It would then reject a binpkg properly and build from source due to the abi mismatch. I might also add that using broken binpkgs in catalyst (like the mpc update) cause delayed breakage, making trouble shooting the problem more difficult. With proper use of subslots for base system pkgs, binpkgs can be re-used safely, saving both build and turn around time to get fully working stages and livecd's etc. in catalyst released. The same holds true for user systems that build and try to reuse binkgs.
[gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014
On Fri, 05 Jul 2013 21:27:46 -0700 Brian Dolbec dol...@gentoo.org wrote: On Fri, 2013-07-05 at 22:18 -0400, Rick Zero_Chaos Farina wrote: On 07/05/2013 09:41 PM, Ryan Hill wrote: On Fri, 05 Jul 2013 15:47:08 -0400 Rick \Zero_Chaos\ Farina zeroch...@gentoo.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Paweł was nice enough to write a patch for us to get toolchain.eclass up to EAPI 5. I believe it still needs some pieces like prefix support and I haven't reviewed it in depth but it looks good so far (and much simpler than I thought (oops)). I'm planning on moving up an EAPI at a time, bumping it whenever we could use new features or people start hucking fruit. I would be forever in your debt if toolchain were on eapi 5 and had proper subslot deps. What use case are you encountering? I hadn't planned on using subslots, but I'm open to good reasons/bribery. Livecd builds work like this: stage3 tarball is unpacked, then toolchain (minimum package set) is built with ROOT=/tmp/stage1root and is linked to the original (seed stage) while being built. When we then move onto stage 2, it uses just the packages built during stage1 (/tmp/stage1root becomes /). This means, if seed stage has mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed. To combat this kind of failure we are currently running emerge --update --deep --newuse --complete-graph --rebuild-if-new-ver gcc which could just be emerge --update gcc if eapi 5 subslots were used properly. Please forgive me if any of these details aren't perfect, I'm using my best recollection of the most recent issue which happened when mpc was bumped a few months ago. I am available on irc quite a bit if you would like to discuss, however, if you want to keep it on the ML we really need to move this to -dev. Thanks, Zero Continuing this on -dev mail list. The other thing we needed to do was completely remove the use of or building of binpkgs during the update_seed stage. Portage has no capability to check binpkg linking to ensure the binpkg was properly usable. EAPI 5 subslots also put that info into the DEPENDS so portage then considers that information. It would then reject a binpkg properly and build from source due to the abi mismatch. I might also add that using broken binpkgs in catalyst (like the mpc update) cause delayed breakage, making trouble shooting the problem more difficult. With proper use of subslots for base system pkgs, binpkgs can be re-used safely, saving both build and turn around time to get fully working stages and livecd's etc. in catalyst released. The same holds true for user systems that build and try to reuse binkgs. Thanks, these are good reasons. -- Ryan Hillpsn: dirtyepic_sk gcc-porting/toolchain/wxwidgets @ gentoo.org 47C3 6D62 4864 0E49 8E9E 7F92 ED38 BD49 957A 8463 signature.asc Description: PGP signature
