Re: Fedora modularity and cyclic dependencies
And of course, the seed is not technically required, everything could be done with a generic * chain build * chain build remainer with bootstrap * chain build bootstrapped builds normally * chain build everything The seed just saves days of processing build farm side -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
Le mardi 16 avril 2019 à 15:39 +0100, Mat Booth a écrit : > > Why wouldn't this work for you? You would just rebuild the module a > second time without the bootstrap flag set. This is fairly standard > process for packages that buildrequire themselves and not unique to > golang. I may be wrong, but does not seem to scale to the level needed by Go. Go is 600-900 interdependent packages with multiple cycles (some that span hundreds of packages, other that cross one another in the same packages). Bootstrapping Go, that will soon be needed because ustream switches to a new payload format in August, and the new payload tools refuse to read previous formats is takes this kind of execution plan: 1. try to chain build a seed of ~86 packages in normal mode 2. for all the seed packages that failed in normal mode, chain build in bootstrap mode (with the Fedora bootstrap conditional) 3. once this is ok, chain buil all the packages that required bootstrap, in normal mode → unfortunately a couple of packages will still fail they participate in cycles too far-ranging to be included in the seed 4. with this seed of almost-completely bootstraped packages you chain build the remaining 600+ packages in normal mode 5. when chain building stops making progress you try chain building all the packages that didn't pass in bootstrap mode 6. if that is successfull you try chain-building all the packages that required the bootstrap pass in normal mode 7. if that is successful (everything has neen built in normal mode, though some may have been built against bootstraped packages) you take the 600-900 packages and re chain build all of them to make sure the result is self-hosting. A package built in bootstrap mode may not have any particular bootstrap logic, but just be in the path of a cycle that needs bootstrapping of one or several of the cycle links And you won't have an explicit list of X needs to beb built before Y, A needs bootstrapping before B, because the dep graph changes regularly, you need chain building logic to discover if your bootstrapping logic still works. And even if it still works the whole process takes days/weeks (depending if you do it on a fast build farm or not, depending if you do all build arches or not) THAT’s what bootstraping means in a Go context. -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
On Tue, 16 Apr 2019 at 14:14, Robert-André Mauchin wrote: > > On Tuesday, 16 April 2019 02:46:49 CEST Mat Booth wrote: > > > > Yes, there is a buildopts section in the yaml file where you can specify > > rpm macros. > > > > Here is the documentation: > > > > https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modu > > les/#_build_macros_optional > > > > Here is an example where I needed to set --with bootstrap in my module: > > > > https://src.fedoraproject.org/modules/tycho/blob/df837b8793fe460d2c7e72ab6d6 > > 38a0f6e9f47a7/f/tycho.yaml#_76 > > > > > Thanks for the tip but that wouldn't work for Golang. When we bootstrap we > disable certain codepath to workaround the cycle, but we still need to > reenable them later to build the final binary. So we need to do both a > bootstrap build, and then a non-bootstrap build to get all the functionality. > Why wouldn't this work for you? You would just rebuild the module a second time without the bootstrap flag set. This is fairly standard process for packages that buildrequire themselves and not unique to golang. -- Mat Booth http://fedoraproject.org/get-fedora ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
On Tue, Apr 16, 2019 at 3:14 PM Robert-André Mauchin wrote: > On Tuesday, 16 April 2019 02:46:49 CEST Mat Booth wrote: > > > > Yes, there is a buildopts section in the yaml file where you can specify > > rpm macros. > > > > Here is the documentation: > > > > > https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modu > > les/#_build_macros_optional > > > > Here is an example where I needed to set --with bootstrap in my module: > > > > > https://src.fedoraproject.org/modules/tycho/blob/df837b8793fe460d2c7e72ab6d6 > > 38a0f6e9f47a7/f/tycho.yaml#_76 > > > > > Thanks for the tip but that wouldn't work for Golang. When we bootstrap we > disable certain codepath to workaround the cycle, but we still need to > reenable them later to build the final binary. So we need to do both a > bootstrap build, and then a non-bootstrap build to get all the > functionality. > This is quite common. One way to approach is is to have two modules, for example: * golang-bootstrap — with the bootstrap macros set, to build the first pass * golang — the final one, it would build-require the golang-bootstrap module Here is an example with Perl: * perl-bootstrap: https://src.fedoraproject.org/modules/perl-bootstrap/blob/5.26/f/perl-bootstrap.yaml * perl: https://src.fedoraproject.org/modules/perl/blob/5.26/f/perl.yaml Would that work for you? > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > -- Adam Šamalík --- Senior Software Engineer Red Hat ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
On Tue, Apr 16, 2019 at 11:11 AM Petr Pisar wrote: > > On 2019-04-16, Igor Gnatenko wrote: > > You can do something like: > > > > rpms: > > foo-bootstrap: > > name: foo > > bar: > > buildafter: > > - foo-bootstrap > > foo: > > buildafter: > > - bar > > > > How does foo.spec get to know that it should enable bootstrapping in > foo-bootstrap component, but disable it in foo component? You can use different refs (and therefore different spec files) for foo and foo-bootstrap. -- Mikolaj Izdebski ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
Le 2019-04-16 15:06, Robert-André Mauchin a écrit : On Tuesday, 16 April 2019 02:46:49 CEST Mat Booth wrote: Yes, there is a buildopts section in the yaml file where you can specify rpm macros. Here is the documentation: https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modu les/#_build_macros_optional Here is an example where I needed to set --with bootstrap in my module: https://src.fedoraproject.org/modules/tycho/blob/df837b8793fe460d2c7e72ab6d6 38a0f6e9f47a7/f/tycho.yaml#_76 Thanks for the tip but that wouldn't work for Golang. When we bootstrap we disable certain codepath to workaround the cycle, but we still need to reenable them later to build the final binary. So we need to do both a bootstrap build, and then a non-bootstrap build to get all the functionality. And that BTW is not Go specific at all. That's the usual bootstrap definition within Fedora. -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
On Tuesday, 16 April 2019 02:46:49 CEST Mat Booth wrote: > > Yes, there is a buildopts section in the yaml file where you can specify > rpm macros. > > Here is the documentation: > > https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modu > les/#_build_macros_optional > > Here is an example where I needed to set --with bootstrap in my module: > > https://src.fedoraproject.org/modules/tycho/blob/df837b8793fe460d2c7e72ab6d6 > 38a0f6e9f47a7/f/tycho.yaml#_76 > Thanks for the tip but that wouldn't work for Golang. When we bootstrap we disable certain codepath to workaround the cycle, but we still need to reenable them later to build the final binary. So we need to do both a bootstrap build, and then a non-bootstrap build to get all the functionality. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
On 2019-04-16, Igor Gnatenko wrote: > You can do something like: > > rpms: > foo-bootstrap: > name: foo > bar: > buildafter: > - foo-bootstrap > foo: > buildafter: > - bar > How does foo.spec get to know that it should enable bootstrapping in foo-bootstrap component, but disable it in foo component? -- Petr ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
You can do something like: rpms: foo-bootstrap: name: foo bar: buildafter: - foo-bootstrap foo: buildafter: - bar However, MBS doesn't support that yet. On Tue, Apr 16, 2019 at 1:35 AM Robert-André Mauchin wrote: > Hi, > > In Golang, we have a lot of cyclic dependencies for which we have to > bootstrap > many packages, and then later build them unbootstrapped when all the deps > are > in place. > How are we supposed to handle this in Modularity? Since it rebuilds all > packages from the start, it will fail to build cyclic deps. Is there a way > to > turn on the --with bootstrap switch in Modularity? Are we supposed to > build > the package two times, one with bootstrap, and later without? > > Best regards, > > Robert-André > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora modularity and cyclic dependencies
On Tue, 16 Apr 2019 at 00:29, Robert-André Mauchin wrote: > > Hi, > > In Golang, we have a lot of cyclic dependencies for which we have to bootstrap > many packages, and then later build them unbootstrapped when all the deps are > in place. > How are we supposed to handle this in Modularity? Since it rebuilds all > packages from the start, it will fail to build cyclic deps. Is there a way to > turn on the --with bootstrap switch in Modularity? Are we supposed to build > the package two times, one with bootstrap, and later without? > > Best regards, > > Robert-André Yes, there is a buildopts section in the yaml file where you can specify rpm macros. Here is the documentation: https://docs.fedoraproject.org/en-US/modularity/making-modules/defining-modules/#_build_macros_optional Here is an example where I needed to set --with bootstrap in my module: https://src.fedoraproject.org/modules/tycho/blob/df837b8793fe460d2c7e72ab6d638a0f6e9f47a7/f/tycho.yaml#_76 -- Mat Booth http://fedoraproject.org/get-fedora ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
