Re: [Base] Proposal for buildrequires cleanup janitorial initiative
Analyzed the BRs more closely and produced some graphs for your viewing pleasure: http://www.harald-hoyer.de/2014/01/14/self-hosting-fedora-base/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Tue, Jan 7, 2014 at 9:01 PM, Bill Nottingham nott...@redhat.com wrote: Miloslav Trmač (m...@volny.cz) said: Actually, even more generally - why a self-hosting Base at all? It would clearly be absurd for the kernel to be self-hosting, and clearly we want the Fedora universe to be self-hosting. Why is it worthwhile to have Base self-hosting? Well, if we want 'the Fedora universe' to be self-hosting, where should the compiler portion that implements the bottom layer of that live? If it doesn't make sense in Cloud (definitely not), Server (maybe), or Workstation (maybe)... then that either leaves Base, or a world where you're building Base (and WS, and Server, and Cloud) using tools from 'the universe' that itself is trying to build on Base. I wouldn't say a compiler makes much sense for Server; perhaps for Workstation but that's only because our Workstation is somewhat special. Really I'd be fine with a compiler in the bigger universe - or, perhaps (NOT actually proposing this, we coordinating between the WGs already requires enough work) in a development tools product. e.g., if base defines standardizes on the minimal set of build packages, that Server/WS/Cloud can then use to build themselves (or selectively override), and the universe can then use (or selectively override), it creates a clean heirarchy. It's not obvious to me that product installation dependencies (Cloud requires Base) need to have any relationship with product creation dependencies (e.g. Base may be built on a build system which is actually a Server running Cloud images, or a Base package may require a Cloud image or a Server-contained database server for its test suite). Yes, there is some value in having as few loops in the build order as possible, to allow new platform bringup, but that should be very rare and, in any case, I'd guess (having no data - what's the ARM experience?) the packaging effort is dwarfed by the effort required to port the tool chain. If the build environment instead lives in the Universe, and all of Base, and Server/WS/Cloud use it... that makes Fedora still (in some respects) one big package repo, and the products more like spins than actual separate products. From the users point of view, I can't see that: Microsoft Office is not a less separate product just because it doesn't ship Visual Studio. Yes, this would make the three products not three Linux distributions as we traditionally think of them; I think that's fine. Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Tue, Jan 14, 2014 at 05:49:03PM +0100, Harald Hoyer wrote: Analyzed the BRs more closely and produced some graphs for your viewing pleasure: http://www.harald-hoyer.de/2014/01/14/self-hosting-fedora-base/ Beautiful! Well, kind of ugly. But it's neat to see! Also humorous that graphviz is in the set, because, whew, you'll still be able to generate these graphics with just the base OS. -- Matthew Miller-- Fedora Project--mat...@fedoraproject.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Tue, Jan 14, 2014 at 07:52:29PM +0100, Miloslav Trmač wrote: Really I'd be fine with a compiler in the bigger universe - or, perhaps (NOT actually proposing this, we coordinating between the WGs already requires enough work) in a development tools product. It doesn't necessarily need to be its own product to have a special status. It wouldn't necessarily even need to be stand-alone -- it could be something like base build path akin to the current critical path in packagedb. -- Matthew Miller-- Fedora Project--mat...@fedoraproject.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
Miloslav Trmač (m...@volny.cz) said: During last weeks Base WG discussion about package set and self hosting of Base we came to a point where especially the self hosting of Base would currently look absurd as we'd require more than 2000 components to do so. Once you reduce the size of this set, do you forsee actually *enforcing* this in some way? For example, by having separate package repositories? If not, what's the point of this initiative? Actually, even more generally - why a self-hosting Base at all? It would clearly be absurd for the kernel to be self-hosting, and clearly we want the Fedora universe to be self-hosting. Why is it worthwhile to have Base self-hosting? Well, if we want 'the Fedora universe' to be self-hosting, where should the compiler portion that implements the bottom layer of that live? If it doesn't make sense in Cloud (definitely not), Server (maybe), or Workstation (maybe)... then that either leaves Base, or a world where you're building Base (and WS, and Server, and Cloud) using tools from 'the universe' that itself is trying to build on Base. e.g., if base defines standardizes on the minimal set of build packages, that Server/WS/Cloud can then use to build themselves (or selectively override), and the universe can then use (or selectively override), it creates a clean heirarchy. If the build environment instead lives in the Universe, and all of Base, and Server/WS/Cloud use it... that makes Fedora still (in some respects) one big package repo, and the products more like spins than actual separate products. Whether the first of these is actually *doable* cleanly is still up in the air, but I think that's some of the idea behind it. Bill -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 20, 2013 at 11:10 PM, Colin Walters walt...@verbum.org wrote: On Thu, 2013-12-12 at 18:50 +0100, Phil Knirsch wrote: Hi everyone. During last weeks Base WG discussion about package set and self hosting of Base we came to a point where especially the self hosting of Base would currently look absurd as we'd require more than 2000 components to do so. Once you reduce the size of this set, do you forsee actually *enforcing* this in some way? For example, by having separate package repositories? If not, what's the point of this initiative? Actually, even more generally - why a self-hosting Base at all? It would clearly be absurd for the kernel to be self-hosting, and clearly we want the Fedora universe to be self-hosting. Why is it worthwhile to have Base self-hosting? I wouldn't expect Server or Cloud to be necessarily self-hosting either (personal opinion, not speaking for any of the WGs). (And being nitpicky, will Base include Koji and all the databases and web servers necessary to run it?) It seems to me that if a complex, Java-and-Erlang-and-Haskel-and-postgresql-and-Mongo-using piece of software could be integrated into the build process of Base to make the final output better (say, by making static analysis a part of each build), it should be possible. The only reason for a small self-hosting base I can think of is bootstrapping a new architecture, but that's a special and rare case AFAICT. What benefits am I missing? Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 20, 2013 at 12:09:34PM +0100, Phil Knirsch wrote: On 12/15/2013 11:54 AM, Richard W.M. Jones wrote: On Fri, Dec 13, 2013 at 12:20:50PM +0100, Vít Ondruch wrote: Dne 12.12.2013 18:50, Phil Knirsch napsal(a): Initiate build requires cleanup for base related packages in Fedora working with maintainers and the community. The goal is to reduce the number of self-hosting packages required for Base from currently over 2000 packages. Just a few (probably silly ;) ideas which comes to my mind reading about this initiative: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. auto-buildrequires: http://people.redhat.com/~rjones/auto-buildrequires/ Rich. Ah, i remember you talked about that before Richard. Thanks for the link, i'll definitely be looking into that. If i have any patches that i'd like to get reviewed shall i send them to you directly or is there a ML somewhere where to send them? I think I only ever got one patch for this project. In any case, you can send any patches to me. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On 12/15/2013 11:54 AM, Richard W.M. Jones wrote: On Fri, Dec 13, 2013 at 12:20:50PM +0100, Vít Ondruch wrote: Dne 12.12.2013 18:50, Phil Knirsch napsal(a): Initiate build requires cleanup for base related packages in Fedora working with maintainers and the community. The goal is to reduce the number of self-hosting packages required for Base from currently over 2000 packages. Just a few (probably silly ;) ideas which comes to my mind reading about this initiative: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. auto-buildrequires: http://people.redhat.com/~rjones/auto-buildrequires/ Rich. Ah, i remember you talked about that before Richard. Thanks for the link, i'll definitely be looking into that. If i have any patches that i'd like to get reviewed shall i send them to you directly or is there a ML somewhere where to send them? Thanks regards, Phil -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch pknir...@redhat.com Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On 12/14/2013 02:25 AM, Adam Williamson wrote: On Fri, 2013-12-13 at 14:45 +0100, Phil Knirsch wrote: Famous last words: Can't be that hard to write a script that compares 2 builds that they provide the have the same provides and requires and filelists. :) That's still not really enough; that metadata doesn't express anything *close* to all the possible capabilities of a package. Not saying this isn't a good idea, just that it should involve careful manual double checking of compose logs and the like. We already have AutoQA running rpmdiff and rpmguard tests on new builds, btw. I doubt it would be difficult to hook this effort up to that AutoQA stuff somehow so you can get nice rpmdiff/rpmguard results between your test builds. tflink/kparal (CCed) may have some thoughts. I'll definitely give those a try, thanks for the suggestions and ideas. And yes, i agree that simply because the builds appear to be the same doesn't mean they actually are. But having a list of BRs that could potentially be removed is obviously the first important step which has to be followed by a proper verification that the actual new build without the BRs really does work the same way as the old one and doesn't have any features removed invisibly or some documentation instead of being regenerated just copied or anything like that. Regards, Phil -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch pknir...@redhat.com Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On 12/14/2013 03:27 PM, Ben Boeckel wrote: On Fri, 13 Dec, 2013 at 13:41:55 GMT, Phil Knirsch wrote: Yea, I suspect both could be combined though in a script that gradually adds new BRs to a package and then iterates over adding and removing BRs until it produces equivalent output (not identical as thats kinda hard to verify and achieve, but same provides/requires of resulting packages and same filelist at least). What about BRs which are needed, but the script removes because of transitive dependencies? IMO, BuildRequires: foo-devel getting you bar-devel, which is also needed, without listing bar-devel means you now depend on foo-devel not losing that dependency. --Ben Right, but shouldn't that then lead to the package building fine but the binary packages would miss some requires? Or am i missing something here? Thanks regards, Phil -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch pknir...@redhat.com Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On 12/13/2013 03:34 PM, Ville Skyttä wrote: On Fri, Dec 13, 2013 at 3:45 PM, Phil Knirsch pknir...@redhat.com wrote: Famous last words: Can't be that hard to write a script that compares 2 builds that they provide the have the same provides and requires and filelists. :) Try /usr/bin/rpmdiff first... Great idea! Thanks regards, Phil -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch pknir...@redhat.com Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 20, 2013 at 12:15:39 +0100, Phil Knirsch wrote: Right, but shouldn't that then lead to the package building fine but the binary packages would miss some requires? How would it miss requires? foo-devel should Requires: bar-devel (see the guidelines[1]), so you'll get it, but the package itself doesn't need bar-devel so there's no need to list it explicitly. If foo drops its usage of bar, it shouldn't appear in your package anymore. --Ben [1]https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Requires_2 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Thu, 2013-12-12 at 18:50 +0100, Phil Knirsch wrote: Hi everyone. During last weeks Base WG discussion about package set and self hosting of Base we came to a point where especially the self hosting of Base would currently look absurd as we'd require more than 2000 components to do so. Once you reduce the size of this set, do you forsee actually *enforcing* this in some way? For example, by having separate package repositories? If not, what's the point of this initiative? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
If not, what's the point of this initiative? To reduce size, complexity, save time. What do you mean by *enforcing*, or how would you imagine that happening? On Fri, Dec 20, 2013 at 4:10 PM, Colin Walters walt...@verbum.org wrote: On Thu, 2013-12-12 at 18:50 +0100, Phil Knirsch wrote: Hi everyone. During last weeks Base WG discussion about package set and self hosting of Base we came to a point where especially the self hosting of Base would currently look absurd as we'd require more than 2000 components to do so. Once you reduce the size of this set, do you forsee actually *enforcing* this in some way? For example, by having separate package repositories? If not, what's the point of this initiative? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- -Jon -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
Hi, On Fri, 2013-12-20 at 16:28 -0600, Jon wrote: To reduce size, complexity, save time. That's quite vague. What do you mean by *enforcing*, or how would you imagine that happening? My original email contained a total of 3 sentences, and you didn't reply to the second sentence, which answers this. But to save you the trouble: For example, by having separate package repositories? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Mon, Dec 16, 2013 at 09:04:21 +, Richard W.M. Jones wrote: Patches welcome. It's only 905 lines of code. Probably not from me; most of my packages are on the small side (or are Haskell which cabal-rpm handles the dependencies for). In the original case in this thread, we're interested in the case where a BR is declared in the spec file but auto-buildrequires doesn't show that BR. This would be evidence that the BR is not needed and should be deleted, or at least carefully examined. Understood; I just tend to see/look for corner cases when looking at problems :) . --Ben -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Sun, Dec 15, 2013 at 06:31:56PM -0500, Ben Boeckel wrote: On Sun, Dec 15, 2013 at 22:12:09 +, Richard W.M. Jones wrote: TBH I don't think that's necessarily a bug. As long as B-devel Requires C-devel, and if A isn't directly including headers from C-devel, it seems fine for A not to BuildRequire C-devel. I was getting at C-devel being a BR is a bug, so we agree here, but it looks like a-br just lists things without a rationale. I don't know how fancy you could get, but maybe tracking open/close recursively to see what is included where would work? Patches welcome. It's only 905 lines of code. In the original case in this thread, we're interested in the case where a BR is declared in the spec file but auto-buildrequires doesn't show that BR. This would be evidence that the BR is not needed and should be deleted, or at least carefully examined. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
Dne 15.12.2013 11:55, Richard W.M. Jones napsal(a): On Fri, Dec 13, 2013 at 10:57:00AM -0500, Matthew Miller wrote: On Fri, Dec 13, 2013 at 12:20:50PM +0100, Vít Ondruch wrote: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. This is kind of hard to do without extensive functional tests, because it may be that a BR was added because the build completes happily without it but misses the related functionality. (This is pretty common, I think.) auto-buildrequires (http://people.redhat.com/~rjones/auto-buildrequires/) uses an LD_PRELOAD hack to find out what BuildRequires are packages are actually touched during the build. Therefore it does not suffer from this problem. Rich. Unfortunately, there are some BR which are needed to pass the test suite, there are also other languages, which does not produce ELF files So it might help, but it does not solve everything. Vít -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Mon, Dec 16, 2013 at 10:39:16AM +0100, Vít Ondruch wrote: Dne 15.12.2013 11:55, Richard W.M. Jones napsal(a): On Fri, Dec 13, 2013 at 10:57:00AM -0500, Matthew Miller wrote: On Fri, Dec 13, 2013 at 12:20:50PM +0100, Vít Ondruch wrote: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. This is kind of hard to do without extensive functional tests, because it may be that a BR was added because the build completes happily without it but misses the related functionality. (This is pretty common, I think.) auto-buildrequires (http://people.redhat.com/~rjones/auto-buildrequires/) uses an LD_PRELOAD hack to find out what BuildRequires are packages are actually touched during the build. Therefore it does not suffer from this problem. Rich. Unfortunately, there are some BR which are needed to pass the test suite, there are also other languages, which does not produce ELF files So it might help, but it does not solve everything. auto-buildrequires will find all those dependencies. The only thing it won't probe are statically linked binaries. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 13, 2013 at 12:20:50PM +0100, Vít Ondruch wrote: Dne 12.12.2013 18:50, Phil Knirsch napsal(a): Initiate build requires cleanup for base related packages in Fedora working with maintainers and the community. The goal is to reduce the number of self-hosting packages required for Base from currently over 2000 packages. Just a few (probably silly ;) ideas which comes to my mind reading about this initiative: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. auto-buildrequires: http://people.redhat.com/~rjones/auto-buildrequires/ Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 13, 2013 at 10:57:00AM -0500, Matthew Miller wrote: On Fri, Dec 13, 2013 at 12:20:50PM +0100, Vít Ondruch wrote: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. This is kind of hard to do without extensive functional tests, because it may be that a BR was added because the build completes happily without it but misses the related functionality. (This is pretty common, I think.) auto-buildrequires (http://people.redhat.com/~rjones/auto-buildrequires/) uses an LD_PRELOAD hack to find out what BuildRequires are packages are actually touched during the build. Therefore it does not suffer from this problem. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Sun, 15 Dec, 2013 at 10:55:41 GMT, Richard W.M. Jones wrote: auto-buildrequires (http://people.redhat.com/~rjones/auto-buildrequires/) uses an LD_PRELOAD hack to find out what BuildRequires are packages are actually touched during the build. Therefore it does not suffer from this problem. Given A needing B which uses C's headers (but never included directly), will C-devel be added as a BR to A? IMO, that would be a mistake (and a bug in A if it is depending on B to bring in C's headers). --Ben -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Sun, Dec 15, 2013 at 09:48:40PM +, Ben Boeckel wrote: On Sun, 15 Dec, 2013 at 10:55:41 GMT, Richard W.M. Jones wrote: auto-buildrequires (http://people.redhat.com/~rjones/auto-buildrequires/) uses an LD_PRELOAD hack to find out what BuildRequires are packages are actually touched during the build. Therefore it does not suffer from this problem. [..] auto-buildrequires doesn't adjust or even look at the declared BuildRequires of a package. All it does is examine every file that is accessed during a build and map those back to packages, thus suggesting a list of BuildRequires that could be used. It tends to over-suggest packages for various reasons, and that is partially worked around by some simple heuristics. For example autoconf-generated ./configure scripts open every file present under /etc/ld.so.conf.d/ thus creating a false impression that programs BuildRequire everything that places a file in that directory -- we use a heuristic to suppress this. Given A needing B which uses C's headers (but never included directly), will C-devel be added as a BR to A? In this case, assuming you mean that C's headers are read by the compiler when compiling A, then, yes, C-devel would be printed as a BuildRequires. IMO, that would be a mistake (and a bug in A if it is depending on B to bring in C's headers). TBH I don't think that's necessarily a bug. As long as B-devel Requires C-devel, and if A isn't directly including headers from C-devel, it seems fine for A not to BuildRequire C-devel. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Sun, Dec 15, 2013 at 22:12:09 +, Richard W.M. Jones wrote: TBH I don't think that's necessarily a bug. As long as B-devel Requires C-devel, and if A isn't directly including headers from C-devel, it seems fine for A not to BuildRequire C-devel. I was getting at C-devel being a BR is a bug, so we agree here, but it looks like a-br just lists things without a rationale. I don't know how fancy you could get, but maybe tracking open/close recursively to see what is included where would work? --Ben -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, 13 Dec, 2013 at 13:41:55 GMT, Phil Knirsch wrote: Yea, I suspect both could be combined though in a script that gradually adds new BRs to a package and then iterates over adding and removing BRs until it produces equivalent output (not identical as thats kinda hard to verify and achieve, but same provides/requires of resulting packages and same filelist at least). What about BRs which are needed, but the script removes because of transitive dependencies? IMO, BuildRequires: foo-devel getting you bar-devel, which is also needed, without listing bar-devel means you now depend on foo-devel not losing that dependency. --Ben -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 13, 2013 at 17:25:49 -0800, Adam Williamson awill...@redhat.com wrote: That's still not really enough; that metadata doesn't express anything *close* to all the possible capabilities of a package. Not saying this isn't a good idea, just that it should involve careful manual double checking of compose logs and the like. Another possible problem is that if package A build requires packages B and C and today package B requires C, then this would likely result in the build requires for C being dropped. Then if later B no longer requires C, this will result in problems. If they are obvious then it won't be that big of a deal. But if the build succeeds and the package just doesn't work correctly there may be confusion as to why it stopped working. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On 12/12/2013 09:00 PM, Matthew Miller wrote: On Thu, Dec 12, 2013 at 06:50:31PM +0100, Phil Knirsch wrote: Now, to figure our how the build chains for these packages look like i've cobbled together a (really bad) hack using python and repoclosure that basically takes a set of packages as an input (actually a set of requirements) and then spits out consecutively the groups of packages needed to build the previous ones, so basically a reverse grouped build order: Nifty. I have the intuition that there might be particular nodes where something pulls in an innocuous-looking dependency that leads to an explosion of build requirements. If so, I bet there's a graphical visualization of this that will make them jump out Right, that's actually on my plan as a next step. Ages ago (like, 5 years or so) we've done it once with the java cloud back then when Karsten Hopp was trying to unravel and bootstrap s390x for Fedora 12 once more. That graph alone huge and printed on A1 hung on his wall to mark off stuff that he managed to get built or fixed. :) We still do have code for that, but it's pretty rotten by now[1] and probably needs quite a bit TLC. I got 2 weeks of PTO over the holidays, so hopefully enough time to get more stuff done there. Thanks regards, Phil [1] https://git.fedorahosted.org/cgit/pyrpm.git/tree/scripts/pyrpmgraph -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch pknir...@redhat.com Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
Dne 12.12.2013 18:50, Phil Knirsch napsal(a): Initiate build requires cleanup for base related packages in Fedora working with maintainers and the community. The goal is to reduce the number of self-hosting packages required for Base from currently over 2000 packages. Just a few (probably silly ;) ideas which comes to my mind reading about this initiative: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. * Second level could be to try to limit the BR, although some extended functionality or binding might not be supported. This extended functionality or bindings could be moved out into separate package, although it would require second build run. Looking into Python BR, I believe they could be trimmed down using this approach. Vít -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 13, 2013 at 12:20:50 +0100, Vít Ondruch vondr...@redhat.com wrote: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. The check for this needs to be careful. When some requirements are missing a build can still succeed, but be missing intended features. So you can't just test whether a build succeeds or fails to determine if a build requirement is really needed. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On 12/13/2013 12:20 PM, Vít Ondruch wrote: Dne 12.12.2013 18:50, Phil Knirsch napsal(a): Initiate build requires cleanup for base related packages in Fedora working with maintainers and the community. The goal is to reduce the number of self-hosting packages required for Base from currently over 2000 packages. Just a few (probably silly ;) ideas which comes to my mind reading about this initiative: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. Good point. For some of the BRs i've already noticed i was wondering whether they are really necessary anymore. But ye, that would have to be at least somewhat automated as doing that manually for single packages would be a gigantic task time wise. /me makes a note. * Second level could be to try to limit the BR, although some extended functionality or binding might not be supported. This extended functionality or bindings could be moved out into separate package, although it would require second build run. Looking into Python BR, I believe they could be trimmed down using this approach. Yea, I suspect both could be combined though in a script that gradually adds new BRs to a package and then iterates over adding and removing BRs until it produces equivalent output (not identical as thats kinda hard to verify and achieve, but same provides/requires of resulting packages and same filelist at least). Thanks regards, Phil -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch pknir...@redhat.com Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On 12/13/2013 02:31 PM, Bruno Wolff III wrote: On Fri, Dec 13, 2013 at 12:20:50 +0100, Vít Ondruch vondr...@redhat.com wrote: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. The check for this needs to be careful. When some requirements are missing a build can still succeed, but be missing intended features. So you can't just test whether a build succeeds or fails to determine if a build requirement is really needed. Yep. I remember back when we did the s390x stuff for F12 that was one of the things we looked at specifically: Hacking things together to get at least building was a good start, but if you didn't install several other packages the build would still succeed but with autoconf automatically disabling several features. Thats why i really like the way we've moved over the past 10 years or so to explicitly only have a pretty small buildsys environment and almost everything else needs to be explicitly required for building from the respective packages. Famous last words: Can't be that hard to write a script that compares 2 builds that they provide the have the same provides and requires and filelists. :) Thanks regards, Phil -- Philipp Knirsch | Tel.: +49-711-96437-470 Manager Core Services| Fax.: +49-711-96437-111 Red Hat GmbH | Email: Phil Knirsch pknir...@redhat.com Wankelstrasse 5 | Web: http://www.redhat.com/ D-70563 Stuttgart, Germany -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
Dne 13.12.2013 14:41, Phil Knirsch napsal(a): On 12/13/2013 12:20 PM, Vít Ondruch wrote: Dne 12.12.2013 18:50, Phil Knirsch napsal(a): Initiate build requires cleanup for base related packages in Fedora working with maintainers and the community. The goal is to reduce the number of self-hosting packages required for Base from currently over 2000 packages. Just a few (probably silly ;) ideas which comes to my mind reading about this initiative: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. Good point. For some of the BRs i've already noticed i was wondering whether they are really necessary anymore. But ye, that would have to be at least somewhat automated as doing that manually for single packages would be a gigantic task time wise. /me makes a note. * Second level could be to try to limit the BR, although some extended functionality or binding might not be supported. This extended functionality or bindings could be moved out into separate package, although it would require second build run. Looking into Python BR, I believe they could be trimmed down using this approach. Yea, I suspect both could be combined though in a script that gradually adds new BRs to a package and then iterates over adding and removing BRs until it produces equivalent output (not identical as thats kinda hard to verify and achieve, but same provides/requires of resulting packages and same filelist at least). Achieve the same output is more for the first point. But the second point is more to have minimal requirements for build. Later, one have to manually review what is essential feature of the package and what can be moved into subpackage. E.g. continuing with my Python example, lets assume that Python is buildable with or without OpenSSL support. Python without enabled OpenSSL would be probably pain, because missing crypto functionality, etc. Other thing is Tcl/Tk support. I believe that Tcl/Tk could be moved into independent package and build independently, while all the features would be available in the end. Vít -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 13, 2013 at 3:45 PM, Phil Knirsch pknir...@redhat.com wrote: Famous last words: Can't be that hard to write a script that compares 2 builds that they provide the have the same provides and requires and filelists. :) Try /usr/bin/rpmdiff first... -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, Dec 13, 2013 at 12:20:50PM +0100, Vít Ondruch wrote: * It might be interesting to have some script, which tries to audit BR, e.g. it removes all BR first and then adds them back as they are required. This could reveal some BR which are actually not needed anymore, but are listed among BR from historic reasons. This is kind of hard to do without extensive functional tests, because it may be that a BR was added because the build completes happily without it but misses the related functionality. (This is pretty common, I think.) * Second level could be to try to limit the BR, although some extended functionality or binding might not be supported. This extended functionality or bindings could be moved out into separate package, although it would require second build run. Looking into Python BR, I believe they could be trimmed down using this approach. That's an interesting idea. That'd take some big changes to the build process and at least to the packaging guidelines if not RPM itself. If making the base self-hosting is decided to be crucial, it might be worth the effort. (As opposed to the alternative approach of having a set of Not in Base But Needed to Build It packages.) -- Matthew Miller -- Fedora Project Architect -- mat...@fedoraproject.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Fri, 2013-12-13 at 14:45 +0100, Phil Knirsch wrote: Famous last words: Can't be that hard to write a script that compares 2 builds that they provide the have the same provides and requires and filelists. :) That's still not really enough; that metadata doesn't express anything *close* to all the possible capabilities of a package. Not saying this isn't a good idea, just that it should involve careful manual double checking of compose logs and the like. We already have AutoQA running rpmdiff and rpmguard tests on new builds, btw. I doubt it would be difficult to hook this effort up to that AutoQA stuff somehow so you can get nice rpmdiff/rpmguard results between your test builds. tflink/kparal (CCed) may have some thoughts. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: [Base] Proposal for buildrequires cleanup janitorial initiative
On Thu, Dec 12, 2013 at 06:50:31PM +0100, Phil Knirsch wrote: Now, to figure our how the build chains for these packages look like i've cobbled together a (really bad) hack using python and repoclosure that basically takes a set of packages as an input (actually a set of requirements) and then spits out consecutively the groups of packages needed to build the previous ones, so basically a reverse grouped build order: Nifty. I have the intuition that there might be particular nodes where something pulls in an innocuous-looking dependency that leads to an explosion of build requirements. If so, I bet there's a graphical visualization of this that will make them jump out -- Matthew Miller -- Fedora Project Architect -- mat...@fedoraproject.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
