Re: [Freeipa-devel] FreeIPA Copr repo plan
On Mon, Nov 10, 2014 at 12:07:46PM +0100, Martin Kosek wrote: > > 1) What Copr repos do we want to maintain and what should be the expectations? > My take: > > a) mkosek/freeipa: latest and greatest *released* FreeIPA. Built for F20+, > EPEL-7.0. Jan, this is the one you use in the FreeIPA CentOS container, right? > Does it fit your needs? It does not install, so no idea. Both the centos-7 tag of https://registry.hub.docker.com/u/adelton/freeipa-server/ and https://registry.hub.docker.com/u/centos/freeipa/ use the IdM in CentOS 7, not FreeIPA from your copr repo, at this point. > b) Branch repos: as mkosek/freeipa Copr repo would contain only the latest and > greatest release, would it make sense to have a Copr repo with *releases* per > supported branch to give users a choice? I.e. > * mkosek/freeipa-4.1 > * mkosek/freeipa-4.0 It is not just users having a choice but the ability to quickly check behaviour on older version when developing or testing regression. > These repos are there already, but not used consistently. I do not think we > should build all the dependency chain (too much overhead) for older systems > (F20/EPEL). But I assume we could at least build the freeipa SRPM itself for > these systems if it uses "mkosek/freeipa" as additional build root in Copr. I don't think it's necessary to build dependency chain to the oldest versions. But if the release was once built on a give OS (as a nightly, for example), we should be able to keep it, including the dependencies. > Should it contain daily master builds for all tracked projects (FreeIPA, SSSD, > 389 DS, bind-dyndb-ldap)? Or do we simply want to let distribute it across > projects "mkosek/freeipa-master", "someone/sssd-master", > "someone/389-ds-base-master? Second option may scale better better, the list > of > such repos may be maintained somewhere (freeipa.org wiki). Having them into separate copr repos would certainly allow better insight into (for example) the dependency chains by people who know how their part should build and install. Currently when we see a huge dependency tree when installing freeipa-server package, it might not be immediatelly obvious, what is causing the possible bloat. -- Jan Pazdziora Principal Software Engineer, Identity Management Engineering, Red Hat ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On 11/10/2014 04:26 PM, Martin Kosek wrote: On 11/10/2014 01:49 PM, Jakub Hrozek wrote: On Mon, Nov 10, 2014 at 12:07:46PM +0100, Martin Kosek wrote: Hi guys, Some time ago we started managing FreeIPA Copr repos (mkosek/freeipa) with a target to have the latest greatest FreeIPA available for older arches (read - RHEL/CentOS) and to allow people using older stable Fedoras (read - Fedora 20) try FreeIPA 4.0+ releases which brought in several dependencies. So far this was a more ad hoc approach, I think a more firm plan and tools are due. I see several questions that needs to be decided: 1) What Copr repos do we want to maintain and what should be the expectations? My take: a) mkosek/freeipa: latest and greatest *released* FreeIPA. Built for F20+, EPEL-7.0. Jan, this is the one you use in the FreeIPA CentOS container, right? Does it fit your needs? +1 b) Branch repos: as mkosek/freeipa Copr repo would contain only the latest and greatest release, would it make sense to have a Copr repo with *releases* per supported branch to give users a choice? I.e. * mkosek/freeipa-4.1 * mkosek/freeipa-4.0 These repos are there already, but not used consistently. I do not think we should build all the dependency chain (too much overhead) for older systems (F20/EPEL). But I assume we could at least build the freeipa SRPM itself for these systems if it uses "mkosek/freeipa" as additional build root in Copr. Is it worth it? Is the older supported branch some kind of LTM or just happens to be alive because of some Fedora or RHEL release using it? I think there is value in providing early access to RHEL/CentOS users prior to dumping the RPMs onto them, but maintaining the repos is hard, I think we should only commit to this work if there is a use-case. In this particular case we wanted to have a repo to build FreeIPA 4.0.x given that mkosek/freeipa repo already contained 4.1. The purpose was to provide option to people who do not want to update to early 4.1.0 yet. DS is building the latest and greatest release in copr. I am using this DS repo to test it against IPA 4.0.x and IPA 4.1.x latests. Currently I am building IPA latests (srpms+rpms) on my own copr repos, so with a risk of taking wrong dependencies. I would prefer to have a global per supported branches copr repos, like mkosek/freeipa-4-0... thanks thierry c) Daily repos Should we deprecate old John's repos (http://www.freeipa.org/page/Downloads#Bleeding_Edge) which is difficult to maintain and replace them with Copr ones? I.e. to have common repo (e.g. mkosek/freeipa-daily) built for the supported Fedoras (F20, F21, rawhide ATM) including dependencies? Is there a build script or other infrastructure that would make the new repo easy to maintain (easier than John's repo)? In general I think there is quite a bit of value in the daily builds -- we can ask users if their problem goes away with the latest builds and we could even use this for some CI setups and we know early if something breaks. There seems to be a traction to use a single supported way of building RPMs and not maintain 2 systems - see John Dennis' response. Should it contain daily master builds for all tracked projects (FreeIPA, SSSD, 389 DS, bind-dyndb-ldap)? Or do we simply want to let distribute it across projects "mkosek/freeipa-master", "someone/sssd-master", "someone/389-ds-base-master? Second option may scale better better, the list of such repos may be maintained somewhere (freeipa.org wiki). I think I might be missing something, but why do you think separate repos are better? My idea was to decentralize it - to not overload FreeIPA developers with maintaining nightly devel builds and it's potentially new dependencies but to let domain experts from different teams to maintain it. Also, people interested in nightly builds may not be interested in nightly builds for all these packages, but maybe just SSSD. So they would just download SSSD yum repo and enable it. But if there is a value in having a united repo with nightly builds of all these packages, maybe there could simply be a cron script merging all the distributed Copr repos RPMs and placing them on fedorapeople. Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On 11/10/2014 04:22 PM, Simo Sorce wrote: > On Mon, 10 Nov 2014 12:07:46 +0100 > Martin Kosek wrote: > >> 3) Scalability of the approach >> Some dependencies are more difficult to maintain than the others. >> Especially the PKI ones often required custom Java packaging >> (resteasy-base) or a complicated dependency chain (the latest >> jackson-jaxrs-json-provider). It would be great if PKI team helps >> with this effort, as Lukas proposed. Downside is that mkosek/freeipa >> installation would require 2 Copr repos. But maybe we could have a >> job syncing the PKI build/runtime dependencies directly to FreeIPA >> copr. Whatever works and scale. > > Do we use the REST interface ? Yes: https://git.fedorahosted.org/cgit/freeipa.git/commit/?id=462beacc9d13968128fa320d155016df2d72a20a And we plan to leverage it even more: https://fedorahosted.org/freeipa/ticket/3473 IMO it is the right way instead of current HTML parsing approach. > Would it be possible to simply build dogtag w/o it and avoid this > massive set of annoying dependencies ? Given above, unfortunately not. Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On 11/10/2014 01:49 PM, Jakub Hrozek wrote: > On Mon, Nov 10, 2014 at 12:07:46PM +0100, Martin Kosek wrote: >> Hi guys, >> >> Some time ago we started managing FreeIPA Copr repos (mkosek/freeipa) with a >> target to have the latest greatest FreeIPA available for older arches (read - >> RHEL/CentOS) and to allow people using older stable Fedoras (read - Fedora >> 20) >> try FreeIPA 4.0+ releases which brought in several dependencies. >> >> So far this was a more ad hoc approach, I think a more firm plan and tools >> are >> due. I see several questions that needs to be decided: >> >> 1) What Copr repos do we want to maintain and what should be the >> expectations? >> My take: >> >> a) mkosek/freeipa: latest and greatest *released* FreeIPA. Built for F20+, >> EPEL-7.0. Jan, this is the one you use in the FreeIPA CentOS container, >> right? >> Does it fit your needs? > > +1 > >> >> b) Branch repos: as mkosek/freeipa Copr repo would contain only the latest >> and >> greatest release, would it make sense to have a Copr repo with *releases* per >> supported branch to give users a choice? I.e. >> * mkosek/freeipa-4.1 >> * mkosek/freeipa-4.0 >> These repos are there already, but not used consistently. I do not think we >> should build all the dependency chain (too much overhead) for older systems >> (F20/EPEL). But I assume we could at least build the freeipa SRPM itself for >> these systems if it uses "mkosek/freeipa" as additional build root in Copr. > > Is it worth it? Is the older supported branch some kind of LTM or just > happens to be alive because of some Fedora or RHEL release using it? > > I think there is value in providing early access to RHEL/CentOS users > prior to dumping the RPMs onto them, but maintaining the repos is hard, > I think we should only commit to this work if there is a use-case. In this particular case we wanted to have a repo to build FreeIPA 4.0.x given that mkosek/freeipa repo already contained 4.1. The purpose was to provide option to people who do not want to update to early 4.1.0 yet. >> c) Daily repos >> Should we deprecate old John's repos >> (http://www.freeipa.org/page/Downloads#Bleeding_Edge) which is difficult to >> maintain and replace them with Copr ones? I.e. to have common repo (e.g. >> mkosek/freeipa-daily) built for the supported Fedoras (F20, F21, rawhide ATM) >> including dependencies? > > Is there a build script or other infrastructure that would make the new > repo easy to maintain (easier than John's repo)? In general I think there > is quite a bit of value in the daily builds -- we can ask users if their > problem goes away with the latest builds and we could even use this for > some CI setups and we know early if something breaks. There seems to be a traction to use a single supported way of building RPMs and not maintain 2 systems - see John Dennis' response. >> Should it contain daily master builds for all tracked projects (FreeIPA, >> SSSD, >> 389 DS, bind-dyndb-ldap)? Or do we simply want to let distribute it across >> projects "mkosek/freeipa-master", "someone/sssd-master", >> "someone/389-ds-base-master? Second option may scale better better, the list >> of >> such repos may be maintained somewhere (freeipa.org wiki). > > I think I might be missing something, but why do you think separate > repos are better? My idea was to decentralize it - to not overload FreeIPA developers with maintaining nightly devel builds and it's potentially new dependencies but to let domain experts from different teams to maintain it. Also, people interested in nightly builds may not be interested in nightly builds for all these packages, but maybe just SSSD. So they would just download SSSD yum repo and enable it. But if there is a value in having a united repo with nightly builds of all these packages, maybe there could simply be a cron script merging all the distributed Copr repos RPMs and placing them on fedorapeople. Martin ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On Mon, 10 Nov 2014 12:07:46 +0100 Martin Kosek wrote: > 3) Scalability of the approach > Some dependencies are more difficult to maintain than the others. > Especially the PKI ones often required custom Java packaging > (resteasy-base) or a complicated dependency chain (the latest > jackson-jaxrs-json-provider). It would be great if PKI team helps > with this effort, as Lukas proposed. Downside is that mkosek/freeipa > installation would require 2 Copr repos. But maybe we could have a > job syncing the PKI build/runtime dependencies directly to FreeIPA > copr. Whatever works and scale. Do we use the REST interface ? Would it be possible to simply build dogtag w/o it and avoid this massive set of annoying dependencies ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On Mon, Nov 10, 2014 at 02:04:34PM +0100, Lukas Slebodnik wrote: > It *is not* possible to merge one COPR repo into another. > It is possible to add another yum repo into build dependencies in COPR Ah, right. Adding the build dependencies allows you to add another SRPM, to be built though.. > but it usually mean you will need to enable 2nd repo for installation of > freeipa as well. > > LS ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
> It *is not* possible to merge one COPR repo into another. > It is possible to add another yum repo into build dependencies in COPR > but it usually mean you will need to enable 2nd repo for installation of > freeipa as well. The script I wrote to manage the IPA repo entire purpose is to pull packages from diverse locations and merge them into one single unified repo. We call this the repo builder. The way this system currently works is the repo builder listens for messages from any number of builders, when it receives a message a new build is available it merges the new package into the repo. To be more precise it actually merges all the different OS versions, arches, debuginfo, multilib etc. to produce one single repo whose layout is identical to a Fedora repo. This is how we get a one-stop shopping repo for users to point to. My contribution to this process does not include doing any builds, instead my repo and the script that drives it assembles a unified repo from builds others do. I though Copr was capable of doing essentially the same thing, but at the same time doing the actual builds. If Copr cannot assemble an entire repo tree of OS's, arches, debuginfo, multilib, etc, then that is a big missing piece which already implemented and has been working well. -- John ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On (10/11/14 07:53), John Dennis wrote: >On 11/10/2014 06:07 AM, Martin Kosek wrote: > >> c) Daily repos Should we deprecate old John's repos >> (http://www.freeipa.org/page/Downloads#Bleeding_Edge) which is >> difficult to maintain and replace them with Copr ones? I.e. to have >> common repo (e.g. mkosek/freeipa-daily) built for the supported >> Fedoras (F20, F21, rawhide ATM) including dependencies? > >Nalin does the actual builds, I noticed he wasn't on the CC list so I >just added Nalin to this reply. > >From what I know of Copr it's a better tool than our homegrown solution. >If you're already doing Copr builds then I don't see much logic in >keeping the old system. It makes sense to me there should be 1 entity >pumping out the builds using the same technology, why duplicate efforts? >Let's use the better technology. > >> Should it contain daily master builds for all tracked projects >> (FreeIPA, SSSD, 389 DS, bind-dyndb-ldap)? Or do we simply want to let >> distribute it across projects "mkosek/freeipa-master", >> "someone/sssd-master", "someone/389-ds-base-master? Second option may >> scale better better, the list of such repos may be maintained >> somewhere (freeipa.org wiki). > >> 3) Scalability of the approach Some dependencies are more difficult >> to maintain than the others. Especially the PKI ones often required >> custom Java packaging (resteasy-base) or a complicated dependency >> chain (the latest jackson-jaxrs-json-provider). It would be great if >> PKI team helps with this effort, as Lukas proposed. Downside is that >> mkosek/freeipa installation would require 2 Copr repos. But maybe we >> could have a job syncing the PKI build/runtime dependencies directly >> to FreeIPA copr. Whatever works and scale. > >I'm not sure I'm a fan of the scattered repo approach simply because mkosek/freeipa contains 44 different packages. and approximately 1/4 of them are related just to dogtag for rhel7 >it's hard for end users, too many yum repo configs to manage. One thing >I think did work well with the old setup is the repo contained all the >necessary dependencies which could not be satisfied from the system >repos. I recognize the difficulty of trying to be a build master for a >collection of difficult to build packages. What were we doing with the >old system was to pull packages built elsewhere (i.e. Kevin did the CS >builds) and merge them into the repo thus a user needed only point to It *is not* possible to merge one COPR repo into another. It is possible to add another yum repo into build dependencies in COPR but it usually mean you will need to enable 2nd repo for installation of freeipa as well. LS ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On 11/10/2014 06:07 AM, Martin Kosek wrote: > c) Daily repos Should we deprecate old John's repos > (http://www.freeipa.org/page/Downloads#Bleeding_Edge) which is > difficult to maintain and replace them with Copr ones? I.e. to have > common repo (e.g. mkosek/freeipa-daily) built for the supported > Fedoras (F20, F21, rawhide ATM) including dependencies? Nalin does the actual builds, I noticed he wasn't on the CC list so I just added Nalin to this reply. >From what I know of Copr it's a better tool than our homegrown solution. If you're already doing Copr builds then I don't see much logic in keeping the old system. It makes sense to me there should be 1 entity pumping out the builds using the same technology, why duplicate efforts? Let's use the better technology. > Should it contain daily master builds for all tracked projects > (FreeIPA, SSSD, 389 DS, bind-dyndb-ldap)? Or do we simply want to let > distribute it across projects "mkosek/freeipa-master", > "someone/sssd-master", "someone/389-ds-base-master? Second option may > scale better better, the list of such repos may be maintained > somewhere (freeipa.org wiki). > 3) Scalability of the approach Some dependencies are more difficult > to maintain than the others. Especially the PKI ones often required > custom Java packaging (resteasy-base) or a complicated dependency > chain (the latest jackson-jaxrs-json-provider). It would be great if > PKI team helps with this effort, as Lukas proposed. Downside is that > mkosek/freeipa installation would require 2 Copr repos. But maybe we > could have a job syncing the PKI build/runtime dependencies directly > to FreeIPA copr. Whatever works and scale. I'm not sure I'm a fan of the scattered repo approach simply because it's hard for end users, too many yum repo configs to manage. One thing I think did work well with the old setup is the repo contained all the necessary dependencies which could not be satisfied from the system repos. I recognize the difficulty of trying to be a build master for a collection of difficult to build packages. What were we doing with the old system was to pull packages built elsewhere (i.e. Kevin did the CS builds) and merge them into the repo thus a user needed only point to one user, but we weren't responsible for doing builds for everything, just integrating, this makes the most sense to me. -- John ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] FreeIPA Copr repo plan
On Mon, Nov 10, 2014 at 12:07:46PM +0100, Martin Kosek wrote: > Hi guys, > > Some time ago we started managing FreeIPA Copr repos (mkosek/freeipa) with a > target to have the latest greatest FreeIPA available for older arches (read - > RHEL/CentOS) and to allow people using older stable Fedoras (read - Fedora 20) > try FreeIPA 4.0+ releases which brought in several dependencies. > > So far this was a more ad hoc approach, I think a more firm plan and tools are > due. I see several questions that needs to be decided: > > 1) What Copr repos do we want to maintain and what should be the expectations? > My take: > > a) mkosek/freeipa: latest and greatest *released* FreeIPA. Built for F20+, > EPEL-7.0. Jan, this is the one you use in the FreeIPA CentOS container, right? > Does it fit your needs? +1 > > b) Branch repos: as mkosek/freeipa Copr repo would contain only the latest and > greatest release, would it make sense to have a Copr repo with *releases* per > supported branch to give users a choice? I.e. > * mkosek/freeipa-4.1 > * mkosek/freeipa-4.0 > These repos are there already, but not used consistently. I do not think we > should build all the dependency chain (too much overhead) for older systems > (F20/EPEL). But I assume we could at least build the freeipa SRPM itself for > these systems if it uses "mkosek/freeipa" as additional build root in Copr. Is it worth it? Is the older supported branch some kind of LTM or just happens to be alive because of some Fedora or RHEL release using it? I think there is value in providing early access to RHEL/CentOS users prior to dumping the RPMs onto them, but maintaining the repos is hard, I think we should only commit to this work if there is a use-case. > > c) Daily repos > Should we deprecate old John's repos > (http://www.freeipa.org/page/Downloads#Bleeding_Edge) which is difficult to > maintain and replace them with Copr ones? I.e. to have common repo (e.g. > mkosek/freeipa-daily) built for the supported Fedoras (F20, F21, rawhide ATM) > including dependencies? Is there a build script or other infrastructure that would make the new repo easy to maintain (easier than John's repo)? In general I think there is quite a bit of value in the daily builds -- we can ask users if their problem goes away with the latest builds and we could even use this for some CI setups and we know early if something breaks. > > Should it contain daily master builds for all tracked projects (FreeIPA, SSSD, > 389 DS, bind-dyndb-ldap)? Or do we simply want to let distribute it across > projects "mkosek/freeipa-master", "someone/sssd-master", > "someone/389-ds-base-master? Second option may scale better better, the list > of > such repos may be maintained somewhere (freeipa.org wiki). I think I might be missing something, but why do you think separate repos are better? > > > 2) We will need to have some tool chain and Jenkins CI jobs watching these > repos to make sure the build & run deps are OK. So far I used the attached 2 > clumsy bash scripts to handle the repos build and one for analysis. But we > will > need something better. > > > 3) Scalability of the approach > Some dependencies are more difficult to maintain than the others. Especially > the PKI ones often required custom Java packaging (resteasy-base) or a > complicated dependency chain (the latest jackson-jaxrs-json-provider). It > would > be great if PKI team helps with this effort, as Lukas proposed. Downside is > that mkosek/freeipa installation would require 2 Copr repos. But maybe we > could > have a job syncing the PKI build/runtime dependencies directly to FreeIPA > copr. > Whatever works and scale. I thought you could 'include' one repo in another with COPR? Wouldn't that solve the problem? ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
[Freeipa-devel] FreeIPA Copr repo plan
Hi guys, Some time ago we started managing FreeIPA Copr repos (mkosek/freeipa) with a target to have the latest greatest FreeIPA available for older arches (read - RHEL/CentOS) and to allow people using older stable Fedoras (read - Fedora 20) try FreeIPA 4.0+ releases which brought in several dependencies. So far this was a more ad hoc approach, I think a more firm plan and tools are due. I see several questions that needs to be decided: 1) What Copr repos do we want to maintain and what should be the expectations? My take: a) mkosek/freeipa: latest and greatest *released* FreeIPA. Built for F20+, EPEL-7.0. Jan, this is the one you use in the FreeIPA CentOS container, right? Does it fit your needs? b) Branch repos: as mkosek/freeipa Copr repo would contain only the latest and greatest release, would it make sense to have a Copr repo with *releases* per supported branch to give users a choice? I.e. * mkosek/freeipa-4.1 * mkosek/freeipa-4.0 These repos are there already, but not used consistently. I do not think we should build all the dependency chain (too much overhead) for older systems (F20/EPEL). But I assume we could at least build the freeipa SRPM itself for these systems if it uses "mkosek/freeipa" as additional build root in Copr. c) Daily repos Should we deprecate old John's repos (http://www.freeipa.org/page/Downloads#Bleeding_Edge) which is difficult to maintain and replace them with Copr ones? I.e. to have common repo (e.g. mkosek/freeipa-daily) built for the supported Fedoras (F20, F21, rawhide ATM) including dependencies? Should it contain daily master builds for all tracked projects (FreeIPA, SSSD, 389 DS, bind-dyndb-ldap)? Or do we simply want to let distribute it across projects "mkosek/freeipa-master", "someone/sssd-master", "someone/389-ds-base-master? Second option may scale better better, the list of such repos may be maintained somewhere (freeipa.org wiki). 2) We will need to have some tool chain and Jenkins CI jobs watching these repos to make sure the build & run deps are OK. So far I used the attached 2 clumsy bash scripts to handle the repos build and one for analysis. But we will need something better. 3) Scalability of the approach Some dependencies are more difficult to maintain than the others. Especially the PKI ones often required custom Java packaging (resteasy-base) or a complicated dependency chain (the latest jackson-jaxrs-json-provider). It would be great if PKI team helps with this effort, as Lukas proposed. Downside is that mkosek/freeipa installation would require 2 Copr repos. But maybe we could have a job syncing the PKI build/runtime dependencies directly to FreeIPA copr. Whatever works and scale. -- Martin Kosek Supervisor, Software Engineering - Identity Management Team Red Hat Inc. build-srpms.sh Description: Bourne shell script #!/usr/bin/python from bs4 import BeautifulSoup from optparse import OptionParser import requests import re class Build(object): def __init__(self): self.build_status = build_status self.files = files self.missing_packages = missing_packages self.p def get_build_status(package): s = requests.Session() url = "http://copr-be.cloud.fedoraproject.org/results/mkosek/freeipa/epel-7-x86_64/%s"; % package r = s.get(url) soup = BeautifulSoup(r.text) files = [] build_result = False for tr in soup.find("tbody").findAll("tr"): tds = tr.findAll("td") f = tds[0].find(text=True) if f == 'Parent Directory': continue if f.endswith('.rpm'): build_result = True files.append(f) missing_packages = [] if not build_result: url = "http://copr-be.cloud.fedoraproject.org/results/mkosek/freeipa/epel-7-x86_64/%s/root.log"; % package r = s.get(url) root_log = r.text m = re.search(r'Error: No Package found for (\S+)', root_log) if m is not None: missing_packages.append(m.groups(1)[0]) return build_result, files, missing_packages def main(): # parse arguments parser = OptionParser() # parse options (options, args) = parser.parse_args() url = 'http://copr-be.cloud.fedoraproject.org/results/mkosek/freeipa/epel-7-x86_64/' s = requests.Session() r = s.get(url, data={}) soup = BeautifulSoup(r.text) built_packages = [] broken_packages = [] missing_dependencies = {} for tr in soup.find("tbody").findAll("tr"): tds = tr.findAll("td") package = tds[0].find(text=True) package_name = package.rsplit("-", 2)[0] if package.endswith(".log") or package in ('Parent Directory', 'repodata'): continue build_result, files, missing_packages = get_build_status(package) print package_name, "OK" if build_result else "FAIL", missing_packages if build_result: built_packages.append(package_name) else: missing_dependencies[package_name