Re: Feedback wanted - pruning old rawhide chroots in Copr
On Mon, Feb 19, 2024 at 10:18 AM Stephen Smoogen wrote: > > > > On Mon, 19 Feb 2024 at 10:08, Kevin Kofler via devel > wrote: >> >> Stephen Smoogen wrote: >> > 1. Drive size is not just what is needed but also throughput. The large >> > drives needed to store the data COPR uses for its hundreds of chroots are >> > much 'slower' on reads and writes even when adding in layers of RAID 1+0. >> > Faster drives are possible but the price goes up considerably. >> > 2. Throughput of individual drives also requires backplane speeds which >> > match peek throughput of all the drives. Otherwise you end up with lots of >> > weird stalling (as seen on certain builders which have such drives). >> >> What kind of throughput is needed for a repository that has not seen any new >> builds for 2 years? Such a repository is going get only a handful downloads >> and no uploads. Instead of deleting old repositories, they can be moved to a >> low-throughput archive storage. This can be made transparent through >> symlinks, union file systems, or even just at the HTTPS level if Copr itself >> knows how to unarchive a repository when internally needed (e.g., if a new >> build is submitted after 2 years of inactivity). >> > > The throughput is actually in several places even for low/no usage > repositories. > 1. RAID rebuilds will need to go through and check data. RAID-1 might seem > like a no-brainer but you tend to end up with 'which of these two disks is > the correct bit' over time problems. > 2. web-spiders and such regularly peruse pretty much every package regularly. > Putting some repositories on slow disks and some on fast tend to cause web > front ends to pause out for unrelated tasks unless you set up your caching > and other middleware to deal with it. [This I know from when I tried to make > something more 'efficient' on downloads.fedoraproject.org and from some other > tooling.] It becomes a complete project of setting up the infrastructure to > best handle mixed loads. If you have a limited staff then it may be too much > work. > > That said, the above does sound like an interesting project to add to copr. I > do not know how much work it would take or who would be able to do it these > days. [My understanding is that COPR is 'one of many things' that the various > developers work on with most of the work done as a volunteer task.] > A lot of these problems may go away in the future once the Pulp backend for COPR is ready. That will allow repository storage to move from EBS to S3, and repositories in S3 can be set with different policies wrt to CloudFront for transparent CDN performance. https://github.com/fedora-copr/copr/issues/2533 -- 真実はいつも一つ！/ Always, there's only one truth! -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
On Mon, 19 Feb 2024 at 10:08, Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote: > Stephen Smoogen wrote: > > 1. Drive size is not just what is needed but also throughput. The large > > drives needed to store the data COPR uses for its hundreds of chroots are > > much 'slower' on reads and writes even when adding in layers of RAID 1+0. > > Faster drives are possible but the price goes up considerably. > > 2. Throughput of individual drives also requires backplane speeds which > > match peek throughput of all the drives. Otherwise you end up with lots > of > > weird stalling (as seen on certain builders which have such drives). > > What kind of throughput is needed for a repository that has not seen any > new > builds for 2 years? Such a repository is going get only a handful > downloads > and no uploads. Instead of deleting old repositories, they can be moved to > a > low-throughput archive storage. This can be made transparent through > symlinks, union file systems, or even just at the HTTPS level if Copr > itself > knows how to unarchive a repository when internally needed (e.g., if a new > build is submitted after 2 years of inactivity). > > The throughput is actually in several places even for low/no usage repositories. 1. RAID rebuilds will need to go through and check data. RAID-1 might seem like a no-brainer but you tend to end up with 'which of these two disks is the correct bit' over time problems. 2. web-spiders and such regularly peruse pretty much every package regularly. Putting some repositories on slow disks and some on fast tend to cause web front ends to pause out for unrelated tasks unless you set up your caching and other middleware to deal with it. [This I know from when I tried to make something more 'efficient' on downloads.fedoraproject.org and from some other tooling.] It becomes a complete project of setting up the infrastructure to best handle mixed loads. If you have a limited staff then it may be too much work. That said, the above does sound like an interesting project to add to copr. I do not know how much work it would take or who would be able to do it these days. [My understanding is that COPR is 'one of many things' that the various developers work on with most of the work done as a volunteer task.] Kevin Kofler > -- > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
Stephen Smoogen wrote: > 1. Drive size is not just what is needed but also throughput. The large > drives needed to store the data COPR uses for its hundreds of chroots are > much 'slower' on reads and writes even when adding in layers of RAID 1+0. > Faster drives are possible but the price goes up considerably. > 2. Throughput of individual drives also requires backplane speeds which > match peek throughput of all the drives. Otherwise you end up with lots of > weird stalling (as seen on certain builders which have such drives). What kind of throughput is needed for a repository that has not seen any new builds for 2 years? Such a repository is going get only a handful downloads and no uploads. Instead of deleting old repositories, they can be moved to a low-throughput archive storage. This can be made transparent through symlinks, union file systems, or even just at the HTTPS level if Copr itself knows how to unarchive a repository when internally needed (e.g., if a new build is submitted after 2 years of inactivity). Kevin Kofler -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
Dne 19. 02. 24 v 14:59 Kevin Kofler via devel napsal(a): Instead of coming up with new aggressive pruning schemes, Copr really needs to come up with a reasonable amount of storage to satisfy user demands. HDDs in the multi-TB-range are available for fairly low budgets (extremely low by the standards of a company like IBM), and it just takes 2 of them to build a RAID 1 that is safe against data loss. Of course, that means that Copr needs to stop locking itself into third-party cloud providers that charge ridiculously high prices for storage. 1) Fedora and Copr are using AWS. We are not locked there as Fedora infrastructure has Ansible playbooks for everything and we can migrate somewhere else in few days. The cost of AWS is very competitive - it costs Fedora $0. Zero dollars! Amazon is sponsoring Fedora this way. And I am really grateful for that. 2) While we still try to optimize the "cost", the maintainability is equally important now. We already use RAID. This is RAID configuration of copr-backend: # cat /proc/mdstat Personalities : [raid1] [raid10] md126 : active raid10 nvme5n1p1[1] nvme0n1p1[2] nvme4n1p1[0] nvme2n1p1[3] 25769536512 blocks super 1.2 512K chunks 2 near-copies [4/4] [] bitmap: 22/192 pages [88KB], 65536KB chunk md127 : active raid1 nvme1n1p1[1] nvme6n1p1[0] 16777081856 blocks super 1.2 [2/2] [UU] bitmap: 11/125 pages [44KB], 65536KB chunk Raid check takes more than week. Backup takes days. We did the training exercise of restore from backup and it took several weeks. We identified places for improvements and get it down to a week. Traversing all repositories to delete old build takes almost a day (and we several times had to work on speed improvements there). Some improvements found the way back to basic tools like createrepo_c. We can easily create 100TB volume using a RAID. Or even 1PB. But the maintenance would be a hell. And several parts of Copr would be painfully slow. It is like a things in a house. From certain size, moving to bigger house does not improve a situation and makes certain things harder: finding things, cleaning house... If anyone think that the maintenance of Copr can be done better (I am sure, it can!) - then I want you to invite to join our group. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
On Mon, 19 Feb 2024 at 08:59, Kevin Kofler via devel < devel@lists.fedoraproject.org> wrote: > Miroslav Suchý wrote: > > What **you** would find as acceptable policy for pruning rawhide chroots? > > As I mentioned several times, I already find the existing policy for > pruning > EOL release chroots unacceptable (because deleting data must never be the > default – notifications can be and are still lost in spam filters, I still > do not ever get any notification from Copr! – and because the UI to extend > the lifetime follows dark patterns, requiring us to click separately for > every single chroot instead of having an "Extend all" button). > > Instead of coming up with new aggressive pruning schemes, Copr really > needs > to come up with a reasonable amount of storage to satisfy user demands. > HDDs > in the multi-TB-range are available for fairly low budgets (extremely low > by > the standards of a company like IBM), and it just takes 2 of them to build > a > RAID 1 that is safe against data loss. Of course, that means that Copr > needs > to stop locking itself into third-party cloud providers that charge > ridiculously high prices for storage. > Kevin, I agree with your general concerns but have problems with your analysis of costs. 1. Drive size is not just what is needed but also throughput. The large drives needed to store the data COPR uses for its hundreds of chroots are much 'slower' on reads and writes even when adding in layers of RAID 1+0. Faster drives are possible but the price goes up considerably. 2. Throughput of individual drives also requires backplane speeds which match peek throughput of all the drives. Otherwise you end up with lots of weird stalling (as seen on certain builders which have such drives). 3. Outside of that you need to have a fast network switch speed of 10G minimum and 40G to 100G to deal with the multiple builders and the storage server. The larger the storage, the more bandwidth needed all the way through as building software eats lots of space. 4. The builders need to be housed in a datacenter which charges for a. power b. cooling c. square footage used d. staff to deal with problems. e. racks and wiring and parts Going from the costs 5 years ago, it took using storage systems from 45drives and systems elsewhere.. The capital costs for COPR was around 350k. The operating expenses were around 80k per year. At the time, chroots and other things needed to be cleaned much more regularly than now. We would probably need to double or triple those costs to meet what we have now. [budgets like what was given then only came around 1/10 years or so.] The amazon systems cost Fedora $0.00 as long as it stays within 'modest' disk space and other resources. That said, I realize there will be a day when you can clearly said "I told you this would happen" when being locked in comes back to bite. -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
> On Sun, Feb 18, 2024 at 4:25 PM Michael J Gruber wrote: > I like this idea. Move things that were built for "rawhide" into the > "fedora-40" chroot, and start Rawhide empty, requiring fresh builds of > things. > Since there is no equivalent to the mass rebuild in COPR, that would > also solve the problem of "stale" packages in Rawhide chroots. > > Fabio Could you clarify what that would mean to a copr (e.g. [1]) that has only a rawhide branch. Would that switch to fedora-41 on the next branching, and I have to manually recreate the rawhide branch every half year? Or will there be in addition an fedora-41 created, where all the builds will go and the rawhide branch will be empty? What I would really like if there would be an option to say that builds get automatically deleted after a few months, but I could not find such an option, only to delete the whole copr after a given time, but that is not what I want. Would such an option avoid the problem? If the user could specify when builds should get pruned? And if the selected time is more then a year, they have to confirm every half year? Looking at the storage statistics it is not quite clear to me to what extend "small" projects matter at all. It seems there are around 30 TB of storage, of which the two biggest user produce 5 TB each. Maybe an opt-in into deletion plus some communication with the biggest storage users could be a solution? David [1] https://copr.fedorainfracloud.org/coprs/davidsch/testing/ -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
Miroslav Suchý wrote: > What **you** would find as acceptable policy for pruning rawhide chroots? As I mentioned several times, I already find the existing policy for pruning EOL release chroots unacceptable (because deleting data must never be the default – notifications can be and are still lost in spam filters, I still do not ever get any notification from Copr! – and because the UI to extend the lifetime follows dark patterns, requiring us to click separately for every single chroot instead of having an "Extend all" button). Instead of coming up with new aggressive pruning schemes, Copr really needs to come up with a reasonable amount of storage to satisfy user demands. HDDs in the multi-TB-range are available for fairly low budgets (extremely low by the standards of a company like IBM), and it just takes 2 of them to build a RAID 1 that is safe against data loss. Of course, that means that Copr needs to stop locking itself into third-party cloud providers that charge ridiculously high prices for storage. Kevin Kofler -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
On Sun, 2024-02-18 at 23:45 +0100, Fabio Valentini wrote:
> On Sun, Feb 18, 2024 at 4:25 PM Michael J Gruber
> wrote:
> >
>
> (snip)
>
> >
> > I see this somehow connected to the discussion about signing keys
> > that
> > we had recently. A radical solution would be: branch rawhide, not
> > from
> > rawhide. So, at the "F40 branch point we had last week", we would:
> > - switch the "alias" rawhide from "meaning f40" to "meaning f41"
> > - rename rawhide chroots to f40 in copr
> > - set up new rawhide chroots ("follow [up] fedora branching")
> >
> > In most cases, "forked" packages in copr are misleading - they are
> > in
> > a chroot without having been built against any version of it.
> >
> > copr users would have to hit "rebuild", which should be OK.
>
> I like this idea. Move things that were built for "rawhide" into the
> "fedora-40" chroot, and start Rawhide empty, requiring fresh builds
> of
> things.
> Since there is no equivalent to the mass rebuild in COPR, that would
> also solve the problem of "stale" packages in Rawhide chroots.
I like the idea , instead Fedora 40 have the fork of rawhide build, it
became rawhide that have the fork of Fedora 40 build ...
I realize if build have {?dist} macro is easy if it is 3 or 4 release
old can be deleted .
And if not have {?dist} macro , they have the build date ...
for example
https://download.copr.fedorainfracloud.org/results/sergiomb/SambaAD/fedora-rawhide-x86_64/01177931-samba/
> Fabio
> --
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
--
Sérgio M. B.
--
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
On Sun, Feb 18, 2024 at 4:25 PM Michael J Gruber wrote:
>
(snip)
>
> I see this somehow connected to the discussion about signing keys that
> we had recently. A radical solution would be: branch rawhide, not from
> rawhide. So, at the "F40 branch point we had last week", we would:
> - switch the "alias" rawhide from "meaning f40" to "meaning f41"
> - rename rawhide chroots to f40 in copr
> - set up new rawhide chroots ("follow [up] fedora branching")
>
> In most cases, "forked" packages in copr are misleading - they are in
> a chroot without having been built against any version of it.
>
> copr users would have to hit "rebuild", which should be OK.
I like this idea. Move things that were built for "rawhide" into the
"fedora-40" chroot, and start Rawhide empty, requiring fresh builds of
things.
Since there is no equivalent to the mass rebuild in COPR, that would
also solve the problem of "stale" packages in Rawhide chroots.
Fabio
--
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
On 18. 02. 24 13:54, Miroslav Suchý wrote: In Copr build system, we noticed that Fedora rawhide chroots can became large and they stay forever as rawhide is never EOLed. We plan to work on this soon, but we are not sure what is best approach. I want to ask you - the users of Copr - what will be convenient for you? The problem is described here https://github.com/fedora-copr/copr/issues/2933 tl;dr version is: * when you build into fedora-39 chroot then the chroot is one day declared as EOLed and if you did not act, then the chroot from the project is deleted and we reclaim the storage space. * when you build into rawhide chroot, then we keep last builds. Even if you do not submit to this project anything for years, we still keep this chroot. And such chroots can occupy gigabytes of storage. The problem is that builds in the rawhide can be packaged configs, and they can be still usable despite the fact that no one rebuilds the RPM for years. Or it can be forgotten build that is not even installable in current chroot. We do not know. And testing installability of package regularly will be expensive task. What **you** would find as acceptable policy for pruning rawhide chroots? If a build wasn't done in the copr chroot for long time, I would consider it acceptable to get a notification that my chroot will be purged unless I take an action. I think the EOLed chroots behave similarly, correct? A long time could be defined as a fixed period (e.g. 1 year, 2 years), or by the meaning of rawhide (e.g. when the last build was done when rawhide was the Fedora that goes EOL now, consider this rawhide chroot for purging). Are there other chroots with the same problems? opensuse-tumbleweed, openmandriva-cooker, openmandriva-rolling, mageia-cauldron... -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Feedback wanted - pruning old rawhide chroots in Copr
Am So., 18. Feb. 2024 um 13:54 Uhr schrieb Miroslav Suchý :
>
> In Copr build system, we noticed that Fedora rawhide chroots can became large
> and they stay forever as rawhide is never
> EOLed.
> We plan to work on this soon, but we are not sure what is best approach. I
> want to ask you - the users of Copr - what
> will be convenient for you?
>
> The problem is described here https://github.com/fedora-copr/copr/issues/2933
>
> tl;dr version is:
>
> * when you build into fedora-39 chroot then the chroot is one day declared
> as EOLed and if you did not act, then the
> chroot from the project is deleted and we reclaim the storage space.
>
> * when you build into rawhide chroot, then we keep last builds. Even if you
> do not submit to this project anything for
> years, we still keep this chroot. And such chroots can occupy gigabytes of
> storage.
>
>
> The problem is that builds in the rawhide can be packaged configs, and they
> can be still usable despite the fact that no
> one rebuilds the RPM for years. Or it can be forgotten build that is not even
> installable in current chroot. We do not
> know. And testing installability of package regularly will be expensive task.
>
> What **you** would find as acceptable policy for pruning rawhide chroots?
I see this somehow connected to the discussion about signing keys that
we had recently. A radical solution would be: branch rawhide, not from
rawhide. So, at the "F40 branch point we had last week", we would:
- switch the "alias" rawhide from "meaning f40" to "meaning f41"
- rename rawhide chroots to f40 in copr
- set up new rawhide chroots ("follow [up] fedora branching")
In most cases, "forked" packages in copr are misleading - they are in
a chroot without having been built against any version of it.
copr users would have to hit "rebuild", which should be OK.
Michael
--
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Feedback wanted - pruning old rawhide chroots in Copr
In Copr build system, we noticed that Fedora rawhide chroots can became large and they stay forever as rawhide is never EOLed. We plan to work on this soon, but we are not sure what is best approach. I want to ask you - the users of Copr - what will be convenient for you? The problem is described here https://github.com/fedora-copr/copr/issues/2933 tl;dr version is: * when you build into fedora-39 chroot then the chroot is one day declared as EOLed and if you did not act, then the chroot from the project is deleted and we reclaim the storage space. * when you build into rawhide chroot, then we keep last builds. Even if you do not submit to this project anything for years, we still keep this chroot. And such chroots can occupy gigabytes of storage. The problem is that builds in the rawhide can be packaged configs, and they can be still usable despite the fact that no one rebuilds the RPM for years. Or it can be forgotten build that is not even installable in current chroot. We do not know. And testing installability of package regularly will be expensive task. What **you** would find as acceptable policy for pruning rawhide chroots? -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys -- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
