Re: Keeping old versions of packages
On Sun, Apr 14, 2013 at 12:31 PM, Richard Hughes hughsi...@gmail.comwrote: On 13 April 2013 23:09, Kevin Kofler kevin.kof...@chello.at wrote: Sure you can! It's a basic rule of QA that small isolated changes can be debugged much better than a huge hodgepodge of many totally unrelated changes. Not true for the majority of GNOME and KDE packages. You can't test gnome-control-center 3.9.1 without installing gnome-settings-daemon 3.9.1 as well. A desktop environment should be a comparatively small subset of the overall universe of packages (in particular, when compared to the applications written for that desktop environment), so optimizing the distribution update system for desktop environment updates doesn't make that much sense. Desktop environment updates can be published as a group with the current model already. Mirek -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On 13 April 2013 23:09, Kevin Kofler kevin.kof...@chello.at wrote: Sure you can! It's a basic rule of QA that small isolated changes can be debugged much better than a huge hodgepodge of many totally unrelated changes. Not true for the majority of GNOME and KDE packages. You can't test gnome-control-center 3.9.1 without installing gnome-settings-daemon 3.9.1 as well. Not because of any library interface issue, but because g-s-d provides the D-Bus API used by g-c-c. The desktop, like it or not, is getting more monolithic, not less. FWIW, that's a totally good thing. Richard. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, Apr 10, 2013 at 5:24 PM, Reindl Harald h.rei...@thelounge.net wrote: but Fedora IS NOT RHEL if you want the RHEL way use it WHO ARE YOU KIDDING? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Richard Hughes wrote: Not true for the majority of GNOME and KDE packages. You can't test gnome-control-center 3.9.1 without installing gnome-settings-daemon 3.9.1 as well. Not because of any library interface issue, but because g-s-d provides the D-Bus API used by g-c-c. The desktop, like it or not, is getting more monolithic, not less. FWIW, that's a totally good thing. Speaking of KDE because that's what I'm familiar with: * The KDE version upgrades are already being pushed as groups. All the 4.10.2 packages are in a single update in Bodhi. That makes a lot of sense because upstream releases them together at the same time. So doing monthly batching in Fedora will only mean we'll be out of sync with upstream, it would not provide any grouping that is not already being done. * In KDE land, you can actually use old kde* on new kdelibs (at least you should be able to; it's not really tested or supported for the core KDE packages because they are updated at the same time as kdelibs and thus expected to be used together), but not new kde* on old kdelibs. So it would be possible to test the updated packages separately in reverse dependency order. It's just neither practical (time-wise) nor necessary. The monthly grouping is already done by upstream there. * KDE is getting less monolithic, not more. KDE Frameworks 5 will even be taking the splitting to extreme levels. Basically, where upstream releases a large set of version updates simultaneously (e.g. KDE coordinated releases), we are already pushing them as one group and downstream batching would bring no benefits whatsoever, whereas for small leaf packages (e.g. colord-kde), the updates can easily be tested in isolation (even much better than when mixed with a huge batch of mostly unrelated changes; if, say, a batch of colord + kdelibs + colord-kde breaks colord-kde, how do we know which updated package broke it?). Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Am 14.04.2013 12:51, schrieb Dan Mashal: On Wed, Apr 10, 2013 at 5:24 PM, Reindl Harald h.rei...@thelounge.net wrote: but Fedora IS NOT RHEL if you want the RHEL way use it WHO ARE YOU KIDDING? ?? if someone aks for a monthly patchday for Fedora instead push updates after bugs are fixed who is kidding there? who the hell needs this? if you do not want updates - FINE do not install them why do someone need handholding by such simple decisions? signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Sun, 14 Apr 2013 12:59:06 +0200 Reindl Harald h.rei...@thelounge.net wrote: if someone aks for a monthly patchday for Fedora instead push updates after bugs are fixed who is kidding there? I don't think anyone actually was asking for that. The proposal around monthly update packs includes the idea that people who want to use the current flow of updates process can continue to do so just fine. It's an additional setup for people who do not want a constant flow of updates. In any case it's not formally been proposed yet that I know of, so it's kind of silly to speculate about a plan the full details of which aren't available. kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Richard Hughes wrote: Using PackageKit and yum on the command line is often painful as we have to always download metadata unless it's less than a few hours old. Being able to update the metadata once a week would be awesome (with the possible exception of security updates) so that we could schedule the 20Mb+ metadata update when the user is idle rather than waiting for updates. For me, only once a week is totally unacceptable. I have Apper check for updates hourly. If the metadata isn't actually going to be refreshed on those hourly runs, that'd really suck! Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Richard Hughes wrote: You can't QA a trickle. Sure you can! It's a basic rule of QA that small isolated changes can be debugged much better than a huge hodgepodge of many totally unrelated changes. If packages are small and self-contained then sure, it might work, but applications depending on ~30 other libraries and ~20 other daemons not so much. The packages are sometimes not (small and self-contained), but the changes usually are! Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Matthew Miller wrote: Overall, it's a more predictable workload, which *is* a good idea, for both volunteer and otherwise. No, sorry, but as volunteers, we have other commitments which mean we cannot always do our Fedora work when some central Fedora schedule dictates it. The mad rushes at release time are already bad enough, adding ones for update batches every month is VERY unhelpful. The schedule will inevitably conflict with real life. The current flexible approach is much better suited for volunteer developers, allowing us to schedule our work on updates when WE have time. And for what is worth, the current approach is also more flexible for our users, for similar reasons. Not to mention that there are users like me who WANT as frequent updates as possible. (I'd personally like instant or hourly pushes, or at least consistent daily pushes (independently of whether it's a business day etc.).) But even those users who want to update only once a month will want to do it on THEIR schedule, not on the Fedora project's. It's also more effective to QA packages in sets, and more effective can mean more efficient. How is it more effective to QA a huge batch of totally unrelated changes which may or may not have interactions as opposed to QAing small isolated changes individually? This just does not make any sense whatsoever. Kevin Kofler -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Am 14.04.2013 00:03, schrieb Kevin Kofler: Richard Hughes wrote: Using PackageKit and yum on the command line is often painful as we have to always download metadata unless it's less than a few hours old. Being able to update the metadata once a week would be awesome (with the possible exception of security updates) so that we could schedule the 20Mb+ metadata update when the user is idle rather than waiting for updates. For me, only once a week is totally unacceptable. I have Apper check for updates hourly. If the metadata isn't actually going to be refreshed on those hourly runs, that'd really suck! OK, hourly is a little bit too much but daily - YES! and weekly - NO! signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
7 days is way, way, way too long for an active repository such as any of the currently supported Fedoras. And yes, it *is* burdensome to download and keep updating all the time. If running more than a few servers, I always maintain a local nightly mirror and point my clients to that by default. The metadata download is still burdensome, but the traffic is local and much faster. And I completely exclude drpms from the mirror, there's just no point to it if you've using a local mirror. On Tue, Apr 9, 2013 at 5:21 AM, Reindl Harald h.rei...@thelounge.netwrote: Am 09.04.2013 11:10, schrieb Richard Hughes: Using PackageKit and yum on the command line is often painful as we have to always download metadata unless it's less than a few hours old. Being able to update the metadata once a week would be awesome (with the possible exception of security updates) so that we could schedule the 20Mb+ metadata update when the user is idle rather than waiting for updates [root@rh:~]$ cat /etc/yum.repos.d/fedora.repo [fedora] name=Fedora $releasever - $basearch failovermethod=priority #baseurl= http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/ mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releaseverarch=$basearch enabled=1 metadata_expire=7d I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time this does not work for many reasons and having more repos would make things worser with low bandwidth because you have to load more metadata at all and if i want search for updates i use yum clean metadata yum upgrade which is a real pain on slow bandwith the reason is that the metadata getting larger and larger because all of the shiny ideas what additional ones would be nice while the developers are on LAN or at least very fast LAN networks -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On 9. 4. 2013 at 12:25:56, seth vidal wrote: On Tue, 9 Apr 2013 11:18:54 -0500 Bruno Wolff III br...@wolff.to wrote: On Wed, Apr 10, 2013 at 00:05:45 +0800, Mathieu Bridon boche...@fedoraproject.org wrote: The current behaviour would be obtained by setting it to 1, and setting it to 2 would already be a positive change as it would allow downgrading a package if the update went wrong. I don't think that is really what you want either. The idea is to keep recently obsoleted updates around, not 2 or 3 versions of everything. The change has some other benefits. Reverting bad updates in rawhide would be easier. You can use yum downgrade instead of having to going look at koji and download builds. Dealing with packages dropping out of repos when moving between test and updates. The latter issue is especially bad with branched during freezes. So - this is just an idea - and not necessarily a good one - but what about moving older pkgs which are not in the initial release repo into an updates-archive repo. We could leave the repo disabled by default and only keep 2 copies of any single pkg name in the repo at a time. That way in the best of all possible worlds you'd have at most 4 copies of a pkg in total: 1 - in the base release 'everything' repo 1 - in updates 2 - in updates-archive I'm not sure this solves the initial problem - downloading new metadata every 6 hours or so ... -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Dne 9.4.2013 18:14, Simo Sorce napsal(a): On Tue, 2013-04-09 at 17:16 +0200, Hans de Goede wrote: Hi, Am 09.04.2013 14:27, schrieb Matthew Miller: On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote: I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time. If anyone is interested in doing this, you'd be awesome. Thanks. I've heard of a plan in development about batching non-critical updates into monthly sets. It seems like these two things could go together I'm sorry, but that is a very bad idea. When users report bugs, and I mean real bugs here, like crashes or non working functionality. I always do my best to get them a fixed package asap, and AFAIK they really appreciate this. Moreover this is just a very non Fedora thing to do, one of the things Fedora is, is about being First. A lot of out users expect us to quickly give them new packages after upstream bug-fix releases. Lumping these all together in a single day in the month just does not feel like a Fedora thing to do. Also many packages in Fedora are maintained by volunteers lumping all the updates together will mean a flag day where all of the packages maintained by someone will get pushed at once, leading to a peak in work load, since despite testing, etc. There will be regressions as well as new packages sometimes leading to questions. And there also will be a peak workload a few days before the flag day to try and get things in now, instead of needing to wait a month. Having such peak workloads is not a good idea in general, and esp. not with volunteers. Can't they get them from updates-testing if they need a fix right now ? Simo. Actually, if update gets enough karma, it could be released immediately. The karma signals that somebody cares to give a karma. If there is no karma, it can be release in one month batch, since it is probably just minor, non-interesting fix. Vít -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, Apr 10, 2013 at 09:24:49 +0200, Jan Zelený jzel...@redhat.com wrote: I'm not sure this solves the initial problem - downloading new metadata every 6 hours or so ... Does the metadata really need to be downloaded or just checked to see if it is current? -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Am 10.04.2013 15:47, schrieb Bruno Wolff III: On Wed, Apr 10, 2013 at 09:24:49 +0200, Jan Zelený jzel...@redhat.com wrote: I'm not sure this solves the initial problem - downloading new metadata every 6 hours or so ... Does the metadata really need to be downloaded or just checked to see if it is current? normally not if proper implemented in the client to check last creation time FTP: no problem to check timestamp HTTP: HEAD-request and E-Tags signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, 10 Apr 2013 09:24:49 +0200 Jan Zelený jzel...@redhat.com wrote: On 9. 4. 2013 at 12:25:56, seth vidal wrote: On Tue, 9 Apr 2013 11:18:54 -0500 Bruno Wolff III br...@wolff.to wrote: On Wed, Apr 10, 2013 at 00:05:45 +0800, Mathieu Bridon boche...@fedoraproject.org wrote: The current behaviour would be obtained by setting it to 1, and setting it to 2 would already be a positive change as it would allow downgrading a package if the update went wrong. I don't think that is really what you want either. The idea is to keep recently obsoleted updates around, not 2 or 3 versions of everything. The change has some other benefits. Reverting bad updates in rawhide would be easier. You can use yum downgrade instead of having to going look at koji and download builds. Dealing with packages dropping out of repos when moving between test and updates. The latter issue is especially bad with branched during freezes. So - this is just an idea - and not necessarily a good one - but what about moving older pkgs which are not in the initial release repo into an updates-archive repo. We could leave the repo disabled by default and only keep 2 copies of any single pkg name in the repo at a time. That way in the best of all possible worlds you'd have at most 4 copies of a pkg in total: 1 - in the base release 'everything' repo 1 - in updates 2 - in updates-archive I'm not sure this solves the initial problem - downloading new metadata every 6 hours or so ... I wasn't trying to solve that problem. The problem I did solve was the updates repodata growing forever if we keep more than one version of the pkgs in there. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, 10 Apr 2013 08:47:38 -0500 Bruno Wolff III br...@wolff.to wrote: On Wed, Apr 10, 2013 at 09:24:49 +0200, Jan Zelený jzel...@redhat.com wrote: I'm not sure this solves the initial problem - downloading new metadata every 6 hours or so ... Does the metadata really need to be downloaded or just checked to see if it is current? The metadata doesn't get downloaded if it hasn't changed - the problem though is that the metadata DOES change often. Normally everyday. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
From: seth vidal skvi...@fedoraproject.org The metadata doesn't get downloaded if it hasn't changed - the problem though is that the metadata DOES change often. Normally everyday. Is there anything that could be done to make it unnecessary to pull the complete metadata for every update? For example, IIRC this is all sqlite data, but what if this was in a plain-text data dump form where something like rsync could be used to efficiently transfer only those bits that have changed. Client CPU time to reconstruct the DB is probably cheaper than the bandwidth. Maybe such a mode would only be used if the DB size exceeded some threshold. -- John Florian -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, 10 Apr 2013 10:53:25 -0400 john.flor...@dart.biz wrote: From: seth vidal skvi...@fedoraproject.org The metadata doesn't get downloaded if it hasn't changed - the problem though is that the metadata DOES change often. Normally everyday. Is there anything that could be done to make it unnecessary to pull the complete metadata for every update? For example, IIRC this is all sqlite data, but what if this was in a plain-text data dump form where something like rsync could be used to efficiently transfer only those bits that have changed. Client CPU time to reconstruct the DB is probably cheaper than the bandwidth. Maybe such a mode would only be used if the DB size exceeded some threshold. You'll have to talk to those folks who are trying to work on a delta metadata format. And rsync is not efficient from a bunch of clients to a mirror server. It would cripple a mirror server in a moment. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Mathieu Bridon (boche...@fedoraproject.org) said: On Tuesday, April 09, 2013 11:52 PM, Bill Nottingham wrote: If you wanted to keep more versions on the mirrors, you have the following options: 1) Have mash create everything, and then run a script that prunes versions older than X, and re-runs createrepo. [... snip ...] 2) Have mash try and implement a date-based expiry itself from what it requests from koji. [... snip ...] 3) Have mash sort through the packages retrieved by koji, and pull some subset less than 'all' (the last 2/3/4 versions). This is probably doable - it requires requesting all tagged packages from koji and then doing some postprocessing on the list before you download them. It's merely a matter of code. It seems to me that all of these solutions focus on making mash do more stuff, without ever changing Koji. It's the (somewhat) more agile component in that it can be changed with less downstream dependencies, fewer worries about deployment on infrastructure, etc. But couldn't the Koji API grow a new parameter, which would be used for specifying how many versions of each package to get at most? It certainly could. However, Koji's concept of 'newer' is (IIRC) entirely tied to build time/tag time in the repo, not NVR. So it may not be what we want. Bill -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On 10. 4. 2013 at 10:53:25, john.flor...@dart.biz wrote: From: seth vidal skvi...@fedoraproject.org The metadata doesn't get downloaded if it hasn't changed - the problem though is that the metadata DOES change often. Normally everyday. Is there anything that could be done to make it unnecessary to pull the complete metadata for every update? For example, IIRC this is all sqlite data, but what if this was in a plain-text data dump form where something like rsync could be used to efficiently transfer only those bits that have changed. Client CPU time to reconstruct the DB is probably cheaper than the bandwidth. Maybe such a mode would only be used if the DB size exceeded some threshold. Adding Zdenek to CC, as he is already working on a proposal to reduce metadata download size. Thanks Jan -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Once upon a time, john.flor...@dart.biz john.flor...@dart.biz said: Is there anything that could be done to make it unnecessary to pull the complete metadata for every update? For example, IIRC this is all sqlite data, but what if this was in a plain-text data dump form where something like rsync could be used to efficiently transfer only those bits that have changed. Client CPU time to reconstruct the DB is probably cheaper than the bandwidth. Maybe such a mode would only be used if the DB size exceeded some threshold. The metadata starts in XML before being loaded into an SQLite DB file, and the XML is in the repodata directory with the DB. However, both are compressed, as they are large. For example, the current updates/18/x86_64 XML is over 34M (5M gzip compressed), and the DB is 41M (9M bzip2 compressed). I'm guessing there are historical reasons why different compression is used; both could be made noticeably smaller with xz (XML to just over 3M, DB to 7M), but that's still a lot of data to download (and there are also other metadata files that have to be downloaded sometimes, especially the filelists.xml.gz, which is 10M gzip compressed). I'm not sure when the XML is downloaded instead of (or in addition to) the DB, but it does appear to happen (I see one example in my mirror server web logs this morning for example). All the metadata changes with every push (usually once per day for the updates repo), so it has to be downloaded constantly. Your only practical choice for distribution is HTTP; rsync has much higher server overhead and only available on some mirrors. If you want anything other than the full download, it'll have to be in the form of additional repodata files generated as part of the push. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
From: Chris Adams cmad...@hiwaay.net The metadata starts in XML before being loaded into an SQLite DB file, and the XML is in the repodata directory with the DB. However, both are compressed, as they are large. For example, the current updates/18/x86_64 XML is over 34M (5M gzip compressed), and the DB is 41M (9M bzip2 compressed). I'm guessing there are historical reasons why different compression is used; both could be made noticeably smaller with xz (XML to just over 3M, DB to 7M), but that's still a lot of data to download (and there are also other metadata files that have to be downloaded sometimes, especially the filelists.xml.gz, which is 10M gzip compressed). I was thinking there had been some xz integration recently. Maybe that was with the delta rpm support. I don't follow that though since we have a local mirror, there's not much point in rsycing, storing, etc. the deltas. I'm not sure when the XML is downloaded instead of (or in addition to) the DB, but it does appear to happen (I see one example in my mirror server web logs this morning for example). I've wondered about that too. I sure hope we didn't add the efficient method on top of the inefficient method, rather than replace it. -- John Florian -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, 10 Apr 2013 12:24:58 -0400 john.flor...@dart.biz wrote: I was thinking there had been some xz integration recently. Maybe that was with the delta rpm support. I don't follow that though since we have a local mirror, there's not much point in rsycing, storing, etc. the deltas. There's a rel-eng ticket to enable xz compressed repodata: https://fedorahosted.org/rel-eng/ticket/5362 Basically yum and friends should be able to handle it fine, but createrepo doesn't default to it, so out repos would be different from most everyone elses. This didn't happen for f19, but perhaps we could enable it for rawhide... kevin signature.asc Description: PGP signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, 10 Apr 2013 10:33:43 -0500 Chris Adams cmad...@hiwaay.net wrote: The metadata starts in XML before being loaded into an SQLite DB file, and the XML is in the repodata directory with the DB. However, both are compressed, as they are large. For example, the current updates/18/x86_64 XML is over 34M (5M gzip compressed), and the DB is 41M (9M bzip2 compressed). I'm guessing there are historical reasons why different compression is used; both could be made noticeably smaller with xz (XML to just over 3M, DB to 7M), but that's still a lot of data to download (and there are also other metadata files that have to be downloaded sometimes, especially the filelists.xml.gz, which is 10M gzip compressed). I'm not sure when the XML is downloaded instead of (or in addition to) the DB, but it does appear to happen (I see one example in my mirror server web logs this morning for example). Here's how it works. the xml metadata put together over a decade ago. It is the canonical representation of the metadata. The sqlite was added maybe 8ish years ago as a way of more quickly reading the same data and not eating up so much memory. At the time bzip2 was the new hotness so we used it instead of gz. the primary, filelists and other xml should not ever be downloaded at this point unless you hit a mirror which is out of sync, badly. the only xml files that should be getting downloaded: 1. repomd.xml - it's fairly small and the index for everything else 2. comps.xml (or groups.xml) - which is where comps is stored per-repo 3. updatemd.xml which is just the security/update info for describing updates yum will grab repomd.xml and look to see if it is newer than what it has already. Then go from there about updating the rest of the metadata. Hope that helps explain it a bit more. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, 10 Apr 2013 12:24:58 -0400 john.flor...@dart.biz wrote: I was thinking there had been some xz integration recently. Maybe that was with the delta rpm support. I don't follow that though since we have a local mirror, there's not much point in rsycing, storing, etc. the deltas. yum and createrepo both support xz now - we can really only do it from f18-f19 iirc - just ot make sure everyone has the required version of yum and friends. I'm not sure when the XML is downloaded instead of (or in addition to) the DB, but it does appear to happen (I see one example in my mirror server web logs this morning for example). I've wondered about that too. I sure hope we didn't add the efficient method on top of the inefficient method, rather than replace it. See my earlier email explaining how that works. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Am 09.04.2013 17:02, schrieb Richard Hughes: On 9 April 2013 13:48, Reindl Harald h.rei...@thelounge.net wrote: if i want monthly pacthdays i use Microsoft or Oracle Not at all. Patchdays make perfect sense for planning reboots/downtime/maintenance and that kind of thing. there where i need this test-machines and internal repos exists but i do NOT need anybody to hold back updates for me you can hardly classify which bug is for which user critical! Sure you can, Red Hat does it for updates in RHEL, and you just have to define what the terms mean. 90% of the updates I'm looking at right now for F18 are really not important at all. Spec file fixups, new versions without bugfixes, updated artwork; that can all wait until a certain point in the month but Fedora IS NOT RHEL if you want the RHEL way use it signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On 9 April 2013 10:21, Reindl Harald h.rei...@thelounge.net wrote: metadata_expire=7d From a package manager point of view, this happens: 1 check expire timeout, all okay 2 depsolve update set 3 download 4 package not found! 5 download needed metadata based on some heuristic 6 goto 2 Richard -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote: I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time. If anyone is interested in doing this, you'd be awesome. Thanks. I've heard of a plan in development about batching non-critical updates into monthly sets. It seems like these two things could go together. -- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mat...@fedoraproject.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Tue, Apr 09, 2013 at 02:48:40PM +0200, Reindl Harald wrote: depends often on the workload and currect jobs and the cirtical apllication wheer a bug will hurt you much may change from project to project As I understand it, you'll be able to opt for an all-updates track. In fact, that will be encouraged for active developers and testers. there are months where i would not care if LibreOffice does not start at all and there are weeks where i need it all day long So, presumably, you don't want it gratuitously updating during those times you need it all day long, and you don't care either way the rest of the time. Sounds perfect. -- Matthew Miller mat...@mattdm.org http://mattdm.org/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Am 09.04.2013 11:10, schrieb Richard Hughes: Using PackageKit and yum on the command line is often painful as we have to always download metadata unless it's less than a few hours old. Being able to update the metadata once a week would be awesome (with the possible exception of security updates) so that we could schedule the 20Mb+ metadata update when the user is idle rather than waiting for updates [root@rh:~]$ cat /etc/yum.repos.d/fedora.repo [fedora] name=Fedora $releasever - $basearch failovermethod=priority #baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/ mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releaseverarch=$basearch enabled=1 metadata_expire=7d I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time this does not work for many reasons and having more repos would make things worser with low bandwidth because you have to load more metadata at all and if i want search for updates i use yum clean metadata yum upgrade which is a real pain on slow bandwith the reason is that the metadata getting larger and larger because all of the shiny ideas what additional ones would be nice while the developers are on LAN or at least very fast LAN networks signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Am 09.04.2013 14:27, schrieb Matthew Miller: On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote: I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time. If anyone is interested in doing this, you'd be awesome. Thanks. I've heard of a plan in development about batching non-critical updates into monthly sets. It seems like these two things could go together a terrible idea for a distribution with a new release all 6 months if i want monthly pacthdays i use Microsoft or Oracle you can hardly classify which bug is for which user critical! depends often on the workload and currect jobs and the cirtical apllication wheer a bug will hurt you much may change from project to project there are months where i would not care if LibreOffice does not start at all and there are weeks where i need it all day long signature.asc Description: OpenPGP digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On 9 April 2013 13:48, Reindl Harald h.rei...@thelounge.net wrote: if i want monthly pacthdays i use Microsoft or Oracle Not at all. Patchdays make perfect sense for planning reboots/downtime/maintenance and that kind of thing. you can hardly classify which bug is for which user critical! Sure you can, Red Hat does it for updates in RHEL, and you just have to define what the terms mean. 90% of the updates I'm looking at right now for F18 are really not important at all. Spec file fixups, new versions without bugfixes, updated artwork; that can all wait until a certain point in the month. Richard. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Hi, Am 09.04.2013 14:27, schrieb Matthew Miller: On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote: I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time. If anyone is interested in doing this, you'd be awesome. Thanks. I've heard of a plan in development about batching non-critical updates into monthly sets. It seems like these two things could go together I'm sorry, but that is a very bad idea. When users report bugs, and I mean real bugs here, like crashes or non working functionality. I always do my best to get them a fixed package asap, and AFAIK they really appreciate this. Moreover this is just a very non Fedora thing to do, one of the things Fedora is, is about being First. A lot of out users expect us to quickly give them new packages after upstream bug-fix releases. Lumping these all together in a single day in the month just does not feel like a Fedora thing to do. Also many packages in Fedora are maintained by volunteers lumping all the updates together will mean a flag day where all of the packages maintained by someone will get pushed at once, leading to a peak in work load, since despite testing, etc. There will be regressions as well as new packages sometimes leading to questions. And there also will be a peak workload a few days before the flag day to try and get things in now, instead of needing to wait a month. Having such peak workloads is not a good idea in general, and esp. not with volunteers. Regards, Hans -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On 9 April 2013 16:16, Hans de Goede hdego...@redhat.com wrote: them new packages after upstream bug-fix releases. Lumping these all together in a single day in the month just does not feel like a Fedora thing to do. You can't QA a trickle. If packages are small and self-contained then sure, it might work, but applications depending on ~30 other libraries and ~20 other daemons not so much. Richard. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Richard Hughes (hughsi...@gmail.com) said: Using PackageKit and yum on the command line is often painful as we have to always download metadata unless it's less than a few hours old. Being able to update the metadata once a week would be awesome (with the possible exception of security updates) so that we could schedule the 20Mb+ metadata update when the user is idle rather than waiting for updates. I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time. If anyone is interested in doing this, you'd be awesome. Thanks. The thing that creates the repos is 'mash', so that's where such a change needs to be. It currently talks to koji to get the builds for a particular tag, and has the following config settings: latest = True Grab only the latest package latest = False Grab *all* the versions of the packages. It only supports these two variations, because that's what koji supports. If you wanted to keep more versions on the mirrors, you have the following options: 1) Have mash create everything, and then run a script that prunes versions older than X, and re-runs createrepo. We used to do this in the old days. It's an utter hack, and not very efficient. 2) Have mash try and implement a date-based expiry itself from what it requests from koji. The problem here is that mash pulls from koji, and only has two times that it could use as a key: - build time (not really what you want, as packages can build and sit for a while) - time it was tagged into the repo (which is affected by rel-eng operations if packages need to be retagged or moved) 3) Have mash sort through the packages retrieved by koji, and pull some subset less than 'all' (the last 2/3/4 versions). This is probably doable - it requires requesting all tagged packages from koji and then doing some postprocessing on the list before you download them. It's merely a matter of code. So, needless to say, I'd suggest anyone interested in this to look at option #3. Note that enabling something like that on rawhide would have a large effect on the repository creation time - there's only so many ways to speed up scanning across 5 packages 100GB on a SAN. Bill (former mash maintainer) -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Tuesday, April 09, 2013 11:52 PM, Bill Nottingham wrote: If you wanted to keep more versions on the mirrors, you have the following options: 1) Have mash create everything, and then run a script that prunes versions older than X, and re-runs createrepo. [... snip ...] 2) Have mash try and implement a date-based expiry itself from what it requests from koji. [... snip ...] 3) Have mash sort through the packages retrieved by koji, and pull some subset less than 'all' (the last 2/3/4 versions). This is probably doable - it requires requesting all tagged packages from koji and then doing some postprocessing on the list before you download them. It's merely a matter of code. It seems to me that all of these solutions focus on making mash do more stuff, without ever changing Koji. But couldn't the Koji API grow a new parameter, which would be used for specifying how many versions of each package to get at most? The current behaviour would be obtained by setting it to 1, and setting it to 2 would already be a positive change as it would allow downgrading a package if the update went wrong. -- Mathieu -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Tue, Apr 09, 2013 at 05:16:45PM +0200, Hans de Goede wrote: I've heard of a plan in development about batching non-critical updates into monthly sets. It seems like these two things could go together I'm sorry, but that is a very bad idea. When users report bugs, and I mean real bugs here, like crashes or non working functionality. I always do my best to get them a fixed package asap, and AFAIK they really appreciate this. To be clear, the plan I heard (which isn't mine and I don't think is finished anyway) isn't to *withhold* updates until a certain date; it's to batch them up and make them available as a collection by default. If want all *or some* updates as soon as they become available, you could still do that. Also many packages in Fedora are maintained by volunteers lumping all the updates together will mean a flag day where all of the packages maintained by someone will get pushed at once, leading to a peak in work load, since despite testing, etc. There will be regressions as well as new packages sometimes leading to questions. And there also will be a peak workload a few days before the flag day to try and get things in now, instead of needing to wait a month. Having such peak workloads is not a good idea in general, and esp. not with volunteers. Overall, it's a more predictable workload, which *is* a good idea, for both volunteer and otherwise. It's also more effective to QA packages in sets, and more effective can mean more efficient. -- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mat...@fedoraproject.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Tue, 2013-04-09 at 17:16 +0200, Hans de Goede wrote: Hi, Am 09.04.2013 14:27, schrieb Matthew Miller: On Tue, Apr 09, 2013 at 10:10:26AM +0100, Richard Hughes wrote: I'm wondering what the interest would be in keeping packages referenced in metadata on the mirrors for say a month? We'd probably need a separate fedora-security repo too that's designed to be kept small enough so that metadata checks every day would be not costly in terms of bandwidth and time. If anyone is interested in doing this, you'd be awesome. Thanks. I've heard of a plan in development about batching non-critical updates into monthly sets. It seems like these two things could go together I'm sorry, but that is a very bad idea. When users report bugs, and I mean real bugs here, like crashes or non working functionality. I always do my best to get them a fixed package asap, and AFAIK they really appreciate this. Moreover this is just a very non Fedora thing to do, one of the things Fedora is, is about being First. A lot of out users expect us to quickly give them new packages after upstream bug-fix releases. Lumping these all together in a single day in the month just does not feel like a Fedora thing to do. Also many packages in Fedora are maintained by volunteers lumping all the updates together will mean a flag day where all of the packages maintained by someone will get pushed at once, leading to a peak in work load, since despite testing, etc. There will be regressions as well as new packages sometimes leading to questions. And there also will be a peak workload a few days before the flag day to try and get things in now, instead of needing to wait a month. Having such peak workloads is not a good idea in general, and esp. not with volunteers. Can't they get them from updates-testing if they need a fix right now ? Simo. -- Simo Sorce * Red Hat, Inc * New York -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wed, Apr 10, 2013 at 00:05:45 +0800, Mathieu Bridon boche...@fedoraproject.org wrote: The current behaviour would be obtained by setting it to 1, and setting it to 2 would already be a positive change as it would allow downgrading a package if the update went wrong. I don't think that is really what you want either. The idea is to keep recently obsoleted updates around, not 2 or 3 versions of everything. The change has some other benefits. Reverting bad updates in rawhide would be easier. You can use yum downgrade instead of having to going look at koji and download builds. Dealing with packages dropping out of repos when moving between test and updates. The latter issue is especially bad with branched during freezes. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Wednesday, April 10, 2013 12:09 AM, Matthew Miller wrote: On Tue, Apr 09, 2013 at 05:16:45PM +0200, Hans de Goede wrote: I've heard of a plan in development about batching non-critical updates into monthly sets. It seems like these two things could go together I'm sorry, but that is a very bad idea. When users report bugs, and I mean real bugs here, like crashes or non working functionality. I always do my best to get them a fixed package asap, and AFAIK they really appreciate this. To be clear, the plan I heard (which isn't mine and I don't think is finished anyway) isn't to *withhold* updates until a certain date; it's to batch them up and make them available as a collection by default. If want all *or some* updates as soon as they become available, you could still do that. For what it's worth, this is exactly what we do at $dayjob. The way I set our repos up is that there are 1 testing repo, and 2 stable repos: current and released. We use Bodhi to move things from testing to current, as they get QA-ed, just like in Fedora. But by default, neither the current nor testing repositories are enabled. Only the released repository is. Once a month, we synchronize current into released. This way, we have the monthly « Patch Tuesday » by default, and it's trivial to move to the model of getting updates as they are published if you want to. It also allows users to be selective: they can get an update without waiting if it's important to them, but without updating everything else right now. -- Mathieu -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Tue, 9 Apr 2013 11:18:54 -0500 Bruno Wolff III br...@wolff.to wrote: On Wed, Apr 10, 2013 at 00:05:45 +0800, Mathieu Bridon boche...@fedoraproject.org wrote: The current behaviour would be obtained by setting it to 1, and setting it to 2 would already be a positive change as it would allow downgrading a package if the update went wrong. I don't think that is really what you want either. The idea is to keep recently obsoleted updates around, not 2 or 3 versions of everything. The change has some other benefits. Reverting bad updates in rawhide would be easier. You can use yum downgrade instead of having to going look at koji and download builds. Dealing with packages dropping out of repos when moving between test and updates. The latter issue is especially bad with branched during freezes. So - this is just an idea - and not necessarily a good one - but what about moving older pkgs which are not in the initial release repo into an updates-archive repo. We could leave the repo disabled by default and only keep 2 copies of any single pkg name in the repo at a time. That way in the best of all possible worlds you'd have at most 4 copies of a pkg in total: 1 - in the base release 'everything' repo 1 - in updates 2 - in updates-archive -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
Once upon a time, Bill Nottingham nott...@redhat.com said: So, needless to say, I'd suggest anyone interested in this to look at option #3. Note that enabling something like that on rawhide would have a large effect on the repository creation time - there's only so many ways to speed up scanning across 5 packages 100GB on a SAN. It would also have a significant impact on the mirror servers' disk space (speaking for a mirror that is running low on disk and no $$ to upgrade). -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Keeping old versions of packages
On Tuesday 09 April 2013 16:02:14 Richard Hughes wrote: now for F18 are really not important at all. Spec file fixups, new versions without bugfixes, updated artwork; that can all wait until a certain point in the month. Security-critical updates are already tagged as such, aren't they? So users are already able to defer installing non-critical updates to a point in time where it's convenient for them. There's even a yum plugin for that. Maybe the existing tools could be enhanced but changing stuff on the repo side? I'm not so sure... Lars -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
