Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Sat, Oct 16, 2021 at 11:09 AM Richard W.M. Jones wrote: > > Looks like several people have reinvented wheels here because Fedora > doesn't really offer this. I've wanted to make more use of virt-builder. Here is the bug that I last encountered with this: https://bugzilla.redhat.com/show_bug.cgi?id=1834972 It would be great if Pungi (or even a script in https://pagure.io/compose-utils) could generate the virt-builder files automatically. Then the virt-builder indexes would happen for every compose, and the CentOS teams, Fedora teams, etc. could work from a shared tool. I guess the wrinkle there is probably GPG signing, like https://pagure.io/pungi/issue/506 - Ken ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Tue, Oct 12, 2021 at 01:52:01PM -0400, Neal Gompa wrote: > Hey all, > > I'm working on extending quickemu[1] to be able to easily spin up > Fedora VMs, but our lack of a static URL formula for fetching ISOs > makes this a bit difficult. > > Do we have some kind of API endpoint that has the necessary > information for this? It'd be nice to be able to fetch some kind of > JSON file with the necessary information so that tools can fetch it > programmatically... > > Thanks in advance and best regards! > > [1]: https://github.com/wimpysworld/quickemu Virt-builder also has metadata for the images we release (which are disk templates, not ISOs). https://builder.libguestfs.org/index.asc https://libguestfs.org/virt-builder.1.html#sources-of-templates Looks like several people have reinvented wheels here because Fedora doesn't really offer this. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Tue, Oct 12, 2021, at 1:52 PM, Neal Gompa wrote: > Hey all, > > I'm working on extending quickemu[1] to be able to easily spin up > Fedora VMs, but our lack of a static URL formula for fetching ISOs > makes this a bit difficult. > > Do we have some kind of API endpoint that has the necessary > information for this? It'd be nice to be able to fetch some kind of > JSON file with the necessary information so that tools can fetch it > programmatically... For Fedora CoreOS we have standard stream metadata for this purpose: https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/#_streams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Wed, 2021-10-13 at 14:45 -0400, Ken Dreyer wrote:
> On Wed, Oct 13, 2021 at 2:21 PM Neal Gompa wrote:
> > Unfortunately, we lack a usable equivalent for releases, though that's
> > good to know for Rawhide.
>
> For that I use the following URL format string:
>
> https://download.fedoraproject.org/pub/fedora/linux/releases/{version}/Cloud/x86_64/images/Fedora-Cloud-Base-{version}-1.2.x86_64.qcow2
That's not reliable. It will only work if we happened to release the
second candidate compose. That's certainly not the case for every
release.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Wed, Oct 13, 2021 at 2:21 PM Neal Gompa wrote:
> Unfortunately, we lack a usable equivalent for releases, though that's
> good to know for Rawhide.
For that I use the following URL format string:
https://download.fedoraproject.org/pub/fedora/linux/releases/{version}/Cloud/x86_64/images/Fedora-Cloud-Base-{version}-1.2.x86_64.qcow2
- Ken
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Wed, Oct 13, 2021 at 2:14 PM Ken Dreyer wrote:
>
> On Tue, Oct 12, 2021 at 3:26 PM Adam Williamson
> wrote:
> > Hanging over all of this is the threat that PDC might go away at some
> > point, which would be a bit of an inconvenience. In A World Where there
> > is no PDC, you have to grab the metadata files for composes that still
> > exist from kojipkgs; there is no record of the metadata for composes
> > that have been garbage-collected. For stable releases you'd have to
> > parse whatever metadata you can just out of the actual release tree on
> > the mirrors.
>
> Yeah, PDC is a big liability to Fedora at this point and we should run
> away from it.
>
> Here's the code I use to download Rawhide images:
>
> get_rawhide_image_url():
> # We don't have productmd data available, so we must download COMPOSE_ID
> # and interpolate that directly into the assumed URL.
> r =
> requests.get('https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/COMPOSE_ID')
> r.raise_for_status()
> compose_id = r.text.strip()
> _, _, numeric = compose_id.split('-')
> url =
> 'https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-%s.x86_64.qcow2'
> % numeric
> return url
>
Unfortunately, we lack a usable equivalent for releases, though that's
good to know for Rawhide.
--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Tue, Oct 12, 2021 at 3:26 PM Adam Williamson
wrote:
> Hanging over all of this is the threat that PDC might go away at some
> point, which would be a bit of an inconvenience. In A World Where there
> is no PDC, you have to grab the metadata files for composes that still
> exist from kojipkgs; there is no record of the metadata for composes
> that have been garbage-collected. For stable releases you'd have to
> parse whatever metadata you can just out of the actual release tree on
> the mirrors.
Yeah, PDC is a big liability to Fedora at this point and we should run
away from it.
Here's the code I use to download Rawhide images:
get_rawhide_image_url():
# We don't have productmd data available, so we must download COMPOSE_ID
# and interpolate that directly into the assumed URL.
r =
requests.get('https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/COMPOSE_ID')
r.raise_for_status()
compose_id = r.text.strip()
_, _, numeric = compose_id.split('-')
url =
'https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-%s.x86_64.qcow2'
% numeric
return url
- Ken
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Wed, 2021-10-13 at 08:11 -0400, Stephen Gallagher wrote: > On Tue, Oct 12, 2021 at 2:42 PM Frantisek Zatloukal > wrote: > > > > Hi, > > > > in testcloud ( > > https://pagure.io/testcloud/blob/master/f/testcloud/util.py#_100 ), I am > > using adam's openqa nightlies.json for rawhide/branched: > > https://openqa.fedoraproject.org/nightlies.json (this isn't a "stable api") > > > > and https://getfedora.org/releases.json for stable releases. > > > > For programmatically determining which are the current fedora releases, I > > am using oraculum's > > https://packager-dashboard.fedoraproject.org/api/v1/releases (this wouldn't > > change nor break). > > For what it's worth, Bodhi provides a stable API for getting the list > of current Fedora releases. See my Github Action at > https://github.com/sgallagher/get-fedora-releases-action/blob/main/get_fedora_releases.py > (which I use to programmatically keep my CI tests running on all > active releases). The main problem I faced with using Bodhi for this purpose is that in Bodhi a new release gets set to stable state quite 'early'. Specifically, it gets marked as stable before the official release date, and so before the release tree is actually available in the expected location on public mirrors. I used Bodhi as the data source for fedfind for a couple of cycles, but both times this caused problems, which is why I stopped doing that. My long-term plan for making this whole area better is to get releasestream[0] deployed and onboard a bunch of the releng processes to it, but I haven't had any roundtuits to work on that lately. [0] https://pagure.io/fedora-qa/releasestream -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Tue, Oct 12, 2021 at 2:42 PM Frantisek Zatloukal wrote: > > Hi, > > in testcloud ( > https://pagure.io/testcloud/blob/master/f/testcloud/util.py#_100 ), I am > using adam's openqa nightlies.json for rawhide/branched: > https://openqa.fedoraproject.org/nightlies.json (this isn't a "stable api") > > and https://getfedora.org/releases.json for stable releases. > > For programmatically determining which are the current fedora releases, I am > using oraculum's https://packager-dashboard.fedoraproject.org/api/v1/releases > (this wouldn't change nor break). For what it's worth, Bodhi provides a stable API for getting the list of current Fedora releases. See my Github Action at https://github.com/sgallagher/get-fedora-releases-action/blob/main/get_fedora_releases.py (which I use to programmatically keep my CI tests running on all active releases). ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Tue, 2021-10-12 at 19:12 -0400, Neal Gompa wrote: > On Tue, Oct 12, 2021 at 3:25 PM Adam Williamson > wrote: > > > > So to answer Neal's question: you can use PDC directly, or you can use > > fedfind which will talk to PDC for you, plus do a bunch of other stuff, > > like the above, and also synthesizing metadata for pre-Pungi 4 stable > > releases so you can interact with them via fedfind using the same > > interfaces as post-Pungi 4 composes... > > > > Hanging over all of this is the threat that PDC might go away at some > > point, which would be a bit of an inconvenience. In A World Where there > > is no PDC, you have to grab the metadata files for composes that still > > exist from kojipkgs; there is no record of the metadata for composes > > that have been garbage-collected. For stable releases you'd have to > > parse whatever metadata you can just out of the actual release tree on > > the mirrors. > > > > Maybe it might be worth considering to generating a json blob to put > into the mirrored tree? You mean the stable release trees, after the messing around we do? I guess possibly. I mean, in theory, fedfind could write its reconciled metadata back out again and we could stuff that somewhere. It's just never really been considered, I guess. > The problem I *actually* have is that we don't > have a predictable/reliable name that I can hard-code in to correctly > fetch the ISO. This is more or less the job I wrote fedfind to do, so it's probably the thing you want. It's just a question of deciding what "the ISO" you want is, exactly, and how to tell that to fedfind. > We don't generate unversioned links to versioned ISO > files, which is where the complication comes from. I guess this is also something that could theoretically be done, though I dunno how complex it'd be for releng. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Tue, Oct 12, 2021 at 3:25 PM Adam Williamson wrote: > > So to answer Neal's question: you can use PDC directly, or you can use > fedfind which will talk to PDC for you, plus do a bunch of other stuff, > like the above, and also synthesizing metadata for pre-Pungi 4 stable > releases so you can interact with them via fedfind using the same > interfaces as post-Pungi 4 composes... > > Hanging over all of this is the threat that PDC might go away at some > point, which would be a bit of an inconvenience. In A World Where there > is no PDC, you have to grab the metadata files for composes that still > exist from kojipkgs; there is no record of the metadata for composes > that have been garbage-collected. For stable releases you'd have to > parse whatever metadata you can just out of the actual release tree on > the mirrors. > Maybe it might be worth considering to generating a json blob to put into the mirrored tree? The problem I *actually* have is that we don't have a predictable/reliable name that I can hard-code in to correctly fetch the ISO. We don't generate unversioned links to versioned ISO files, which is where the complication comes from. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
On Tue, 2021-10-12 at 20:40 +0200, Frantisek Zatloukal wrote: > Hi, > > in testcloud ( > https://pagure.io/testcloud/blob/master/f/testcloud/util.py#_100 ), I am > using adam's openqa nightlies.json for rawhide/branched: > https://openqa.fedoraproject.org/nightlies.json (this isn't a "stable api") > > and https://getfedora.org/releases.json for stable releases. > > For programmatically determining which are the current fedora releases, I > am using oraculum's > https://packager-dashboard.fedoraproject.org/api/v1/releases (this wouldn't > change nor break). > > Hope it helped at least a bit! Oh, that's what you're using it for? Hmm. Well, that's a bit indirect... Both of those sources are ultimately getting the data via my fedfind tool: https://pagure.io/fedora-qa/fedfind so unless you actually need the extra information nightlies adds (i.e. whether the images passed openQA tests), you might as well just use fedfind directly. fedfind's current source of truth for "what are the current releases" is a JSON blob I maintain by hand: https://fedorapeople.org/groups/qa/metadata/release.json which you could also just query directly (to figure out the Rawhide release number, add one to the otherwise highest release number). There are alternative sources of truth for this, but they're all a bit problematic. I have a speech I can give on this if you'd like to hear it. :D To go back to Neal's original question, there more or less *is* an API for this, yes: PDC - https://pdc.fedoraproject.org/ Specifically, you want the compose-images endpoint: https://pdc.fedoraproject.org/rest_api/v1/compose-images/ you pass it a compose ID, like this: https://pdc.fedoraproject.org/rest_api/v1/compose-images/Fedora-Rawhide-20211012.n.0/ and get lots of lovely data. What it's actually giving you is the data produced by pungi and included in the compose tree itself so long as that's available: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20211012.n.0/compose/metadata/images.json but PDC keeps the metadata around forever, while the composes disappear after a couple of weeks. Querying stable releases is a mite tricky. For a start, you need to know the compose ID of the candidate compose that we ultimately released. fedfind does this by parsing the compose label out of the image filenames (using a regex with an expected format), then using PDC to find the compose ID from the label. For a second, the data you get will be in some senses a lie, because we hack up the compose when we ship it as a stable release: we throw away some deliverables (so they'll be in the metadata but you won't be able to find the files anywhere once they're purged from kojipkgs / stage), and move others around (so some of the relative paths in the metadata are wrong). fedfind has some clever-clever logic to kinda meld what it knows about the actual layout of the stable compose tree with the metadata from PDC and present a more-or-less correct view of things. So to answer Neal's question: you can use PDC directly, or you can use fedfind which will talk to PDC for you, plus do a bunch of other stuff, like the above, and also synthesizing metadata for pre-Pungi 4 stable releases so you can interact with them via fedfind using the same interfaces as post-Pungi 4 composes... Hanging over all of this is the threat that PDC might go away at some point, which would be a bit of an inconvenience. In A World Where there is no PDC, you have to grab the metadata files for composes that still exist from kojipkgs; there is no record of the metadata for composes that have been garbage-collected. For stable releases you'd have to parse whatever metadata you can just out of the actual release tree on the mirrors. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
Hi, in testcloud ( https://pagure.io/testcloud/blob/master/f/testcloud/util.py#_100 ), I am using adam's openqa nightlies.json for rawhide/branched: https://openqa.fedoraproject.org/nightlies.json (this isn't a "stable api") and https://getfedora.org/releases.json for stable releases. For programmatically determining which are the current fedora releases, I am using oraculum's https://packager-dashboard.fedoraproject.org/api/v1/releases (this wouldn't change nor break). Hope it helped at least a bit! On Tue, Oct 12, 2021 at 7:53 PM Neal Gompa wrote: > Hey all, > > I'm working on extending quickemu[1] to be able to easily spin up > Fedora VMs, but our lack of a static URL formula for fetching ISOs > makes this a bit difficult. > > Do we have some kind of API endpoint that has the necessary > information for this? It'd be nice to be able to fetch some kind of > JSON file with the necessary information so that tools can fetch it > programmatically... > > Thanks in advance and best regards! > > [1]: https://github.com/wimpysworld/quickemu > > -- > 真実はいつも一つ！/ Always, there's only one truth! > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > -- Best regards / S pozdravem, František Zatloukal Quality Engineer Red Hat ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
API endpoint listing ISOs and checksums for Fedora releases and Rawhide?
Hey all, I'm working on extending quickemu[1] to be able to easily spin up Fedora VMs, but our lack of a static URL formula for fetching ISOs makes this a bit difficult. Do we have some kind of API endpoint that has the necessary information for this? It'd be nice to be able to fetch some kind of JSON file with the necessary information so that tools can fetch it programmatically... Thanks in advance and best regards! [1]: https://github.com/wimpysworld/quickemu -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
