Re: Packaging guidelines - validation of AppStream metadata files
On Wed, Oct 25, 2023 at 3:23 PM Daniel Alley wrote: > > >There are arguments > > about whether it's appstream-builder specific (given that Debian's > > archive is bigger, their package format is more complex to "pluck" > > from, etc and appstream-generator does fine there), but the point is > > that for Fedora with appstream-builder, it's too slow. Note that the > > context of this is back in the era where Fedora composes took half a > > day or longer. Adding even more hours to that process was not > > appealing (it would have to be done for every repo being composed for > > every architecture...). > > So, if it's theoretically possible to be much faster than it is, has anyone > sat down with a profiler and tried to improve appstream-builder? Nope. If someone is interested in doing so, they can now. :) -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
>There are arguments > about whether it's appstream-builder specific (given that Debian's > archive is bigger, their package format is more complex to "pluck" > from, etc and appstream-generator does fine there), but the point is > that for Fedora with appstream-builder, it's too slow. Note that the > context of this is back in the era where Fedora composes took half a > day or longer. Adding even more hours to that process was not > appealing (it would have to be done for every repo being composed for > every architecture...). So, if it's theoretically possible to be much faster than it is, has anyone sat down with a profiler and tried to improve appstream-builder? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
Seconding that, the did, at least in the past, and they do interpret the severity of deviations differently than the written standard. On 24.10.23 14:26, Michael Catanzaro wrote: On Tue, Oct 24 2023 at 08:06:12 AM -0400, Neal Gompa wrote: The two tools don't have incompatible ideas of valid metadata, we intentionally don't do strict validation. Well for one example incompatibility, you can review that issue: https://github.com/ximion/appstream/issues/476 Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Tue, Oct 24 2023 at 08:06:12 AM -0400, Neal Gompa wrote: The two tools don't have incompatible ideas of valid metadata, we intentionally don't do strict validation. Well for one example incompatibility, you can review that issue: https://github.com/ximion/appstream/issues/476 Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Fri, Sep 29, 2023 at 12:39 PM Kevin Fenzi wrote: > > On Thu, Sep 28, 2023 at 01:03:08PM +0100, Richard Hughes wrote: > > On Wed, 27 Sept 2023 at 22:41, Kevin Fenzi wrote: > > > I'm not sure at all that it would be possible to do at compose-time... > > > composes are taking around 3.5-4hours and thats after I have done > > > a lot to speed them up, but might be worth some benchmarking > > > to see how much slower it would be. If we are going to go that route, we > > > should see if support could be added to pungi. > > > > On server class hardware with fast disk access it's an additional 10 > > minutes or so. > > > > > Moving this from a package to in repodata would grow the repodata quite > > > a lot right? > > > > 6.9Mfedora-39-icons.tar.gz > > 6.3Mfedora-39.xml.gz > > So, on re-reading this thread and the upstream issues... it really seems > like the best case would be createrepo_c just adding support, but that > seems stalled/blocked due to i/o concerns? (which I can understand, but > it could be default off?) > > How does the local/rpm data interact with the repodata (if it exists)? > Or does it depend on the tool? > The data works by keying off "pkgname" and "source_pkgname" attributes that are merged into the AppStream metainfo files when building AppStream repodata. Software centers like Plasma Discover and GNOME Software will lookup these attributes for a given AppStream definition for an application and use that to request to install the package for the application. To put it more succinctly: components.component.pkgname (AppStream) maps to package.name (RPM-MD) directly. The AppStream repodata lag is tolerable because the mapping isn't to a full NEVRA. The software centers figure that out themselves on the fly. > I suppose in the short term at least if you wanted you could hand off > the generation/rpm updating to releng? (Not that releng wants more to > do, but I don't think you should carry this by yourself). > I could probably help since I know how it works, if needed/wanted. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Tue, Oct 24, 2023 at 7:54 AM Michael Catanzaro wrote: > > On Mon, Sep 25 2023 at 09:15:37 PM -0400, Neal Gompa > wrote: > > There was no switching. Both appstream-util and appstreamcli are > > considered conformant. > > > > Ultimately, the only way we can stop relying on appstream-glib is if > > appstream-builder[1] was reimplemented on top of libappstream-compose. > > > > Long ago, I tried to see if we could have our AppStream data produced > > as repodata instead of a package, but the answer I got was that we > > need the functionality added to createrepo to do it[2]. If someone > > ever got around to making that happen, a lot of things in our pipeline > > could be simplified. > > > > [1]: > > https://github.com/hughsie/appstream-glib/tree/main/libappstream-builder > > [2]: https://github.com/rpm-software-management/createrepo_c/issues/75 > > Hi, > > Matthias Klumpp has suggested using a tool called appstream-generator: > > https://github.com/ximion/appstream/issues/476#issuecomment-1776286096 > > It's pretty clear the two tools have incompatible ideas of what > metadata is valid, and the apptsream-glib repo has a big warning that > you shouldn't actually use it, so upstreams will continue to migrate > from appstream-util to appstreamcli regardless of what Fedora does. > We'll probably encounter more problems as time goes on. > I know appstream-generator (I'm the maintainer of it in Fedora). The RPM backend is not equivalent to the one we have for appstream-builder. It may or may not need work for us to use it. The two tools don't have incompatible ideas of valid metadata, we intentionally don't do strict validation. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Mon, Sep 25 2023 at 09:15:37 PM -0400, Neal Gompa wrote: There was no switching. Both appstream-util and appstreamcli are considered conformant. Ultimately, the only way we can stop relying on appstream-glib is if appstream-builder[1] was reimplemented on top of libappstream-compose. Long ago, I tried to see if we could have our AppStream data produced as repodata instead of a package, but the answer I got was that we need the functionality added to createrepo to do it[2]. If someone ever got around to making that happen, a lot of things in our pipeline could be simplified. [1]: https://github.com/hughsie/appstream-glib/tree/main/libappstream-builder [2]: https://github.com/rpm-software-management/createrepo_c/issues/75 Hi, Matthias Klumpp has suggested using a tool called appstream-generator: https://github.com/ximion/appstream/issues/476#issuecomment-1776286096 It's pretty clear the two tools have incompatible ideas of what metadata is valid, and the apptsream-glib repo has a big warning that you shouldn't actually use it, so upstreams will continue to migrate from appstream-util to appstreamcli regardless of what Fedora does. We'll probably encounter more problems as time goes on. Michael ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
appstream does not allow for being an email address, how to validate? (was: Packaging guidelines - validation of AppStream metadata files)
Hi list,
just a heads up: because we're talking about how to validate appstream metadata: When
validating metadata, one of the "default" warnings you get is:
> In the past, mailto: URL schemas to link to email addresses were also supported for
this URL type. It is recommended to not use them in new metadata, as they provide poor
usability on most systems when users click on such a link and no local email client is
configured.
I just got word from upstream that the official standing is that a point of contact for a
software package can not validly be an email address:
https://github.com/ximion/appstream/issues/331
I believe they do that in good faith ("most probably don't even have an email client set
up"), but this basically means I can not write *correct* and *valid* metadata at the same
time:
Our whole social maintenance infrastructure (and, debian all the same) is built on the
fact that packagers have email addresses. It's the *one* common medium.
I've hence asked whether that ruling could be softened; after all, a user interface
problem ("can't open mail client upon a click") is not the same as a metadata issue.
Sadly, appstream maintenance and I seem to disagree here.
Either way, as far as I understand we aim for zero-warning, yet correct metadata. I'm
involved in upstream packaging of software myself, so I wonder how I can make my upstream
metainfo ideally 100% applicable for fedora, yet correct. Email *is* the way to reach the
software maintainers.
Best,
Marcus
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Thu, Sep 28, 2023 at 01:03:08PM +0100, Richard Hughes wrote: > On Wed, 27 Sept 2023 at 22:41, Kevin Fenzi wrote: > > I'm not sure at all that it would be possible to do at compose-time... > > composes are taking around 3.5-4hours and thats after I have done > > a lot to speed them up, but might be worth some benchmarking > > to see how much slower it would be. If we are going to go that route, we > > should see if support could be added to pungi. > > On server class hardware with fast disk access it's an additional 10 > minutes or so. > > > Moving this from a package to in repodata would grow the repodata quite > > a lot right? > > 6.9Mfedora-39-icons.tar.gz > 6.3Mfedora-39.xml.gz So, on re-reading this thread and the upstream issues... it really seems like the best case would be createrepo_c just adding support, but that seems stalled/blocked due to i/o concerns? (which I can understand, but it could be default off?) How does the local/rpm data interact with the repodata (if it exists)? Or does it depend on the tool? I suppose in the short term at least if you wanted you could hand off the generation/rpm updating to releng? (Not that releng wants more to do, but I don't think you should carry this by yourself). kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Thu, 28 Sept 2023 at 08:03, Richard Hughes wrote: > On Wed, 27 Sept 2023 at 22:41, Kevin Fenzi wrote: > > I'm not sure at all that it would be possible to do at compose-time... > > composes are taking around 3.5-4hours and thats after I have done > > a lot to speed them up, but might be worth some benchmarking > > to see how much slower it would be. If we are going to go that route, we > > should see if support could be added to pungi. > > On server class hardware with fast disk access it's an additional 10 > minutes or so. > > Assume a virtual machine on a possibly busy server, with limited IO to a shared netapp server. I don't expect it will do more than double that estimate but the bigger question will be how often will it 'fail' on the Fedora repositories and what kind of failure. Most failures I remember on any architecture will kill the entire compose which then requires Kevin to mangle whatever broke, and relaunch a compose (which again will take 4 hours) > > Moving this from a package to in repodata would grow the repodata quite > > a lot right? > > 6.9Mfedora-39-icons.tar.gz > 6.3Mfedora-39.xml.gz > > So not small, but cough, passim, cough :) > > Richard. > ___ > -- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Wed, 27 Sept 2023 at 22:41, Kevin Fenzi wrote: > I'm not sure at all that it would be possible to do at compose-time... > composes are taking around 3.5-4hours and thats after I have done > a lot to speed them up, but might be worth some benchmarking > to see how much slower it would be. If we are going to go that route, we > should see if support could be added to pungi. On server class hardware with fast disk access it's an additional 10 minutes or so. > Moving this from a package to in repodata would grow the repodata quite > a lot right? 6.9Mfedora-39-icons.tar.gz 6.3Mfedora-39.xml.gz So not small, but cough, passim, cough :) Richard. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
I must say that I found appstreamcli to be to unstable in CI of projects that I used to maintain, and file format documentation and validator source code reality diverge, and behaviour is understandably changed between appstreamcli revisions. We had to disable and later re-enable the metainfo tests in GNU Radio for various CI runners, because slightly different versions of the same validator couldn't agree amongst each other and with the file format documentation what is legal and what not. There used to be a formal definition of what is legal in that file format (an XSD file), but it got, again, understandably, removed by the appstream maintainer, because they couldn't stem the workload of writing both a C parser/validator for this XML format as well as keep that XSD up to date [1]. So, now: we're left in a situation where we have a human-oriented documentation that we can't trust, a choice of two validation tools that are both buggy in our experience, and although this is an XML format, no format definition. If Fedora depends on that validation, we might **really** want to help maintainers get the formal specification of the format back into shape. That would also make checking the documents tool-independent, and allow for validation in rpmbuild/lint steps without the wealth of dependencies that the specific tools bring. My 2ct, anyway. Cheers, Marcus [1] https://github.com/ximion/appstream/issues/279 On 24.09.23 00:04, Michael Catanzaro wrote: On Sat, Sep 23 2023 at 10:26:48 PM +0200, Alexander Ploumistos wrote: Could someone involved with AppStream please provide some information? Shouldn't our documentation be changed to reflect these changes? Does the FPC need to decide on this? From upstream perspective: appstream-util is indeed obsolete. Upstream software should stop using it and switch to appstreamcli instead. But as you've noticed, it seems Fedora isn't ready for that yet ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
Il 27/09/23 23:49, Neal Gompa ha scritto: > On Wed, Sep 27, 2023 at 5:41 PM Kevin Fenzi wrote: >> On Wed, Sep 27, 2023 at 10:05:38AM -0400, Neal Gompa wrote: >>> On Wed, Sep 27, 2023 at 9:48 AM Richard Hughes wrote: On Wed, 27 Sept 2023 at 13:23, Mattia Verga via devel wrote: > Can't this script be moved to run in Openshift as cron-based? Yes! In fact, that's what I proposed about a decade ago when I wanted to include the data in the metadata like Debian does. I do think it should be managed by someone in the rel-eng/infra team rather than me. >>> openSUSE and Mageia both do it this way too. If we move to running the >>> script in infra, we should just make the switch to appending it to the >>> repodata. When I helped Mageia set it up[1], we elected to compose it weekly >>> and re-append the results for every repo push until the appstream >>> repodata was regenerated. >>> >>> Also, our compose process *has* gotten faster over the past decade, so >>> we may be able to do this at compose-time now. >> Here's the old releng ticket about this (I think): >> https://pagure.io/releng/issue/5721 >> >> In addition to appstream data, there's the screenshots. >> >> I agree much has changed and we should revisit how best to do this >> process. >> >> I'm not sure at all that it would be possible to do at compose-time... >> composes are taking around 3.5-4hours and thats after I have done >> a lot to speed them up, but might be worth some benchmarking >> to see how much slower it would be. If we are going to go that route, we >> should see if support could be added to pungi. >> >> Moving this from a package to in repodata would grow the repodata quite >> a lot right? >> > It's additional repodata files, files that are not even downloaded by > anything except PackageKit (currently). So the main repodata sizes > don't change. > Perhaps we can mimic the bodhi-critpathcron pod and run the script within the bodhi openshift role once a week? I haven't looked in depth into the script, but I think we'll need to mount the archive from somewhere in network to avoid cloning the main mirror, run the script and then synch the output somewhere? I'd suggest to avoid adding the step within pungi or bodhi-composer: Richard said it takes about an hour per release and I don't think we need to have it run at each compose (it would be nice, though, but we're now running it once per month and the world is still here). Also, bodhi-composer already does too much things and when a compose fails a lot of garbage is left behind (broken synch between update states in bodhi and koji builds tags), so better not add another possible failure point. Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Wed, Sep 27, 2023 at 5:41 PM Kevin Fenzi wrote: > > On Wed, Sep 27, 2023 at 10:05:38AM -0400, Neal Gompa wrote: > > On Wed, Sep 27, 2023 at 9:48 AM Richard Hughes wrote: > > > > > > On Wed, 27 Sept 2023 at 13:23, Mattia Verga via devel > > > wrote: > > > > Can't this script be moved to run in Openshift as cron-based? > > > > > > Yes! In fact, that's what I proposed about a decade ago when I wanted > > > to include the data in the metadata like Debian does. I do think it > > > should be managed by someone in the rel-eng/infra team rather than me. > > > > > > > openSUSE and Mageia both do it this way too. If we move to running the > > script in infra, we should just make the switch to appending it to the > > repodata. When I helped Mageia set it up[1], we elected to compose it weekly > > and re-append the results for every repo push until the appstream > > repodata was regenerated. > > > > Also, our compose process *has* gotten faster over the past decade, so > > we may be able to do this at compose-time now. > > Here's the old releng ticket about this (I think): > https://pagure.io/releng/issue/5721 > > In addition to appstream data, there's the screenshots. > > I agree much has changed and we should revisit how best to do this > process. > > I'm not sure at all that it would be possible to do at compose-time... > composes are taking around 3.5-4hours and thats after I have done > a lot to speed them up, but might be worth some benchmarking > to see how much slower it would be. If we are going to go that route, we > should see if support could be added to pungi. > > Moving this from a package to in repodata would grow the repodata quite > a lot right? > It's additional repodata files, files that are not even downloaded by anything except PackageKit (currently). So the main repodata sizes don't change. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Wed, Sep 27, 2023 at 10:05:38AM -0400, Neal Gompa wrote: > On Wed, Sep 27, 2023 at 9:48 AM Richard Hughes wrote: > > > > On Wed, 27 Sept 2023 at 13:23, Mattia Verga via devel > > wrote: > > > Can't this script be moved to run in Openshift as cron-based? > > > > Yes! In fact, that's what I proposed about a decade ago when I wanted > > to include the data in the metadata like Debian does. I do think it > > should be managed by someone in the rel-eng/infra team rather than me. > > > > openSUSE and Mageia both do it this way too. If we move to running the > script in infra, we should just make the switch to appending it to the > repodata. When I helped Mageia set it up[1], we elected to compose it weekly > and re-append the results for every repo push until the appstream > repodata was regenerated. > > Also, our compose process *has* gotten faster over the past decade, so > we may be able to do this at compose-time now. Here's the old releng ticket about this (I think): https://pagure.io/releng/issue/5721 In addition to appstream data, there's the screenshots. I agree much has changed and we should revisit how best to do this process. I'm not sure at all that it would be possible to do at compose-time... composes are taking around 3.5-4hours and thats after I have done a lot to speed them up, but might be worth some benchmarking to see how much slower it would be. If we are going to go that route, we should see if support could be added to pungi. Moving this from a package to in repodata would grow the repodata quite a lot right? kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Wed, Sep 27, 2023 at 9:48 AM Richard Hughes wrote: > > On Wed, 27 Sept 2023 at 13:23, Mattia Verga via devel > wrote: > > Can't this script be moved to run in Openshift as cron-based? > > Yes! In fact, that's what I proposed about a decade ago when I wanted > to include the data in the metadata like Debian does. I do think it > should be managed by someone in the rel-eng/infra team rather than me. > openSUSE and Mageia both do it this way too. If we move to running the script in infra, we should just make the switch to appending it to the repodata. When I helped Mageia set it up[1], we elected to compose it weekly and re-append the results for every repo push until the appstream repodata was regenerated. Also, our compose process *has* gotten faster over the past decade, so we may be able to do this at compose-time now. [1]: https://bugs.mageia.org/show_bug.cgi?id=18669 -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Wed, 27 Sept 2023 at 13:23, Mattia Verga via devel wrote: > Can't this script be moved to run in Openshift as cron-based? Yes! In fact, that's what I proposed about a decade ago when I wanted to include the data in the metadata like Debian does. I do think it should be managed by someone in the rel-eng/infra team rather than me. > like the Package Review Status app, although the storage requirements > are totally different (but, maybe, there's a way to avoid cloning the > mirror "locally" in Openshift and use a network mounted directory?). I'm sure someone could hook up an openshift instance to the SAN we store the archive on -- but that obviously needs to be someone in [internal?] IT. Richard. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
Il 27/09/23 13:24, Richard Hughes ha scritto: > On Tue, 26 Sept 2023 at 03:16, Neal Gompa wrote: >> So a stopgap solution was implemented: appstream-data. Richard Hughes >> maintains a local mirror of the full Fedora repositories and generates >> the appstream data from that using scripts[1] and extra data[2] to >> produce the appstream-data package[3]. > That's a really good writeup, thanks. Note, I'm not the only person > who can generate metadata, and I'd *love* someone to take over the > task if they've got more time than me. We used to build the new > metadata weekly, but it's slipped to monthly, if that frequently. > > The storage requirement of a "full mirror" is ~8G for centos and ~720G > for Fedora -- and although NVMe is faster, I use an external HDD. You > can also generate the metadata on laptop-class hardware -- it takes > about an hour per release or an order of magnitude faster with fast > NMVe and server class hardware. I've not tried to use > appstream-compose yet, but it might work. > > Ideally this is something that could be picked up by release > engineering, but I've fought that battle and lost each time. C'est la > vie. > > Richard. Can't this script be moved to run in Openshift as cron-based? Something like the Package Review Status app, although the storage requirements are totally different (but, maybe, there's a way to avoid cloning the mirror "locally" in Openshift and use a network mounted directory?). Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Tue, 26 Sept 2023 at 03:16, Neal Gompa wrote: > So a stopgap solution was implemented: appstream-data. Richard Hughes > maintains a local mirror of the full Fedora repositories and generates > the appstream data from that using scripts[1] and extra data[2] to > produce the appstream-data package[3]. That's a really good writeup, thanks. Note, I'm not the only person who can generate metadata, and I'd *love* someone to take over the task if they've got more time than me. We used to build the new metadata weekly, but it's slipped to monthly, if that frequently. The storage requirement of a "full mirror" is ~8G for centos and ~720G for Fedora -- and although NVMe is faster, I use an external HDD. You can also generate the metadata on laptop-class hardware -- it takes about an hour per release or an order of magnitude faster with fast NMVe and server class hardware. I've not tried to use appstream-compose yet, but it might work. Ideally this is something that could be picked up by release engineering, but I've fought that battle and lost each time. C'est la vie. Richard. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Mon, Sep 25, 2023 at 1:45 PM Alexander Ploumistos wrote: > > On Mon, Sep 25, 2023 at 7:33 PM Artur Frenszek-Iwicki > wrote: > > > > Without docs, the whole process is a black box. > > +1 > > That was actually one of the two reasons I started this thread. > > For instance, is this the package that must be rebuilt in order to see > an AppStream metadata file appear/refreshed in GNOME Software? > https://src.fedoraproject.org/rpms/appstream-data > Could we get an explainer (for dummies) of how the entire pipeline works? > > The other reason was someone from Debian telling one of my upstreams > "Don't use appstream-util to validate the file, it's deprecated, use > appstreamcli". Frankly, that struck a chord, considering x years ago I > had spent a considerable amount of time convincing dozens of projects > to adopt "AppData" (at the time), filing tickets and submitting PRs. So here's some backstory... Way back in the days of yore (about 10 years ago), people were working on a cross-distro Software Center: https://fedoraproject.org/wiki/Features/SoftwareCenter For that "software center", the AppData (renamed to AppStream) format was created initially to port over desktop files to XML for more reliable parsing at scale. The goal here was to create a cheaper and more reliable method of providing rich metadata in repositories that could be indexed and used to present useful information about applications. My understanding is that the original AppStream implementation at the time was moribund due to its maintainer being in school, so a new one was made (unhelpfully named appstream-glib... why "unhelpfully" you ask? well, the original appstream was *also* glib based!). This got plugged into what became GNOME Software. And because Fedora needed a tool to generate the indexes by scanning all the RPMs, appstream-builder was created to do that. (As an aside, appstream-glib was never used by KDE because there was no Qt/C++ binding for it, but the original AppStream did have one, so that's what Plasma Discover has always used.) But there was a problem: unlike RPM repodata generation, AppStream repodata generation is slow. Really slow. This is because AppStream repodata requires opening up the RPM, plucking data out of it, and applying the necessary transformations per repodata generation policy (downloading remote images, recompression, etc.). There are arguments about whether it's appstream-builder specific (given that Debian's archive is bigger, their package format is more complex to "pluck" from, etc and appstream-generator does fine there), but the point is that for Fedora with appstream-builder, it's too slow. Note that the context of this is back in the era where Fedora composes took half a day or longer. Adding even more hours to that process was not appealing (it would have to be done for every repo being composed for every architecture...). So a stopgap solution was implemented: appstream-data. Richard Hughes maintains a local mirror of the full Fedora repositories and generates the appstream data from that using scripts[1] and extra data[2] to produce the appstream-data package[3]. The goal is to someday get to a point where we can have this done properly as repodata through createrepo_c[4], but we're not able to do that yet. Now, appstream-util is the appstream-glib equivalent of appstreamcli. We test using appstream-util because we need the AppStream data to be valid for appstream-builder to consume it. If it is not, the application will be skipped in the next round of appstream repodata updates. If we ever move to a process that uses libappstream-compose as the backend library for producing AppStream repodata, then we can stop caring about appstream-util. But for now, here we are. [1]: https://github.com/hughsie/appstream-scripts [2]: https://github.com/hughsie/fedora-appstream [3]: https://src.fedoraproject.org/rpms/appstream-data [4]: https://github.com/rpm-software-management/createrepo_c/issues/75 -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Sun, Sep 24, 2023 at 2:46 AM Mattia Verga via devel wrote: > > Il 24/09/23 01:18, Neal Gompa ha scritto: > > On Sat, Sep 23, 2023 at 6:05 PM Michael Catanzaro > > wrote: > >> On Sat, Sep 23 2023 at 10:26:48 PM +0200, Alexander Ploumistos > >> wrote: > >>> Could someone involved > >>> with AppStream please provide some information? Shouldn't our > >>> documentation be changed to reflect these changes? Does the FPC need > >>> to decide on this? > >> From upstream perspective: appstream-util is indeed obsolete. Upstream > >> software should stop using it and switch to appstreamcli instead. > >> > >> But as you've noticed, it seems Fedora isn't ready for that yet > > Ultimately, appstream-util's output is what matters since we use > > appstream-builder (from appstream-glib) to generate our AppStream > > metadata index. If it doesn't pass that tool, it doesn't show up. > > > If upstream switched to appstreamcli, what tool do they use to generate > the index to not fail validation? > Can Fedora switch to that other tool? > There was no switching. Both appstream-util and appstreamcli are considered conformant. Ultimately, the only way we can stop relying on appstream-glib is if appstream-builder[1] was reimplemented on top of libappstream-compose. Long ago, I tried to see if we could have our AppStream data produced as repodata instead of a package, but the answer I got was that we need the functionality added to createrepo to do it[2]. If someone ever got around to making that happen, a lot of things in our pipeline could be simplified. [1]: https://github.com/hughsie/appstream-glib/tree/main/libappstream-builder [2]: https://github.com/rpm-software-management/createrepo_c/issues/75 -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Mon, Sep 25, 2023 at 7:33 PM Artur Frenszek-Iwicki wrote: > > Without docs, the whole process is a black box. +1 That was actually one of the two reasons I started this thread. For instance, is this the package that must be rebuilt in order to see an AppStream metadata file appear/refreshed in GNOME Software? https://src.fedoraproject.org/rpms/appstream-data Could we get an explainer (for dummies) of how the entire pipeline works? The other reason was someone from Debian telling one of my upstreams "Don't use appstream-util to validate the file, it's deprecated, use appstreamcli". Frankly, that struck a chord, considering x years ago I had spent a considerable amount of time convincing dozens of projects to adopt "AppData" (at the time), filing tickets and submitting PRs. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
> Ultimately, appstream-util's output is what matters since we use > appstream-builder (from appstream-glib) to generate our AppStream > metadata index. If it doesn't pass that tool, it doesn't show up. Bit of a tangent, but: is this documented anywhere? Because if if is, then I can't find it. Without docs, the whole process is a black box. I recall that a few years ago an announcement was made saying that apps need to provide at least a 64x64 icon to show up in the index; once again, I can't find any reference to that apart from the mailing list history. This makes figuring out why an app doesn't show up in Gnome Software a frustrating experience. To be clear, I don't want this to be a "someone do something" kind of message. I'd be happy to document this myself! But I have zero idea what's actually going on. A.FI. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
Il 24/09/23 01:18, Neal Gompa ha scritto: > On Sat, Sep 23, 2023 at 6:05 PM Michael Catanzaro > wrote: >> On Sat, Sep 23 2023 at 10:26:48 PM +0200, Alexander Ploumistos >> wrote: >>> Could someone involved >>> with AppStream please provide some information? Shouldn't our >>> documentation be changed to reflect these changes? Does the FPC need >>> to decide on this? >> From upstream perspective: appstream-util is indeed obsolete. Upstream >> software should stop using it and switch to appstreamcli instead. >> >> But as you've noticed, it seems Fedora isn't ready for that yet > Ultimately, appstream-util's output is what matters since we use > appstream-builder (from appstream-glib) to generate our AppStream > metadata index. If it doesn't pass that tool, it doesn't show up. > If upstream switched to appstreamcli, what tool do they use to generate the index to not fail validation? Can Fedora switch to that other tool? Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
24. syyskuuta 2023 2.18.14 GMT+03:00 Neal Gompa kirjoitti: >> On Sat, Sep 23 2023 at 10:26:48 PM +0200, Alexander Ploumistos >> wrote: >> > Could someone involved >> > with AppStream please provide some information? Shouldn't our >> > documentation be changed to reflect these changes? Does the FPC need >> > to decide on this? > >Ultimately, appstream-util's output is what matters since we use >appstream-builder (from appstream-glib) to generate our AppStream >metadata index. If it doesn't pass that tool, it doesn't show up. Since this is not the first time this suggestion has been made, perhaps it would make sense to document the reason in Packaging Guidelines? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Sat, Sep 23, 2023 at 6:05 PM Michael Catanzaro wrote: > > On Sat, Sep 23 2023 at 10:26:48 PM +0200, Alexander Ploumistos > wrote: > > Could someone involved > > with AppStream please provide some information? Shouldn't our > > documentation be changed to reflect these changes? Does the FPC need > > to decide on this? > > From upstream perspective: appstream-util is indeed obsolete. Upstream > software should stop using it and switch to appstreamcli instead. > > But as you've noticed, it seems Fedora isn't ready for that yet Ultimately, appstream-util's output is what matters since we use appstream-builder (from appstream-glib) to generate our AppStream metadata index. If it doesn't pass that tool, it doesn't show up. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
On Sat, Sep 23 2023 at 10:26:48 PM +0200, Alexander Ploumistos wrote: Could someone involved with AppStream please provide some information? Shouldn't our documentation be changed to reflect these changes? Does the FPC need to decide on this? From upstream perspective: appstream-util is indeed obsolete. Upstream software should stop using it and switch to appstreamcli instead. But as you've noticed, it seems Fedora isn't ready for that yet ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Packaging guidelines - validation of AppStream metadata files
The current state of things, Fedora-wise, is summarized in FPC ticket #1053: https://pagure.io/packaging-committee/issue/1053 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Packaging guidelines - validation of AppStream metadata files
Hello, According to our packaging guidelines[1], "you MUST run appstream-util validate-relax (in %check or %install) and have BuildRequires: libappstream-glib, to help ensure the validity and safety of the appdata files you’re installing". For quite some time now, I've been seeing references to another tool, appstreamcli (provided by the appstream[2] package), being used instead of appstream-util. Debian guidelines recommend it[3], it's documented upstream[4] and apparently there are a few fedora packages that use it already[5] (google found 10, I know of a couple more). By the way, if you take a closer look, some of these packages appear to use the "--nonet" flag from appstream-util instead of "--no-net", which is used by appstreamcli. Furthermore, on the web page of AppStream-Glib there's a warning that it is in heavy maintenance mode and to use appstream instead[6]. By all indications, appstream will replace appstream-glib and in the case of the corresponding validator utilities, this has already begun. However, at least on this list, I was unable to find any clue that this was happening and there are zero mentions of appstreamcli on docs.fedoraproject org, hence this message. Could someone involved with AppStream please provide some information? Shouldn't our documentation be changed to reflect these changes? Does the FPC need to decide on this? Best regards, Alexander 1. https://docs.fedoraproject.org/en-US/packaging-guidelines/AppData/ 2. https://github.com/ximion/appstream 3. https://wiki.debian.org/AppStream/Guidelines 4. https://www.freedesktop.org/software/appstream/docs/api/re32.html 5. https://www.google.com/search?q=%22appstreamcli+validate%22+site%3Asrc.fedoraproject.org%2Frpms%2F 6. https://github.com/hughsie/appstream-glib ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
