Re: Proposal: changes to "default application functionality" release criteria
On Fri, May 6, 2022 at 7:29 AM Matthias Clasen wrote: > > On Thu, May 5, 2022 at 1:23 PM Matthew Miller > wrote: > > Wading into this discussion pretty late... there's far too much "GNOME does > this" or "GNOME wants that" in this discussion. > > That is really not how it works, neither for GNOME nor for Fedora - its down > to small teams and individual maintainers to > make changes and decisions. For improving the QA situation between upstream > GNOME and downstream Fedora, a good > team to talk to is the GNOME release team of which both Michael and I happen > to be members. > >> >> > Testing that is much more useful to upstream than testing Rawhide, >> > because we certainly aren't updating all the bits of GNOME in Rawhide >> > nightly. But if it's not built on Fedora (I don't think it is), it may >> > well have differences in theme and font configuration that might make >> > Fedora's openQA needles not match, which is the problem I'm concerned >> > about with sharing the tests. >> >> Could we (GNOME and Fedora in collaboration) set up something where they >> _are_ updating everything on top of Rawhide nightly? If not instead of GNOME >> OS, at least alongside it? >> > > This would be a useful discussion to have, Having it in a bof at guadec > sounds great. > One of the things we're working on in the KDE SIG is eventually auto-building all our KDE software in a COPR on a regular schedule, with the goal of eventually producing something that we could wire up to run through OpenQA tests using that COPR repo for precisely this purpose. I think it would absolutely make sense if the folks maintaining GNOME did the same thing, since they're both release blocking desktops. -- 真実はいつも一つ！/ Always, there's only one truth! ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Thu, May 5, 2022 at 1:23 PM Matthew Miller wrote: Wading into this discussion pretty late... there's far too much "GNOME does this" or "GNOME wants that" in this discussion. That is really not how it works, neither for GNOME nor for Fedora - its down to small teams and individual maintainers to make changes and decisions. For improving the QA situation between upstream GNOME and downstream Fedora, a good team to talk to is the GNOME release team of which both Michael and I happen to be members. > > Testing that is much more useful to upstream than testing Rawhide, > > because we certainly aren't updating all the bits of GNOME in Rawhide > > nightly. But if it's not built on Fedora (I don't think it is), it may > > well have differences in theme and font configuration that might make > > Fedora's openQA needles not match, which is the problem I'm concerned > > about with sharing the tests. > > Could we (GNOME and Fedora in collaboration) set up something where they > _are_ updating everything on top of Rawhide nightly? If not instead of > GNOME > OS, at least alongside it? > > This would be a useful discussion to have, Having it in a bof at guadec sounds great. ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Wed, May 04, 2022 at 05:47:33PM -0700, Adam Williamson wrote: > > So, um, maybe this is another place we could talk with GNOME upstream. > > Could we have those upstream tests run on Rawhide? > > > > Also: where does GNOME _expect_ the human-based QA to happen? Especially > > for the applications, it's going to be hard to automate thoroughly. > > Maybe we can figure out something better there too. > > So the answer to both those questions is, AIUI, kinda the same: GNOME > wants the testing run on GNOME OS. > > https://os.gnome.org/ > > which is (supposed to be, I don't know how closely it meets the goal > ATM) a nightly live GNOME image with the latest versions of everything > in it. > > Testing that is much more useful to upstream than testing Rawhide, > because we certainly aren't updating all the bits of GNOME in Rawhide > nightly. But if it's not built on Fedora (I don't think it is), it may > well have differences in theme and font configuration that might make > Fedora's openQA needles not match, which is the problem I'm concerned > about with sharing the tests. Could we (GNOME and Fedora in collaboration) set up something where they _are_ updating everything on top of Rawhide nightly? If not instead of GNOME OS, at least alongside it? That seems like it would: 1) Give a better practical test environment for GNOME folks 2) Make the needles match 3) Maybe make it so developers can fix their own needle-breaking problems? 4) Help everything get tested earlier -- Matthew Miller Fedora Project Leader ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Wed, 2022-05-04 at 19:04 -0400, Matthew Miller wrote: > On Tue, May 03, 2022 at 09:20:35PM -0700, Adam Williamson wrote: > > We could look at what it would take to upstream the app tests we have > > already for Fedora. What I'm afraid of is that we'd wind up finding > > some subtle difference in theme or font rendering meant we'd have to > > duplicate all the needles, and re-do them all twice every time freetype > > or cantarell sneezes... > > So, um, maybe this is another place we could talk with GNOME upstream. Could > we have those upstream tests run on Rawhide? > > Also: where does GNOME _expect_ the human-based QA to happen? Especially for > the applications, it's going to be hard to automate thoroughly. Maybe we can > figure out something better there too. So the answer to both those questions is, AIUI, kinda the same: GNOME wants the testing run on GNOME OS. https://os.gnome.org/ which is (supposed to be, I don't know how closely it meets the goal ATM) a nightly live GNOME image with the latest versions of everything in it. Testing that is much more useful to upstream than testing Rawhide, because we certainly aren't updating all the bits of GNOME in Rawhide nightly. But if it's not built on Fedora (I don't think it is), it may well have differences in theme and font configuration that might make Fedora's openQA needles not match, which is the problem I'm concerned about with sharing the tests. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Tue, May 03, 2022 at 09:20:35PM -0700, Adam Williamson wrote: > We could look at what it would take to upstream the app tests we have > already for Fedora. What I'm afraid of is that we'd wind up finding > some subtle difference in theme or font rendering meant we'd have to > duplicate all the needles, and re-do them all twice every time freetype > or cantarell sneezes... So, um, maybe this is another place we could talk with GNOME upstream. Could we have those upstream tests run on Rawhide? Also: where does GNOME _expect_ the human-based QA to happen? Especially for the applications, it's going to be hard to automate thoroughly. Maybe we can figure out something better there too. -- Matthew Miller Fedora Project Leader ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Tue, May 03, 2022 at 05:14:53PM -0700, Adam Williamson wrote: > but these bugs weren't discovered at that time. This is likely because > of your idea 2: "basic functionality" is a bit up for debate. You can > take an extremely minimalist approach to this (run the app, click a > couple of buttons, say it's OK if nothing explodes and no babies are > eaten) or a slightly more maximalist one (run the app, and actually try > and do something useful with it). In this case, before Final, when we > ran this test we mostly did the minimalist thing. At Final RC stage, we > went a bit more maximalist. Yeah, the tension: we're moving towards more polish, but the available time to fix gets smaller. And it's not like fixing the things that are at that "more maximalist" end takes less time. > >As a barometer for "embarrasing", you can imagine me trying to explain > >the issue to a tech reporter, and weigh how awkward I will feel saying > >"this is fine" vs. how I will feel explaining that we delayed the whole > >release for that same issue. > > I agree with the sentiment but I'm not sure about the phrasing. It's > extremely subjective, and I don't think subjective criteria work very > well. It also wouldn't necessarily "solve the problem": the bugs we Yes, sorry. That was meant to be a sentiment-based way to hopefully convey what I'm trying to say, not a suggested phrasing. > discovered this time really are pretty embarrassing, honestly. Imagine > doing a keynote showing off the sleek default apps included in GNOME, > running them, and trying to do...well...actually anything at all useful > with them. It wouldn't go very well. This is why I don't do live demos. :) > > 3. Problems found which are not regressions should not be blockers. We're > >just hurting ourselves when we make this our forcing function to get > >something fixed. > This is one of those things that sounds great until it doesn't. I can't > quite recall any specifics, but I definitely think there have been > cases recently where we've had strong support for a bug that is not a > regression to be a blocker. Sometimes a bug is just really bad but we > didn't see it before; even if you can make a wonk-y argument that > there's no point making it a blocker because if we do, it just means > the previous release stays as the "current" one for longer and it has > the same bug, in practice it's hard to hold that line when now there's > a bug report that everyone can see that says how badly this thing is > broken. I feel like I can defend that line pretty confortably. Send 'em to me. :) In seriousness, I think we look to blocking the relase because it's such a big signal. The train comes to a stop and everyone can see. But it's a big signal _because it's painful_, and pain isn't the best motivator. And while that pain does sometimes land on people who can do something, even if that _would_ help, it's indiscriminate. In the metaphor, people can't get to where they are going, the tracks are blocked, etc., etc., and none of that has anything to do with the issue to fix. (I'm tempted to keep running with the train metaphor, but reluctantly will stop.) This is the idea of Prioritized Bugs, and I think it's relatively successful. It's another way to make sure we're focusing on serious problems. Making that more visible could help. > >I don't know how to phrase this in a way that doesn't make Adam sad with > >me, but maybe something like: Desktop application blockers discovered > >during the final freeze are automatically waived unless the relevant Spin > >or Edition team decides otherwise. > You're right that this makes me sad. I don't think it's a good > approach. I think it's an attempt to solve a problem that I would maybe > look at differently, and which we're currently discussing in a ticket: > https://pagure.io/fedora-workstation/issue/304 > > For me, the big question your mail never quite arrived at is, *why* did > these bugs show up in Fedora 36 Final RCs at all? They really should > not have done. They are bugs in applications that are, supposedly, core > parts of upstream GNOME. They appear in 42.0 releases of those > applications - i.e. in releases of those applications that are, > according to upstream's versioning scheme, stable releases for public > consumption. > > Stable releases of core components of a major desktop should never > contain bugs like "deleting contacts sometimes doesn't work" or "you > can't add photos to an album in the Photos application because the > dialog where you're supposed to do it is completely broken and the list > entries multiply like rabbits who've been dosed up on viagra". > Distribution validation testing is not *for* finding bugs like this. > > The reason we're all instinctively feeling that something is Not Right > here is that something *is* Not Right, but the big thing that's Not > Right is the upstream GNOME release process. It's not (IMHO) any part >
Re: Proposal: changes to "default application functionality" release criteria
On Wed, May 04, 2022 at 02:46:16PM +0200, Kamil Paral wrote: > > popularity and reviews noting how polished everything is makes us very much > > want to build on that. So I understand why this is here, including the > > expanded "all installed applications" Workstation criteria. > As Adam already noted, it is actually cut down. We used to have higher > standards in this area (and we lowered them because we couldn't keep them, > especially when KDE ships a bazillion of preinstalled apps). Okay, fair. I didn't mean "expanded" in a temporal sense, just the "expanded universe" one. But that wasn't clear. > We could have a GNOME Apps Test Day, or perhaps a GNOME Low Profile Apps > Test Day. Separating that from GNOME DE basics + settings etc would perhaps > motivate people to focus more on those apps and spend more time with them. I think that would help. More on this coming in my (will write after this one!) reply to Adam's message. > > 2. "Basic functionality" seems scoped too broadly currently. I propose, > >for the release criteria, we change this to: A) "the app doesn't > >crash on launch", and B) "the app's behavior does not seem > >immediately embarrassing with a few minutes of playing around with > >it". > If you only want to block on app launch and close, let's be honest about it > and call it that. The basic functionality requirement can stay for those > high-profile apps listed explicitly in the criterion, and the other apps > would only be required to launch and close without crashing. I would be fine with calling it something else. > However, that's a *massive* step down in quality. Do we really want that? Of course not. But > Your B) is too vague for me. If we ship with our current > photos/contacts/etc bugs, I'll feel embarrassed. I guess another way to put it is: blocking the release causes huge problems. It means people don't get updates they were looking for. It messes up the schedules of people counting on is. It derails publicity momentum. It means other parts of the project can't get the stuff _they_ have worked so hard on out to users. And more. Of course, shipping something that's broken *also* has negative effects (I don't think I need to argue for that, but can if people want!). Because there are going to be far more things that don't work perfectly than we possibly have resources to fix, it's always a balance. So it's always a matter of "which is worse?". Right now, we mostly push that out and only (officially) consider it at basically the last minute. I think that for some things, it'd be better to consider that factor earlier. Does that make sense? I'm not sure of the best way to do that in practice. And, I know we've _always_ worked hard on that balance with the "hybrid" approach to the schedule — I don't mean to discount that. [things snipped but not because I didn't read them... just trying to not repeat too much...] > > 3. Problems found which are not regressions should not be blockers. We're > >just hurting ourselves when we make this our forcing function to get > >something fixed. > > Such a broad rule is a really bad idea. You're probably thinking "this bug > was already there, and very few people complained, so why block our next > release on it?". Yes, if we include a safeguard "and very few people > complained", the proposal starts to sound more reasonable. But imagine that > there was a massive disaster in our last release - e.g. a Nautilus > data-corruption bug slipped through our fingers, or Anaconda ate hard > drives in certain cases, stuff like that. We only found out after the > release, when we received a flood of angry bug reports of people leaving > Fedora for good, and by your definition... we are **prevented** from > blocking our next release on that. That's surely not a good idea? I'm not convinced! If it is at the high end of catastrophic, to reset ourselves we might do something entirely outside of the normal process (like when we did the whole-year cycle for F20). And if it's not that bad — but still important — how does blocking the release for an issue that's already out there _help_? That data-eating Anaconda would still be on the Get Fedora page. Rogue Nautilus still out there. Obviously we should fix it, but why is _blocking_ the tool? What benefit does it have that other approaches to prioritizing a fix wouldn't? > >trackers in pagure or whereever _or_ having it as more targets in the > >blockerbugs app. I tend towards the latter: I think it might help > >with the problem where blockers feel like the only obvious way to > >bugs tracked and fixed. But either way, there should be lists! > > Well, if there are lists which those teams actually follow and maintain, > that would of course be a good thing. Bugzilla tracker would be the easiest > implementation. I'm a bit worried that those teams will get flooded with an > avalanche of reports, but that's not really my problem to solve. I'm m
Re: Proposal: changes to "default application functionality" release criteria
On Wed, 2022-05-04 at 10:27 +0100, Allan Day wrote: > Adam Williamson wrote: > ... > > Do GNOME users actually use Contacts and Calendar and Photos? I mean, > > I'm kind of a GNOME ultra, and I don't. > ... > > One of the reasons we need metrics is so we can answer these questions > using actual data, and I'll be much more comfortable making these > kinds of decisions once we have that. > > That said, my own personal sense is that Calendar is used to some > extent, and Contacts and Photos are used much less. That is somewhat > reflected in the app reviews: > > Firefox - 7236 reviews, 3.4 average rating > gedit - 896 reviews, 4.0 average rating > Calendar - 321 reviews, 3.0 average rating > Photos - 128 reviews, 2.3 average rating > Contacts - 84 reviews, 3.2 average rating > > The challenge we face is that, if we only included really high quality > apps with high levels of usage, we'd be left with very few apps > remaining! You could make an argument for removing Photos, Contacts, > Calendar, Cheese, Maps, Weather, Rhythmbox... and if we went down that > road, aside from leaving the desktop looking fairly empty, we'd also > potentially fall short in terms of user expectations. While people > might not always use these apps, there is something of an expectation > that they will be there, and indeed competing desktops do still > include equivalents. > > I'm also conscious of users who prefer not to use web services from > Google and Microsoft, as well as users who don't have continuous > access to high bandwidth internet connections. I get the perspective, but I'm not sure it's right. For a start, there's no email app, so - if we ignore the continued existence of Evolution, as GNOME seems determined to do, that means pretty much everyone is going to either use a completely non-GNOMEified app (Thunderbird), or a web service, to read their email. Both Thunderbird and every common webmail service I can think of can also handle contacts and calendar, so, why would someone take the trouble to use an app for those instead? I think it's fine to only include high-quality applications that are actually useful for something. I just don't see the value in including under-featured, buggy apps for purposes where there are much better choices out there, purely to make the desktop feel less 'empty'. That feels like the kind of thinking that causes smartphone vendors to fill their default home pages with junky home-made apps that nobody *ever* uses. I'm not super familiar with KDE's equivalents, but at least from my quick testing of them, they are way more featured than these apps (of course, it's *KDE*) and they have a fairly long history of development around stable core libraries so tend to work pretty well. > > That all said, my personal view is that we should drop Photos. > Contacts and Calendar I'm less sure about, because there is a more > active upstream in both cases. > > I'd also like to see us slim down and/or refresh our preinstalled > apps, in order to reduce the maintenance and testing burden. We have > designs in place for this in a number of cases - the image viewer, > photos, videos, the camera - I'd be really happy to see progress > there. I kinda like this idea in theory, but in practice so far what it seems to have resulted in is the replacement of apps which were maybe a little wonky and under-maintained, but which fundamentally did useful stuff - like Shotwell and Evolution - with apps that maybe have more guideline-compliant UIs and smaller codebases, but barely do anything useful and seem to be consistently buggy - like Photos and Contacts/Calendar. I'm not sure this has been a win overall. If this had been a short transition period during which the functionality and reliability of the new apps had been rapidly improved, that would be one thing (this is what I was hoping for at first). But so far it doesn't feel like that. gedit/gnome-text-editor is a closer call, to be fair. I still use gedit because I want a text editor that's good for coding but isn't a full- blown opinionated IDE, but I can see the case for g-t-e, and it doesn't seem as underfeatured and buggy as the others. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Wed, May 4, 2022 at 11:35 AM Allan Day wrote: > That said, my own personal sense is that Calendar is used to some > extent, and Contacts and Photos are used much less. That is somewhat > reflected in the app reviews: > > Firefox - 7236 reviews, 3.4 average rating > gedit - 896 reviews, 4.0 average rating > Calendar - 321 reviews, 3.0 average rating > Photos - 128 reviews, 2.3 average rating > Contacts - 84 reviews, 3.2 average rating > That's a great idea to look for app reviews, thanks for providing the numbers. ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Wed, May 4, 2022 at 2:15 AM Adam Williamson wrote: > > 1. I know we have had GNOME Test days, but this > >stuff didn't come up. Presumably, it would have if someone had > happened > >to look at GNOME Photos. Can we formally go through > >https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic and > > > https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic_others > >much, much earlier (probably when the GNOME pre-release is available), > >either as part of a test day or some other formal thing? Or is there > >something else going on here that I'm not looking closely enough to > see? > > Well, kinda, yes. The thing going on is that we *did* go through that > test at several earlier points: > > > https://openqa.fedoraproject.org/testcase_stats/36/Desktop/QA_Testcase_desktop_app_basic_others_Release_blocking_desktops___lt_b_gt_x86___x86_64_lt__b_gt_.html > > but these bugs weren't discovered at that time. This is likely because > of your idea 2: "basic functionality" is a bit up for debate. You can > take an extremely minimalist approach to this (run the app, click a > couple of buttons, say it's OK if nothing explodes and no babies are > eaten) or a slightly more maximalist one (run the app, and actually try > and do something useful with it). In this case, before Final, when we > ran this test we mostly did the minimalist thing. At Final RC stage, we > went a bit more maximalist. > I wouldn't call that maximalist, that would mean testing everything possible. A *realistic* approach is more appropriate, I think. To "do something useful with it" is a great description. What good is it if it can start and survive a few button clicks, if you can't do useful tasks? The tasks it was actually designed for. What's the point of shipping a photo organizer where you can't create and organize albums? This is the realistic scenario, and ideally the thing we would always try to test. Of course, not always people have time to do that, and only some pathways can be broken while not others. Discovering bugs in the realistic scenarios requires time and also some luck. At the same time, bugs in realistic scenarios will very likely prevent actual users from actually using that app, even if just certain pathways are broken. Even if some of those bugs seem trivial to discover, they are not. If you edit any other field than the email address in gnome-contacts, you'll not see contacts duplication. So as a QA, you only have a certain chance to spot it. In the real world with regular usage, however, sooner or later you'll definitely edit someone's email address and then you'll see that contact doubled. (This is just a nice example, this particular bug was not accepted as a blocker, as it is not ground-breaking). ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Wed, May 4, 2022 at 12:13 AM Matthew Miller wrote: > > Time to talk about > > https://www.fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#Default_application_functionality > again! > > Lots of desktop-app-related blockers this time around, and last time too. I > think we're hitting a symptom-of-our-success problem here: increasing > popularity and reviews noting how polished everything is makes us very much > want to build on that. So I understand why this is here, including the > expanded "all installed applications" Workstation criteria. > As Adam already noted, it is actually cut down. We used to have higher standards in this area (and we lowered them because we couldn't keep them, especially when KDE ships a bazillion of preinstalled apps). > So, ideas for discussion: > > > 1. I know we have had GNOME Test days, but this >stuff didn't come up. Presumably, it would have if someone had happened >to look at GNOME Photos. Can we formally go through >https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic and >https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic_others >much, much earlier (probably when the GNOME pre-release is available), >either as part of a test day or some other formal thing? Or is there >something else going on here that I'm not looking closely enough to see? > We could have a GNOME Apps Test Day, or perhaps a GNOME Low Profile Apps Test Day. Separating that from GNOME DE basics + settings etc would perhaps motivate people to focus more on those apps and spend more time with them. > 2. "Basic functionality" seems scoped too broadly currently. I propose, for >the release criteria, we change this to: A) "the app doesn't crash on >launch", and B) "the app's behavior does not seem immediately >embarrassing with a few minutes of playing around with it". > If you only want to block on app launch and close, let's be honest about it and call it that. The basic functionality requirement can stay for those high-profile apps listed explicitly in the criterion, and the other apps would only be required to launch and close without crashing. However, that's a *massive* step down in quality. Do we really want that? Your B) is too vague for me. If we ship with our current photos/contacts/etc bugs, I'll feel embarrassed. If needed, we can try to define "basic functionality" clearer. For example: a) We can say that the tested feature must be in line with the primary goal of the application. For Nautilus, that would probably be managing local files, but not connecting to remote filesystems, or a functional bookmarking system. For Photos, that would be local albums organization (and perhaps viewing remote ones), but not exporting photo thumbnails. For Cheese, that would be recording from your camera, but not applying effects. b) We can say that functionality which is only available through app menus (as opposed to user facing buttons) is not basic. c) We can say that bugs which only occur if you modify default app settings do not qualify. We can make many clarifications like these, and perhaps it would help us sometimes to decide our arguments. At the same time it can also burn us. And we'd have to decide whether we want to apply the same standards for both high-profile apps listed in the criterion and also low-profile (all the rest) apps, or if we want to have different standards. But, if we keep at least some reduced "basic functionality" requirement for those low-profile apps, I don't think that would help the current situation. A photo organizer which can't organize photos doesn't meet that criterion, whichever way you look at it. A contacts app which duplicates contacts on edit, crashes when you add a new contact quickly, and fails to delete contacts more often than not... doesn't block the release already (however weird that sounds), so again no change. So as I see it, we can update the basic functionality description, but as long as it affects low-profile apps, these situations will keep happening. Or we remove it completely (at least for low-profile apps), but have a massive fall in quality. 3. Problems found which are not regressions should not be blockers. We're >just hurting ourselves when we make this our forcing function to get >something fixed. > Such a broad rule is a really bad idea. You're probably thinking "this bug was already there, and very few people complained, so why block our next release on it?". Yes, if we include a safeguard "and very few people complained", the proposal starts to sound more reasonable. But imagine that there was a massive disaster in our last release - e.g. a Nautilus data-corruption bug slipped through our fingers, or Anaconda ate hard drives in certain cases, stuff like that. We only found out after the release, when we received a flood of angry bug reports of people leaving Fedora for good, and by your definition... we are **prevented** from blocking our next r
Re: Proposal: changes to "default application functionality" release criteria
On Tue, 2022-05-03 at 19:37 -0500, Michael Catanzaro wrote: > On Tue, May 3 2022 at 05:14:53 PM -0700, Adam Williamson > wrote: > > Stable releases of core components of a major desktop should never > > contain bugs like "deleting contacts sometimes doesn't work" or "you > > can't add photos to an album in the Photos application because the > > dialog where you're supposed to do it is completely broken and the > > list > > entries multiply like rabbits who've been dosed up on viagra". > > Distribution validation testing is not *for* finding bugs like this. > > I kinda agree, but reality is GNOME has no QA, and Fedora has good QA, > so Fedora is gonna find the bugs. > > (IMO GNOME quality has actually improved considerably over the past > decade. But surely not enough so!) GNOME, yes. The thing this makes me wonder is: does it *really* make sense for GNOME to be shipping all these desktop apps, if it doesn't have the resources to properly maintain or test them? Do GNOME users actually use Contacts and Calendar and Photos? I mean, I'm kind of a GNOME ultra, and I don't. I still use Evolution because it still works and does everything I want and I'm used to it. I still use Shotwell because all my metadata is in it and there doesn't seem to be any way to export it to Photos, so why would I switch? We (Fedora) are more likely to happen across bugs in the core of GNOME than in apps nobody really uses, if that's the kind of QA upstream GNOME is relying on now. > > GNOME does have OpenQA running now, but there are almost no tests. > Maintaining OpenQA tests requires effort that nobody has been > interested in. I don't want this to sound like an invitation "adamw > come prepare and maintain loads of OpenQA tests for us! do even more > work! don't you want a second unpaid job?" but if anybody happens to > think this would be an interesting way to contribute, pushing existing > tests further upstream would not be a bad idea. We could look at what it would take to upstream the app tests we have already for Fedora. What I'm afraid of is that we'd wind up finding some subtle difference in theme or font rendering meant we'd have to duplicate all the needles, and re-do them all twice every time freetype or cantarell sneezes... -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Proposal: changes to "default application functionality" release criteria
On Tue, 2022-05-03 at 18:13 -0400, Matthew Miller wrote: > Time to talk about > https://www.fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#Default_application_functionality > again! > > Lots of desktop-app-related blockers this time around, and last time too. I > think we're hitting a symptom-of-our-success problem here: increasing > popularity and reviews noting how polished everything is makes us very much > want to build on that. So I understand why this is here, including the > expanded "all installed applications" Workstation criteria. To be clear, it's not exactly that the Workstation x86_64 requirement is expanded, but that the other requirements are reduced. Up until a couple of years ago, the requirement was "all installed applications" for all release-blocking desktops, on all release-blocking arches. We narrowed it down to being "all installed applications" for Workstation on x86_64, and just the specified list of apps for other cases (KDE on any arch, Workstation on aarch64). > But I think we might be using the wrong tool for some of this polish, and I > think we also need to give ourselves some escape hatches. > > By way of concrete example, the Photos application is meant to be a photo > organizer, so "album picker duplicates fields, preventing photo > organization" https://bugzilla.redhat.com/show_bug.cgi?id=2081291 is easy to > classify as failing the basic functionality test currently. But, it's really > painful for that to be blocking the release. > > > So, ideas for discussion: > > > 1. I know we have had GNOME Test days, but this >stuff didn't come up. Presumably, it would have if someone had happened >to look at GNOME Photos. Can we formally go through >https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic and >https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic_others >much, much earlier (probably when the GNOME pre-release is available), >either as part of a test day or some other formal thing? Or is there >something else going on here that I'm not looking closely enough to see? Well, kinda, yes. The thing going on is that we *did* go through that test at several earlier points: https://openqa.fedoraproject.org/testcase_stats/36/Desktop/QA_Testcase_desktop_app_basic_others_Release_blocking_desktops___lt_b_gt_x86___x86_64_lt__b_gt_.html but these bugs weren't discovered at that time. This is likely because of your idea 2: "basic functionality" is a bit up for debate. You can take an extremely minimalist approach to this (run the app, click a couple of buttons, say it's OK if nothing explodes and no babies are eaten) or a slightly more maximalist one (run the app, and actually try and do something useful with it). In this case, before Final, when we ran this test we mostly did the minimalist thing. At Final RC stage, we went a bit more maximalist. > >(Same for KDE and any other theoretical release-blocking desktops!) > > > 2. "Basic functionality" seems scoped too broadly currently. I propose, for >the release criteria, we change this to: A) "the app doesn't crash on >launch", and B) "the app's behavior does not seem immediately >embarrassing with a few minutes of playing around with it". > >As a barometer for "embarrasing", you can imagine me trying to explain >the issue to a tech reporter, and weigh how awkward I will feel saying >"this is fine" vs. how I will feel explaining that we delayed the whole >release for that same issue. I agree with the sentiment but I'm not sure about the phrasing. It's extremely subjective, and I don't think subjective criteria work very well. It also wouldn't necessarily "solve the problem": the bugs we discovered this time really are pretty embarrassing, honestly. Imagine doing a keynote showing off the sleek default apps included in GNOME, running them, and trying to do...well...actually anything at all useful with them. It wouldn't go very well. > > 3. Problems found which are not regressions should not be blockers. We're >just hurting ourselves when we make this our forcing function to get >something fixed. This is one of those things that sounds great until it doesn't. I can't quite recall any specifics, but I definitely think there have been cases recently where we've had strong support for a bug that is not a regression to be a blocker. Sometimes a bug is just really bad but we didn't see it before; even if you can make a wonk-y argument that there's no point making it a blocker because if we do, it just means the previous release stays as the "current" one for longer and it has the same bug, in practice it's hard to hold that line when now there's a bug report that everyone can see that says how badly this thing is broken. It also, again, wouldn't have solved this problem, because most of these bugs *are* regressions, IIRC. The Photos bugs weren't in F35, it worked better there. >I propose that the teams responsible
Proposal: changes to "default application functionality" release criteria
Time to talk about https://www.fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#Default_application_functionality again! Lots of desktop-app-related blockers this time around, and last time too. I think we're hitting a symptom-of-our-success problem here: increasing popularity and reviews noting how polished everything is makes us very much want to build on that. So I understand why this is here, including the expanded "all installed applications" Workstation criteria. But I think we might be using the wrong tool for some of this polish, and I think we also need to give ourselves some escape hatches. By way of concrete example, the Photos application is meant to be a photo organizer, so "album picker duplicates fields, preventing photo organization" https://bugzilla.redhat.com/show_bug.cgi?id=2081291 is easy to classify as failing the basic functionality test currently. But, it's really painful for that to be blocking the release. So, ideas for discussion: 1. I know we have had GNOME Test days, but this stuff didn't come up. Presumably, it would have if someone had happened to look at GNOME Photos. Can we formally go through https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic and https://www.fedoraproject.org/wiki/QA:Testcase_desktop_app_basic_others much, much earlier (probably when the GNOME pre-release is available), either as part of a test day or some other formal thing? Or is there something else going on here that I'm not looking closely enough to see? (Same for KDE and any other theoretical release-blocking desktops!) 2. "Basic functionality" seems scoped too broadly currently. I propose, for the release criteria, we change this to: A) "the app doesn't crash on launch", and B) "the app's behavior does not seem immediately embarrassing with a few minutes of playing around with it". As a barometer for "embarrasing", you can imagine me trying to explain the issue to a tech reporter, and weigh how awkward I will feel saying "this is fine" vs. how I will feel explaining that we delayed the whole release for that same issue. 3. Problems found which are not regressions should not be blockers. We're just hurting ourselves when we make this our forcing function to get something fixed. Some of these could be Prioritized Bugs, but I don't want to overload that too much. I propose that the teams responsible for blocking desktop deliverables keep their own prioritized lists of this kind of problem that the team agrees should be fixed for a good user experience. Not just add to the general queues of bugs or tickets, but specific lists of "application experience issues". These lists, of course, could include problems which also don't qualify for point #2 but which seem important. Like the Photos app issues. (This could also extend to "desktop experience issues" rather than just "application". Or for that matter there could be a similar mechanism for non-desktop blocking deliverables.) I could be convinced either way on having these in the teams' issue trackers in pagure or whereever _or_ having it as more targets in the blockerbugs app. I tend towards the latter: I think it might help with the problem where blockers feel like the only obvious way to bugs tracked and fixed. But either way, there should be lists! 4. Desktop application problems discovered during at the last minute should not be blockers. If the problem is really going to impact a lot of people, it should have been discovered in the beta. (Exception for _new_ regressions, of course.) By the time we're in final freeze, this ends up being hero work for everyone. I don't know how to phrase this in a way that doesn't make Adam sad with me, but maybe something like: Desktop application blockers discovered during the final freeze are automatically waived unless the relevant Spin or Edition team decides otherwise. That lets us still block if something is really bad and just happened to slip through. 5. Okay, and... bigger: we should aim for more approaches which let us decouple as much as possible from the Release. (My grand hope is that we can release every deliverable on its own schedule, but I also understand the _highly aspirational_ nature of that idea. But...) What if we could just easily ship GNOME Photos from GNOME 41 until a fix is found in the updated one? -- Matthew Miller Fedora Project Leader ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to sp
