[flashrom] Re: master state / potential release
Hi all (again), We (Angel) just took the time to reorder the previous replies so that we can respond to them in-line. On Mon, Oct 17, 2022 at 11:34 AM Richard Hughes wrote: > > On Mon, 17 Oct 2022 at 12:24, Edward O'Callaghan wrote: > > > > On Mon, 17 Oct 2022 at 20:27, Richard Hughes wrote: > > > > > > Hi all, > > > > > > Perhaps a way forward would be to tag a rc release, ask distributors > > > to package and test it (e.g. pushing to Fedora Rawhide, but not Fedora > > > 36), and then push the actual official release a week or two later? > > > > > > Certainly aiming to do releases monthly is much healthier than doing > > > releases every few years. I think it's much more achievable to set the > > > expectation to the end user "sorry for the regression! it'll be fixed > > > in your distro in ~3 weeks, in the meantime use the previous release" > > > than trying to squash every bug and add all the features before > > > tagging a mythical beautiful bug-free "feature complete" release. We (Angel) agree that striving for a "feature complete" release is not good: in a way, it resembles maladaptive daydreaming. Having a faster-paced release cadence is a good idea, too. However, flashrom isn't the kind of project where regressions can be taken lightly, especially those which can render users' hardware unusable. It's important to not forget that flashrom's userbase is extremely wide, with users that just want to update their BIOS without having to install Windows, experienced coreboot developers, and anything in between and even out-of-bounds. Some people even use flashrom on DOS: if there's a bug that corrupts the flash chip contents, it may not be possible to get a different flashrom version without rebooting the system (in other words, they are doomed). > > > Richard. > > > > I very much like Richard's pragmatic approach here. > > > > It would be my view that we should re-evaluate branch critical bugs > > (sb600spi map issue + build system stuff are the two the most prominent in > > my mind at the moment) and just cut a release branch, stabilise that with > > some cherry-picks of any residue items. Forge forwards from that with a > > more regular cadence exactly how Richard suggests. If some critical issue > > comes up we can do a point release. Sounds good, thank you. Not sure which are the most pressing concerns, but definitely not everything currently listed in https://ticket.coreboot.org/issues/353 is required to make a release. Another option would be to make a new branch for the next (probably v1.3, but it could be something else like v1.2.1 too?) release: one could base the next release off the master branch with the potentially broken parts removed, or one could base it off the v1.2 branch and cherry-pick the most important changes from master. At first, we (Angel) were unsure which approach would be better, but the second approach sounds more appealing the more we think about it: it would allow us (the flashrom community) to calmly re-review (and possibly re-design) all the changes since the v1.2 release. It will be a *lot* of work, but it will allow us (the flashrom community) to address any concerns about review thoroughness (for lack of a better word) and settle them down once and for all. > > Kindest regards, > > Edward. > > If it helps, a monthly release seems to be the sweet spot for fwupd, > from a "writing release notes" and "getting fixes into users hands" > point of view. There's no point having a "no regressions" rule as this > is software, and even the most benign of changes can have some unknown > side effect -- it's much better to just make releases "cheap" and do > lots of them. In fwupd the main branch is always "releasable" so if we > have a high-priority fix or security issue we just merge, write > release notes, tag, push outside the usual release cadence. This is what coreboot strives for, too: coreboot's master branch is supposed to work at all times, and releases are just "glorified" commits from the master branch. The release cadence for coreboot is about 3 months (it used to be 6 months), which should also work for flashrom. However, flashrom's master branch isn't "releasable" at the moment even though it should be. > Richard Best regards, Angel ___ flashrom mailing list -- flashrom@flashrom.org To unsubscribe send an email to flashrom-le...@flashrom.org
[flashrom] Re: master state / potential release
Hi list, We (Angel) sense that this thread might potentially drift towards an unpleasant direction and we think the outcome wouldn't be forward-useful, i.e. it would make things worse. How are all of you doing? Is anyone feeling hurt? On Sat, Oct 15, 2022 at 11:53 AM Nico Huber wrote: > > Hi all, > > On 14.10.22 03:53, Anastasia Klimchuk wrote: > > A quick answer first: > > We are closer to the release, and the state of the master is better > > than a half year ago. > > hmmm, I may not have made myself clear with my question. I didn't > mean to ask for an overall state but wrt. stability (iow. absence > of code flaws) in particular. And if/how we can trust that there > are no or few enough flaws. In order to make a release, stability (iow. absence of code flaws) is more important than having more features. So, we (Angel) would suggest prioritizing bug fixes over new features, at least to get the release done. Another option would be to branch off current master, radically deal with existing bugs (e.g. delete the problematic sb600spi code, revert the progress state patches) and use this branch for the flashrom v1.3 release. Then we can decide how to proceed for future releases without the pressure of not having done a release in a long time. > > More details. > > In general, the progress is visible here > > https://ticket.coreboot.org/issues/353 > > Yes, this is the progress of tickets. But how to put this... I know > the state as far as we measure it. What I'm concerned about is what > we don't measure. Do you have any examples of things that we do not measure? Hmmm, looks like they're explained below. > Maybe this helps to understand the idea of the release process better: > In the Development Guidelines[1] it says > > "Branching for a new release can happen at any point in time when >a commit (branch point) on master seems to be in good shape and >was reasonably tested after previous invasive changes." > > The last part tries to protect us from very new regressions / issues > that nobody found (measured) yet. And this was written in a time when > flashrom builds from the master branch were much more tested. We could > tell people that it's safe to use the master branch (which it isn't > anymore). And even the people who used releases implicitly tested a > state of the code for us that was much closer to master. In other words, it's best to submit new features and potentially breaking changes after a release. Otherwise, things in a release don't get tested properly. On the coreboot side, we had to make the 4.8.1 release because 4.8 was broken: a small patch got submitted just before the 4.8 release that broke loading several payloads, and people who tried the 4.8 release quickly noticed that it did not boot... > Also, what led to the long list in #353 was a development process that > created issues faster than they were fixed. Many of the issues were > only made visible by a re-review. Can we say that the ratio of creating > and fixing issues is more balanced now? How can we trust that if we'd > re-review again, we wouldn't end up with an even longer list? We (Angel) found several patches with bugs after a re-review or a post-submit review in the past. Some of these bugs could have been oversights, but other bugs were too conspicuous to be accidental oversights. It felt as if those reviews (mainly those for CrOS flashrom upstreaming patches) were heavily expedited for some reason that we (Angel, and possibly the upstream flashrom community) don't know about. If anyone knows, we'd appreciate if they could help understand why things happened this way, especially if they can shed some light regarding what happened inside Google at that time. > > There is one issue that came up around May > > https://ticket.coreboot.org/issues/390 . This one is new. It does not > > prevent from using any functionality, and also if the user does not > > specify `--progress` they won’t even notice. > > Can the latter be proved? I'd say yes, but it seems like more effort > then re-writing the whole patch. I never fully reviewed the patch but > in the few places I looked into I saw wrong calculations and overflows. > And those run independently from the `--progress` switch. And please > remember we are talking about C code. To say that it doesn't affect > existing functionality would require a proof of defined behavior. > Because if there is undefined behavior, flashrom could just crash. > > Also worth to remember: flashrom often runs with root privileges. So > every coding mistake is a security issue unless we can prove that it > isn't. We fully agree with Nico's view. We hadn't thought about the safety and security implications of such changes. Hmmm, maybe we should rewrite flashrom in at-least-Ada then? :D > > Lots of code improvements: for example we have way less global state > > than it used to be 1/2yr ago, and we have more unit tests of all > > types. > > The former is in the category of "invasive
[flashrom] Re: master state / potential release
If it helps, a monthly release seems to be the sweet spot for fwupd, from a "writing release notes" and "getting fixes into users hands" point of view. There's no point having a "no regressions" rule as this is software, and even the most benign of changes can have some unknown side effect -- it's much better to just make releases "cheap" and do lots of them. In fwupd the main branch is always "releasable" so if we have a high-priority fix or security issue we just merge, write release notes, tag, push outside the usual release cadence. Richard On Mon, 17 Oct 2022 at 12:24, Edward O'Callaghan wrote: > > I very much like Richard's pragmatic approach here. > > It would be my view that we should re-evaluate branch critical bugs (sb600spi > map issue + build system stuff are the two the most prominent in my mind at > the moment) and just cut a release branch, stabilise that with some > cherry-picks of any residue items. Forge forwards from that with a more > regular cadence exactly how Richard suggests. If some critical issue comes up > we can do a point release. > > Kindest regards, > Edward. > > > On Mon, 17 Oct 2022 at 20:27, Richard Hughes wrote: >> >> Hi all, >> >> Perhaps a way forward would be to tag a rc release, ask distributors >> to package and test it (e.g. pushing to Fedora Rawhide, but not Fedora >> 36), and then push the actual official release a week or two later? >> >> Certainly aiming to do releases monthly is much healthier than doing >> releases every few years. I think it's much more achievable to set the >> expectation to the end user "sorry for the regression! it'll be fixed >> in your distro in ~3 weeks, in the meantime use the previous release" >> than trying to squash every bug and add all the features before >> tagging a mythical beautiful bug-free "feature complete" release. >> >> Richard. >> >> On Mon, 17 Oct 2022 at 10:19, Nico Huber wrote: >> > >> > Hi Anastasia, >> > >> > On 16.10.22 23:41, Anastasia Klimchuk wrote: >> > > Nico what was the goal why you started this thread? >> > >> > I wanted to evaluate how immediate a release is. Also, maybe >> > subconsciously (on my end), to raise awareness among developers >> > that we want to do releases. >> > >> > I learned some new things between my first two emails, overall >> > that has changed my intentions to raising awareness. >> > >> > > >> > > I thought for a moment, you have a *quick* question. So I answered. >> > > Now you seem to be unhappy to learn that we are closer to the release >> > > than 1/2 yr ago. >> > >> > It seems hard to assess to me. I want to make sure that we don't >> > miss anything. For instance, the progress in the tickets: A lot >> > have been closed. But some of them look like they were low-hanging >> > fruits, i.e. easy to fix. While bigger issues remain and at least >> > one new issue popped up, as you mentioned. I find it very hard to >> > measure. >> > >> > > You are saying "But I don't believe you", what was the point of asking >> > > then? >> > >> > I did not say that nor did I intend to make it look like this. I still >> > feel misunderstood and that we are talking past each other. I didn't >> > mean to ask "how many tickets are open" or "what was merged". These are >> > things that are easily visible. I guess what I really want to know is >> > "did the regression rate decline?", "does it look like the review >> > process changed enough so that it could?". >> > >> > Nico >> > >> > PS. Anastasia, there is a lot of "you, you, you" in your email. This >> > can get very emotional. I don't see a need to get personal nor to fight. >> > ___ >> > flashrom mailing list -- flashrom@flashrom.org >> > To unsubscribe send an email to flashrom-le...@flashrom.org >> ___ >> flashrom mailing list -- flashrom@flashrom.org >> To unsubscribe send an email to flashrom-le...@flashrom.org ___ flashrom mailing list -- flashrom@flashrom.org To unsubscribe send an email to flashrom-le...@flashrom.org
[flashrom] Re: master state / potential release
I very much like Richard's pragmatic approach here. It would be my view that we should re-evaluate branch critical bugs (sb600spi map issue + build system stuff are the two the most prominent in my mind at the moment) and just cut a release branch, stabilise that with some cherry-picks of any residue items. Forge forwards from that with a more regular cadence exactly how Richard suggests. If some critical issue comes up we can do a point release. Kindest regards, Edward. On Mon, 17 Oct 2022 at 20:27, Richard Hughes wrote: > Hi all, > > Perhaps a way forward would be to tag a rc release, ask distributors > to package and test it (e.g. pushing to Fedora Rawhide, but not Fedora > 36), and then push the actual official release a week or two later? > > Certainly aiming to do releases monthly is much healthier than doing > releases every few years. I think it's much more achievable to set the > expectation to the end user "sorry for the regression! it'll be fixed > in your distro in ~3 weeks, in the meantime use the previous release" > than trying to squash every bug and add all the features before > tagging a mythical beautiful bug-free "feature complete" release. > > Richard. > > On Mon, 17 Oct 2022 at 10:19, Nico Huber wrote: > > > > Hi Anastasia, > > > > On 16.10.22 23:41, Anastasia Klimchuk wrote: > > > Nico what was the goal why you started this thread? > > > > I wanted to evaluate how immediate a release is. Also, maybe > > subconsciously (on my end), to raise awareness among developers > > that we want to do releases. > > > > I learned some new things between my first two emails, overall > > that has changed my intentions to raising awareness. > > > > > > > > I thought for a moment, you have a *quick* question. So I answered. > > > Now you seem to be unhappy to learn that we are closer to the release > > > than 1/2 yr ago. > > > > It seems hard to assess to me. I want to make sure that we don't > > miss anything. For instance, the progress in the tickets: A lot > > have been closed. But some of them look like they were low-hanging > > fruits, i.e. easy to fix. While bigger issues remain and at least > > one new issue popped up, as you mentioned. I find it very hard to > > measure. > > > > > You are saying "But I don't believe you", what was the point of asking > then? > > > > I did not say that nor did I intend to make it look like this. I still > > feel misunderstood and that we are talking past each other. I didn't > > mean to ask "how many tickets are open" or "what was merged". These are > > things that are easily visible. I guess what I really want to know is > > "did the regression rate decline?", "does it look like the review > > process changed enough so that it could?". > > > > Nico > > > > PS. Anastasia, there is a lot of "you, you, you" in your email. This > > can get very emotional. I don't see a need to get personal nor to fight. > > ___ > > flashrom mailing list -- flashrom@flashrom.org > > To unsubscribe send an email to flashrom-le...@flashrom.org > ___ > flashrom mailing list -- flashrom@flashrom.org > To unsubscribe send an email to flashrom-le...@flashrom.org > ___ flashrom mailing list -- flashrom@flashrom.org To unsubscribe send an email to flashrom-le...@flashrom.org
[flashrom] Re: master state / potential release
Hi all, Perhaps a way forward would be to tag a rc release, ask distributors to package and test it (e.g. pushing to Fedora Rawhide, but not Fedora 36), and then push the actual official release a week or two later? Certainly aiming to do releases monthly is much healthier than doing releases every few years. I think it's much more achievable to set the expectation to the end user "sorry for the regression! it'll be fixed in your distro in ~3 weeks, in the meantime use the previous release" than trying to squash every bug and add all the features before tagging a mythical beautiful bug-free "feature complete" release. Richard. On Mon, 17 Oct 2022 at 10:19, Nico Huber wrote: > > Hi Anastasia, > > On 16.10.22 23:41, Anastasia Klimchuk wrote: > > Nico what was the goal why you started this thread? > > I wanted to evaluate how immediate a release is. Also, maybe > subconsciously (on my end), to raise awareness among developers > that we want to do releases. > > I learned some new things between my first two emails, overall > that has changed my intentions to raising awareness. > > > > > I thought for a moment, you have a *quick* question. So I answered. > > Now you seem to be unhappy to learn that we are closer to the release > > than 1/2 yr ago. > > It seems hard to assess to me. I want to make sure that we don't > miss anything. For instance, the progress in the tickets: A lot > have been closed. But some of them look like they were low-hanging > fruits, i.e. easy to fix. While bigger issues remain and at least > one new issue popped up, as you mentioned. I find it very hard to > measure. > > > You are saying "But I don't believe you", what was the point of asking then? > > I did not say that nor did I intend to make it look like this. I still > feel misunderstood and that we are talking past each other. I didn't > mean to ask "how many tickets are open" or "what was merged". These are > things that are easily visible. I guess what I really want to know is > "did the regression rate decline?", "does it look like the review > process changed enough so that it could?". > > Nico > > PS. Anastasia, there is a lot of "you, you, you" in your email. This > can get very emotional. I don't see a need to get personal nor to fight. > ___ > flashrom mailing list -- flashrom@flashrom.org > To unsubscribe send an email to flashrom-le...@flashrom.org ___ flashrom mailing list -- flashrom@flashrom.org To unsubscribe send an email to flashrom-le...@flashrom.org
[flashrom] Re: master state / potential release
Hi Anastasia, On 16.10.22 23:41, Anastasia Klimchuk wrote: > Nico what was the goal why you started this thread? I wanted to evaluate how immediate a release is. Also, maybe subconsciously (on my end), to raise awareness among developers that we want to do releases. I learned some new things between my first two emails, overall that has changed my intentions to raising awareness. > > I thought for a moment, you have a *quick* question. So I answered. > Now you seem to be unhappy to learn that we are closer to the release > than 1/2 yr ago. It seems hard to assess to me. I want to make sure that we don't miss anything. For instance, the progress in the tickets: A lot have been closed. But some of them look like they were low-hanging fruits, i.e. easy to fix. While bigger issues remain and at least one new issue popped up, as you mentioned. I find it very hard to measure. > You are saying "But I don't believe you", what was the point of asking then? I did not say that nor did I intend to make it look like this. I still feel misunderstood and that we are talking past each other. I didn't mean to ask "how many tickets are open" or "what was merged". These are things that are easily visible. I guess what I really want to know is "did the regression rate decline?", "does it look like the review process changed enough so that it could?". Nico PS. Anastasia, there is a lot of "you, you, you" in your email. This can get very emotional. I don't see a need to get personal nor to fight. ___ flashrom mailing list -- flashrom@flashrom.org To unsubscribe send an email to flashrom-le...@flashrom.org
