On Wed, Jan 16, 2013 at 11:43 PM, Dan Scott <[email protected]> wrote:
> So, on the bright side, we got three releases out today, and turned > around a fix for a critical security problem in minimal time in a > collaborative fashion, and had blog posts and everything. > > I have a few concerns, though, which at the risk of sounding overly > negative I'll list out here... > > I saw no mention of testing the release tarballs on various distros > before they were released. I had asked for sanity testing of my 2.1.5 > tarball but didn't get any response. Security releases suck but still > need QA. How much sanity testing did these most recent releases get? > You ask the toughest question first, so please stay with me after this answer. I'm not going to avoid looking this question in the eye. I generally confirm a successful build, start the staff client, and test the database upgrade script (all on one distro only, plus Windows for the staff client). I don't believe any significant number of Evergreen releases, in my hands or anyone else's, get any more direct testing than that. I know that many releases have had less. If code gets tested when merged master, I believe we've rested on that as an excuse to act as if releases are just a detail of publishing code that's already likely to work. I am aware of how lame that is (really), but I also don't know where we're going to get more QA without making users wait unreasonably long times for releases. Despite major, laudable efforts from diverse contributors, who do we have who can drop everything to put a package (or several) through its paces every month? We have to improve, I agree, but it's not clear to me how we meet timed release goals, which we all agree are good, and at the same time wait for proper testing. Where is the manpower? In the spirit of being constructive, maybe we could move back release cutting by a week, but keep the date when we actually release them the same? More testing would be possible then. Is this the kind of thing you might go for? I thought the outcome of one of the last meetings was that we were going > to adopt a more open security process, one that would enable community > members to contribute towards testing? > > Security releases seem to invoke our lizard brains, and we just act to get these releases out fast. We need to change that, and if we concretized those ideas from the recent meeting, let's bring that document out front and center to remind ourselves. Perhaps due to the lack of QA, our 2.2 and 2.3 tarballs once again > contain the full xulrunner binary packages for both win32 and > linux-i686, bloating their sizes by tens of megabytes. Bloat aside, > for legal reasons it's generally not advisable to ship binaries > from other projects if you can avoid it. > By "once again" I can tell there has been a conversation about this before, which I must have missed, but I was not aware of this problem at all. I agree, let's avoid packaging these binaries going forward. > > I think our release processes need some work. > > Exhibit A: Even for a security release, we should be able to prep a > git branch over in the security repo, cut a tarball from it, and > then when the release is announced, just "git push origin > security/rel_2_1_5:rel_2_1" > so that the upstream release branch is a perfect replica of what went > into the tarball. > Instead, at least for the rel_2_1 branch, two times today > commits went into the rel_2_1 branch despite my having painstakingly > prepped a complete branch over in the security repo last night (which > also included commits from rel_2_1_4 that were missing from rel_2_1). I > found that pretty frustrating, although I certainly overreacted today > (probably due to staying up until 2:00 AM to prep the materials for the > release and not getting enough sleep to act like a decent human being). > Neither Bill nor I were aware of the importance of the perfect replica. We weren't trying to wreck your work, we just didn't understand that was a thing. We probably need further discussions, in IRC perhaps, to make sure we understand the most correct way of structuring the tags in relation to the release branch. I'm also unclear on how we deal with the generated ChangeLog that goes into releases, but presumably shouldn't go into the rel_2_n branch. > > Exhibit B: We seem to keep stumbling over how to keep upgrade scripts in > sync. Back when I proposed the version-upgrade directory as a means > of stamping out the problems we had historically had with fixes for > point-to-point scripts not making their way into all the right > releases, I also proposed that all point-to-point version upgrade > scripts should be committed first to master, then backported to each > applicable release. Can we commit to taking this approach? Or is there a > better alternative? > We can, and should. We need to change the make_release script, which creates the upgrade script and commits it in such a way as to encourage us to push it to the wrong place first. Is this also possibly the right answer for the generated ChangeLogs? > Exhibit C: Open-ILS/src/perlmods/lib/OpenILS.pm keeps saying 'our > $VERSION = "2.00";' - in rel_2_1 land I've been trying to remember > to rev that (although I think the 2.1.5 release actually says 2.14 - > sigh). The release script is all well and good--I believe in better > living through automation--but the script needs regular injections of > human intelligence too. See also QA above! (Aside--what happens when we > get to 2.2.10? Maybe we should be using 2.015, 2.025, 2.033 instead of > 2.15, 2.25, 2.33...) > > So... maybe we can document our release process a bit more, with a > checklist of steps to follow to create a tarball and of things to QA? > Yes. We need a human script first. > > Exhibit D: General confusion around translation processes. They're a > black box for most people, they don't result in quality output for > groups serious about their translations, there's an impedance mismatch > between the tools some translation groups want to use and between casual > translators... we could really use a dedicated translation coordinator > to step up and sort out some improvements on this front. Any volunteers? > I'm not the volunteer, but I second your call for one. I would like to have further talks about the translation merging processes. Today I follow them only mechanically, with little understanding of how they work. I'd like it if all the release managers/maintainers could talk to you about the big picture of how they work. I think you're the only one here who at least mostly gets it. > On tagging releases: > > I've brought this up in IRC a few times, but as far as I can tell the > only reason that we're not using actual "git tag" to create annotated > GPG-signed tags for releases is because one of the git admins doesn't > like git tags. I have to say thanks to Galen for having documented > the procedure for OpenSRF, which has worked well for me over there to > date, and I wish we would use actual tags in Evergreen instead of > branches that are named "tag/blah". > I assume the resistance is based on the idea that we're likely to cut a release and then need to make a quick correction when something has been missed (if not that, then what?) I think others of us (me anyway) are just neutral on this question, because we aren't aware of any hard advantages of real, signed tags other than a vague idea of "correctness." I am willing to be enlightened. -- Lebbeous Fogle-Weekley | Software Developer | Equinox Software, Inc. / The Open Source Experts | phone: 1-877-OPEN-ILS (673-6457) | email: [email protected] | web: http://www.esilibrary.com
