2018-03-23 9:52 GMT+01:00 Robert Bradshaw <rober...@google.com>: > To put this in context, this was a brain dump of some of the things I > encountered while doing the release. Were I to do a release again, it would > be a lot easier (though still not ideal). > > At the high level, rather than focusing on steps, I think it's more > interesting to focus on why we need a human to do the release. IMHO, the > role of a human is > > 1) Choose a commit. Any commit in master should do, if we had proper test > and code hygiene. > 2) Sign the release artifacts. (Frankly, I would be happy with a robot > signing everything but the tag in github, if the keys could be properly > maintained.) > 3) Manage the email thread. > > To this end, I would like a process where one would propose a release > candidate via a lightweight, fast tool. and jenkins (or some other system) > would recognize the release branch and test, generate artifacts, push > artifacts, and test against those artifacts (including, ideally, nexus and > svn staging). One should be able to easily run this locally, but that > shouldn't be needed. If a human needs to sign, there could be a script for > one to download artifacts, sign them, and push the signatures. On success, > an email body (with all the links and details) would be created, to be sent > out by a human. > > More responses inline below (and thanks for the feedback!). > > > On Thu, Mar 22, 2018 at 11:10 PM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > >> Hi >> >> Le 23 mars 2018 04:29, "Alan Myrvold" <amyrv...@google.com> a écrit : >> >> Robert explained his experience with the release process >> <https://beam.apache.org/contribute/release-guide/> as the release >> engineer for 2.4.0, and we discussed the prototype shell script for >> checking release progress in pull/4896 >> <https://github.com/apache/beam/pull/4896>. >> >> I'd like to help automate the release process, initially just checking >> that all steps look ok, then automating all feasible steps, with the goal >> of reducing the effort of the release engineer per release to less than 1 >> hour for creating the first RC. >> >> Overall, it is a large, scattered process. For someone who has done this >> many times (like jb@ in the previous release), it is likely easy. Robert >> was familiar with pypi, making that part easy for him. He was not as >> familiar with the java release artifacts or nexus, making that more of a >> challenge. >> >> - Several steps are not reversible, so it isn't a restartable process >> if there are errors. >> >> Can we ensure they are done last? Maven introduced the deployAtEnd >> feature so i guess it can be a model used here. >> > > In retrospect, aside from pushing stuff, things weren't strictly > irreversible. I got to be good friends with git clean, git checkout, and > git reset. That last one, however, always feels wrong to use (and is > error-prone). > > I also don't think I'll ever get used to the build system (mvn or gradle) > editing code and creating commits, but perhaps that's needed with the way > -SNAPSHOTS are special (though having to edit every file seems overkill). >
it is fully automated and hidden behind mvn release:prepare normally (gradle as the equivalent but its design is a bit different so it can be a bit more natural for some people), did you do it manually? > >> - Several steps have high latency; it may take 30min of work before >> prompting for a password. >> >> What prevents to put the pwd in settings.xml or use an agent - to not be >> prompted? >> > > Foreknowledge :). I did end up using an agent. (Personally, I have some > qualms putting my gpg password in a plaintext xml file). There were also > gitbox prompts in some of the steps, and of course svn commit (though the > svn steps weren't high latency). can't remember if there were others. > Can it be worth a word in the doc? Side note: on linux you dont have to put it in clear. > >> - Problems with both his laptop and desktop; GCS wasn't working well >> on the laptop preventing running tests. Maven extremely slow on his >> desktop, but he discovered a workaround in his configuration. Would have >> been nice to use jenkins for most steps instead of relying on >> laptop/desktop configurations. >> >> As JB mentioned, these were environmental issues. But it wasn't clear at > the time (being new with the release process) and could have been elevated > had I not had to do so many manual steps (including running the tests). I > happened to get particularly unlucky with timing with one of them too. > >> >> - >> - Many of the steps ran the same tests over and over. Sometimes tests >> were flaky, so needed to restart a long process due to a test that had >> passed earlier now flaking. >> >> Wonder if remote tests shouldnt be mocked *during* a release to avoid >> that no luck effect. >> > > We should be able to release from a known good commit, as verified by > jenkins, and never run tests again. > You can put <skipTests>true</> (+ others like invoker.skip if needed) in your settings.xml properties and do it ;) - good/not good debate can start but if you built twice before it should be good enough IMHO. > > >> >> - Robert was new to Nexus, so setting up permissions and navigating >> the UI was confusing. >> >> When it happens dont hesitate to ping here. >> >> >> - >> - Needed to rebuild the google cloud dataflow containers to get >> dataflow working with the RC, and that ended up being a painful process. >> The github sdk/python/container is part of the portability effort and >> should help eliminate googlers needing to do steps like this with each >> release because that container can be built externally. >> >> Is there a way to see this process or is it "closed"? >> > > No, there's no way to see this process. Nothing secret here, but the > spanning of multiple revision control systems, multiple languages, > branching (on both sides) and dependencies (upstream and downstream) and > some archaic setups make this less than ideal. The move to portability > should be a huge win here in decoupling what's built internally vs. > externally (containers). > > >> >> >> - >> - Automated release notes were not seen as valuable due to >> limitations in any automated documentation. >> >> >> If needed i have a script to do it from jira, i filter issue by the label >> "changelog". RM must review tickets before the release with that. >> >> >> - Missed a step of changing the java/python version numbers but was >> able to fix that. >> >> Is it doable through maven/gradle filtering? >> > > Note sure. Ideally the version numbers could be pulled out of the pom/a > central location. > > >> >> - Some copy/paste errors when creating the voting emails. >> >> >> At tomee we had a cli to do it if interested. >> > > +1 > work was pushed here http://svn.apache.org/repos/asf/tomee/sandbox/release-tools/, there still have some String constants to change manually and it was backed on svn for some part but the whole process was supposed to be here. Can be a start to fork. > >> - Many steps are not possible for non-committers. >> >> Sadly intended/expected but you cannhelp on jira review, snapshot >> validation before the release. >> >> >> - The prototype shell script was seen as helpful, especially since it >> can be restarted. Some concerns over the maintainability of such a large >> shell script. >> >> Move to groovy? Or is it just a size issue? >> > > In my (first and second hand) experience, bash scripts become difficult to > maintain (readability and correctness issues) once they reach a certain > size and complexity (not just size, but control structure, etc.) There are > several alternatives, groovy and Python among them. > If the ^^ repo seems interesting we can create a beam-release repo and just use CLI java based "scripts", maybe. Also note asf has tentacle/creadur to help to check the legal part of a release so maybe projects which can be enriched with more automotion for releases. > > >> >> The steps that should not be automated, and need human involvement: >> >> 1. All emails (propose release, ask for votes) >> 2. Picking the commit to start the release >> 3. Signing artifacts >> >> Most everything else should be possible to automate, although >> non-committers do not have access to logging into jira, nexus, or the >> jenkins ui, making some of this tricky to automate for non-contributors. >> Also not clear to us how nexus picks he sequential artifact suffix (1031)? >> >> >> It is a sequence - as db sequence, dont recall if it is per user or repo >> (repo from memory but not 100% sure). If that is to automate close and >> mailing maybe check sonatype plugin which replace deploy one - ensure to >> deactivate auto release - or just nexus api which returns this value. A >> maven (gradle) extension should be writable too. >> >> >> Next steps for me are to enhance the release-checking script, automate >> feasible actions, and pair with the next release engineer to make this >> smoother, especially if they are at google, but even if they are not. >> >> >> +1, anything should be bound to release:perform except the vote process >> and the postvote tasks which can be automated (dist update, site update, >> release staging, ....) >> >> >> Alan >> >> >> > > On Thu, Mar 22, 2018 at 11:53 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > >> * Robert was new to Nexus, so setting up permissions and navigating > the UI was > >> confusing. > > > > Not a release concern IMHO, Nexus is straight forward for release with > staging > > repo. I think it's well explained in the release guide. > > Nexus felt a bit clunky, but was usable. I think it would make a lot more > sense to someone well versed in maven packaging structure and terminology. > E.g. I still don't really know what the difference is between > https://repository.apache.org/content/repositories/staging/org/ > apache/beam/ and https://repository.apache.org/content/repositories/ > orgapachebeam-1031/ > First is a repo called "staging" whereas last is a generated repository at deploy time for staging purposes (see it as being volatile). You can see the first at https://repository.apache.org/#view-repositories;staging~browsestorage whereas the second will be only at https://repository.apache.org/#stagingRepositories. Concretely the first is a proxy (nexus group) like releases or snapshots but usable by staging repos IIRC whereas the second is a detached repo which will join releases one once released. > > (It also didn't help that this is my first release with the exception of a > Python-only bugfix release last year, and so I skipped the "one time setup" > steps at first this time thinking I'd done them, but last time I didn't > bother with nexus because I didn't need it. Again, a one-off, but someone > else might have similar one-offs.) > > > >> * Many steps are not possible for non-committers. > > > > And it's normal per Apache rule. > > Yes, I think it's fine to require a committer to be in the loop, but maybe > not eveywhere. > > > * Missed a step of changing the java/python version numbers but was > able to > > fix that. > > > Changing version numbers where ? Release plugin (in the prepare goal) > already > > change the versions in the POMs (like a mvn versions:set). I guess you > are > > talking about some other files ? Maybe those files could use the > project.version > > from the pom ? > > There are version numbers in the code as well. Yes, there should be one > source of truth. > There is a java template maven plugin to filter java sources, i don't see any blocker to filter any files from the pom (build.gradle).
