Digging up this old thread as any offer to help really deserves a response. I had travelled to help dear old mom these two weeks and didn't get back to you after I had a bit more bandwidth.
> On Sep 18, 2021, at 9:31 AM, Jenkins, Rodney J (Rod) > <[email protected]> wrote: > > David, > > Thank you for the insights and explanation! > > I completely understand the technical debt and the challenge of making this > better during a release. I would like to jump in and see where I can help. > My problem is I am not a java developer. What I am good it is automating > tasks, if I can be taught to execute them. > > The big ask is: Would anyone want to take the time with me to educate me on > what has to happen for a release (not during a release)? I am thinking that > we could set up a dummy repo that has some simple small java code in it to be > a dummy TomEE release candidate. Create some dummy destinations that mimic > where the artifacts must be placed. Once I understand the process, I can see > about making it repeatable. > > Personally, I would like to see it done in a way that someone with lesser > skills (like, but not necessarily, me) does releases. The way I see it now > is the heavy hitters do the releases. I think their time would be better > spent on the technical debt, bugs, etc. Maybe we could find a small few that > would be wiling the be release specialists. I know some where I work that > MAY be interested. If someone would teach me, I would teach them. You have the right spirit. In the early days of Geronimo I did the majority of releases and while I did a good job, wasn't the job I wanted to be doing and could see how me doing all the releases created a knowledge vacuum that actually hurts the community. What I did was suggest we start a system where each release had a pilot and a copilot. The pilot would be an experienced person who know what they were doing, the copilot would be someone learning. Next release the copilot would become the pilot and do the release and a new person would become the copilot. Each release the documentation got a bit better, the technical debt paid down a bit. Things improved, releases got more frequent and quality went up. A significant blocker, however, is that a large number of the steps can't be done unless you're an Apache committer. Not because of policy, just because they require access to systems only committers can reach. You won't be able to run a lot of the commands. That's not a show stopper, it just means it our creativity in how to leverage you will be heavily challenged. I'm up for it if you are. I have been trying to revitalize our release tools in java for automating as many release tasks as possible. I do think that's the right way to go as the majority of people who do releases are java developers. In the past I've written elaborate scripts in bash and inevitably they decay as I'd be the only one who understood them. That doesn't necessarily mean there's no role for scripting. One of the hardest parts of doing releases is remembering all the steps that have to be done and double checking they were done correctly. This is absolutely something you could script up and run. I will say in all my attempts to document release processes, I've noticed even in the best circumstances they're always a little out of date. Turn your back even a little and they become just out of date enough no one even looks at the doc anymore, than it's game over. If we had a release auditing script that could be pretty amazing. It could be the checklist people use to "see" the release process as a whole. Again I'm imagining something like the System V startup output where there's one line per task. Each line is a step that has to be done a green/red colored status on the right. Here are some things I frequently see done wrong and not noticed: - Forgetting to update the keys file. Topic of the original thread. Because it's something you only need to do once in a while, it's easy to forget. There's a release tools command to help people do it, but you have to remember you rotated your key a few months ago and need to run it. Perfect thing to audit. - Missing signatures once published. We've had infrastructure ask us to add signatures because we've forgotten a few times. There's already a script/command to do it, but nothing to check if people use them. As mentioned, people stop using scripts they find confusing, so they decide to wing it instead. A script to audit this part would be great. Apache infra has such a script and does run it every few months, but it's not ideal when they're the ones point it out. - Leaving SNAPSHOT or old TomEE dependencies in the examples or build. I spent three days cleaning out references in the examples to older TomEE 7 and 8 snapshots missed by past releases. The way you check is delete your ~/.m2/repository directory and build the release tag from git. If there is anything with a SNAPSHOT dependency in there, that's a fail. If there are any jars from previous releases in there, that's a fail. We should not pass these release votes, we should fix the source and roll again. Again, there is a tool to do this (maven) but it doesn't always catch everything. It only renames references to the current version to the next version. If there's a reference to a version say two releases ago, it will not see it. It will also not rename references for any modules that are commented out, which happens with the examples from time to time. It really needs auditing. - Updating the generated download page HTML. Frequently people will directly update the download.html page, just hacking in the new release. The problem of course is all the html pages are generated and it will inevitably get overridden and set back to the previous release the next time the website is generated. It doesn't seem to matter that repo containing generated HTML files has "do make updates here, this is all generated" written on it. We still get this mistake from committers and contributors alike. We in fact just had a PR opened against the generated html repo. On release, people need to know and remember to update the markdown/asciidoc source which is in a different repo. There's a specific file we could audit via a simple grep and flag up "tomee-site-generator download.adoc page updated [FAIL]" - Forgetting to publish the maven artifacts. Only happened once, but there was an occasion where we forgot to click the publish button on the nexus repository holding the maven distributions of all the release jars. They expired and were deleted. That meant we had a release that was on the website, in the download area for people that wanted to just grab a zip file, but was nowhere maven user could access. During the release process a special temporary release area is created on repository.apache.org, we need to keep an eye on that and audit that it was in fact released and everything there has reached Maven Central. I could probably think of a half dozen more checks, but the above is hopefully enough to paint the picture that it would be useful. There are systems to do everything above, just you know ... darn people being all human :) None of the above is enough detail to actually implement most of this, so we can have better discussions if we actually want to do any of it. Another aspect to this idea, Richard Zowalla has been doing a lot of fun experimentations with Github Actions. There is definitely scripting potential there and some of the above could be triggered when release tags show up in git. Unlike setting up Jenkins jobs, it's way easier to contribute to Github Actions as it's all git-based so PRs work perfectly. You could test on a forked repo and then submit a PR. Some real potential to make outside help possible with this capability. Some ideas and thoughts. Hope not an overwhelming amount. :) Any thoughts extremely welcome! -David
smime.p7s
Description: S/MIME cryptographic signature
