Hi Everybody Yesterday we released a weekly version from the new release environment and while we managed to ship our weekly version, we also faced unplanned issues that we had to solve so here a quick retro.
The process was done in two stages the release and then the packaging Stage 1, Release: The release part, was the easiest one. We only add two issues. The first issue was due to the password used to authenticate on Maven repository, containing special characters that couldn't be parsed by Maven release plugin. We just changed the password. The second problem was a permission issue on repo.jenkins-ci.org which was easily fixed. In the result of this stage, as expected every artifact was pushed to repo.jenkins-ci.org Stage 2, Package: Well, this stage was more tricky as we heavily refactored that part while also heavily relying on existing service. We discovered multiple side cases that we didn't catch during the testing period and I will describe here the most important ones. * GPG: While this was something that we discussed a long time ago we forgot to communicate in advance but the GPG key used to sign releases changed for weekly release in order to not use Kohsuke gpg key. In a result of this, people who use RedHat/Debian/Suse repository need to reimport the new key. Remark: this is something that will also have to be done for Lts once we also release Lts with the current process New Key: pub rsa4096 2020-03-30 [SC] [expires: 2023-03-30] 62A9756BFD780C377CF24BA8FCEF32E745F2C3D5 uid Jenkins Project <[email protected]> sub rsa4096 2020-03-30 [E] [expires: 2023-03-30]. Old Key: pub dsa1024 2009-02-01 [SC] 150FDE3F7787E7D11EF4E12A9B7D32F2D50582E6 uid Kohsuke Kawaguchi <[email protected]> uid [jpeg image of size 3704] uid Kohsuke Kawaguchi <[email protected]> uid Kohsuke Kawaguchi <[email protected]> uid Kohsuke Kawaguchi <[email protected]> sub elg2048 2009-02-01 [E] sub rsa4096 2016-11-01 [S] * mirror.jenkins-ci.org: This service has now additional Html files `HEADER.html` and `FOOTER.html` per directory. Those two files are used by apache directory listing to prepend and append Html information to the directory listing. The main advantage of this is to provide information from mirrors about how to install those packages from your operating system from every mirror. At least apache with the correct configuration, an example of this is http://mirrors.jenkins-ci.org/debian/ or http://archives.jenkins-ci.org/debian/ Please note that the instruction to configure os package manager is referencing the wrong URL and it should be pkg.jenkins.io. A fix needs to be provided, other remark CSS needs to align with other Jenkins website if someone is willing to work on this, feel free to do, it's defined here https://github.com/jenkinsci/packaging/blob/INFRA-910-core-release-automation/templates/base.html * pkg.jenkins.io: this service is used by RedHat/Debian/Suse package manager to know which version can be installed, previously a list of packages that could be download from pkg.jenkins.io was generated for each release, this won't be the case anymore instead a link to mirrors should be provided. As a result of this, the latest weekly version is not listed even though you can easily craft an URL. We still have improvement coming and the next release can only be smoother. Thanks to Tim Jacob, Marky Jackson, Oleg Nenashev, Alex Earl, Mark Wait, Daniel Beck, Kohsuke, Oliver Ogondza who helped me to ship this first release yesterday but also to all the people who helped me, over the last two years to make this project a thing. Have a good day, Cheers On Wed, Apr 15, 2020, at 10:53 PM, Olblak wrote: > After some discussion with the people involved in the process, I'll start at > 11AM UTC instead of 3PM UTC, so we have more time to fix any unplanned issues. > > On Wed, Apr 15, 2020, at 9:51 AM, Oleg Nenashev wrote: >> I confirm my availability for tomorrow. Will also spend some time today to >> merge changes into the Jenkins core so that the release includes some real >> enhancements and fixes for Jenkins users. >> >> Regarding the schedule, it would be great to start the release process a bit >> earlier so that we have more time to fix issues if any. E.g. we could start >> the release Pipeline early so that artifacts are ready to packaging when US >> wakes up. >> >> Best regards, >> Oleg >> >> >> >> On Tuesday, April 14, 2020 at 9:04:03 PM UTC+2, Olblak wrote: >>> Today, during the infra meeting, we discussed creating a new weekly release >>> from the new process this Thursday the 16th at 3PM UTC. >>> While there won't be major features, the purpose of this release is to >>> validate that everything is working as expected in a real scenario. >>> >>> >>> I am proposing the following agenda >>> >>> >>> * 3PM(UTC), I start the release job and we release directly to the master >>> branch and directly on repo.jenkins-ci.org/releases. >>> Currently the release job take approximatively 1h30 >>> >>> >>> * 4:30PM(UTC), or early if ready, I start the packaging process in order to >>> publish artifacts to pkg.jenkins.io >>> Currently, the packaging job take approximatively 15min to run and I >>> estimate the script sync.sh to take 15min to run >>> >>> * 5PM(UTC), we verify that the release is working correctly. >>> >>> During the whole all process we'll be on IRC, #jenkins-infra channel >>> >>> Again feel free to raise any concerns you may have. >>> >>> Cheers >>> >>> On Tue, Apr 14, 2020, at 4:44 PM, Oleg Nenashev wrote: >>>> Thanks a lot to Olivier and all other contributors who invested lots of >>>> time to get it done! I am looking forward to finally get it over the line, >>>> at least for weeklies. >>>> >>>> FWIW, the Sunday's weekly got delayed (due to Easter). We could make an >>>> attempt to cut this release using the new flow without waiting for the >>>> next weekly. >>>> >>>> Best regards, >>>> Oleg >>>> >>>> >>>> On Tue, Apr 14, 2020 at 4:32 PM Olblak <[email protected]> wrote: >>>>> __ >>>>> Hi Everybody, >>>>> >>>>> This project is getting to the end and I am looking for a last round of >>>>> reviewers and testers before planning a weekly release with the new >>>>> process. >>>>> I opened a pull request (link >>>>> <https://github.com/jenkins-infra/release/pull/53/files>) with >>>>> instructions about how to trigger a release but currently, it only covers >>>>> weekly releases. >>>>> Our first objective is to successfully release weekly versions from >>>>> https://release.ci.jenkins.io, then we'll focus on stable and security >>>>> releases. >>>>> >>>>> For security releases, the way I see it: >>>>> 1) We add two security profile, one for weekly and the second one for the >>>>> stable in order to trigger a release from jenkinsci-cert/jenkins:master >>>>> to https://repo.jenkins-ci.org/releases/ >>>>> 2) Trigger packaging job(weekly or stable profile), >>>>> 3) We manually merge from jenkinsci-cert/jenkins:master to >>>>> jenkinsci/jenkins:master >>>>> >>>>> For stable releases: >>>>> 1) We update the stable >>>>> <https://github.com/jenkins-infra/release/blob/master/profile.d/stable> >>>>> profile branch to 'stable-<version>', variable JENKINS_GIT_BRANCH) >>>>> 2) We trigger stable release job >>>>> 3) We trigger stable packaging job >>>>> >>>>> Another topic that needs clarification is who should be able to trigger a >>>>> release from release.ci.jenkins.io and who will trigger the different >>>>> releases? >>>>> >>>>> Cheers >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Jenkins Infrastructure" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To view this discussion on the web, visit >>>>> https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com?utm_medium=email&utm_source=footer>. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Jenkins Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/cb175d2d-b8f3-403c-a0d1-5c2ebfdbfea6%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jenkinsci-dev/cb175d2d-b8f3-403c-a0d1-5c2ebfdbfea6%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/460c46ba-3205-4748-a90e-f52e1da5f36b%40www.fastmail.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/460c46ba-3205-4748-a90e-f52e1da5f36b%40www.fastmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/649dfada-eaa8-4169-9f96-e6173ee8e216%40www.fastmail.com.
