Wow, congratulations!! After 15 years and ~900 releases, this process has finally left my hands...
On Fri, Apr 17, 2020 at 2:32 AM Olblak <[email protected]> wrote: > Hi Everybody > > Yesterday we released a weekly version from the new release environment > and while we managed to ship our weekly version, we also faced unplanned > issues that we had to solve so here a quick retro. > > The process was done in two stages the release and then the packaging > > Stage 1, Release: > > The release part, was the easiest one. We only add two issues. > The first issue was due to the password used to authenticate on Maven > repository, containing special characters that couldn't be parsed by Maven > release plugin. We just changed the password. > The second problem was a permission issue on repo.jenkins-ci.org which > was easily fixed. > In the result of this stage, as expected every artifact was pushed to > repo.jenkins-ci.org > > Stage 2, Package: > > Well, this stage was more tricky as we heavily refactored that part while > also heavily relying on existing service. We discovered multiple side cases > that we didn't catch during the testing period and I will describe here the > most important ones. > > * GPG: While this was something that we discussed a long time ago we > forgot to communicate in advance but the GPG key used to sign releases > changed for weekly release in order to not use Kohsuke gpg key. In a result > of this, people who use RedHat/Debian/Suse repository need to reimport the > new key. Remark: this is something that will also have to be done for Lts > once we also release Lts with the current process > > New > Key: > pub rsa4096 2020-03-30 [SC] [expires: 2023-03-30] > 62A9756BFD780C377CF24BA8FCEF32E745F2C3D5 > uid Jenkins Project <[email protected]> > sub rsa4096 2020-03-30 [E] [expires: 2023-03-30]. > > Old Key: > pub dsa1024 2009-02-01 [SC] > 150FDE3F7787E7D11EF4E12A9B7D32F2D50582E6 > uid Kohsuke Kawaguchi <[email protected]> > uid [jpeg image of size 3704] > uid Kohsuke Kawaguchi <[email protected]> > uid Kohsuke Kawaguchi <[email protected]> > uid Kohsuke Kawaguchi <[email protected]> > sub elg2048 2009-02-01 [E] > sub rsa4096 2016-11-01 [S] > > * mirror.jenkins-ci.org: This service has now additional Html files > `HEADER.html` and `FOOTER.html` per directory. Those two files are used by > apache directory listing to prepend and append Html information to the > directory listing. The main advantage of this is to provide information > from mirrors about how to install those packages from your operating system > from every mirror. At least apache with the correct configuration, an > example of this is > > http://mirrors.jenkins-ci.org/debian/ or > http://archives.jenkins-ci.org/debian/ > > Please note that the instruction to configure os package manager is > referencing the wrong URL and it should be pkg.jenkins.io. A fix needs to > be provided, other remark CSS needs to align with other Jenkins website if > someone is willing to work on this, feel free to do, it's defined here > https://github.com/jenkinsci/packaging/blob/INFRA-910-core-release-automation/templates/base.html > > * pkg.jenkins.io: this service is used by RedHat/Debian/Suse package > manager to know which version can be installed, previously a list of > packages that could be download from pkg.jenkins.io was generated for > each release, this won't be the case anymore instead a link to mirrors > should be provided. As a result of this, the latest weekly version is not > listed even though you can easily craft an URL. > > We still have improvement coming and the next release can only be smoother. > > Thanks to Tim Jacob, Marky Jackson, Oleg Nenashev, Alex Earl, Mark Wait, > Daniel Beck, Kohsuke, Oliver Ogondza who helped me to ship this first > release yesterday but also to all the people who helped me, over the last > two years to make this project a thing. > > Have a good day, > > Cheers > > On Wed, Apr 15, 2020, at 10:53 PM, Olblak wrote: > > After some discussion with the people involved in the process, I'll start > at 11AM UTC instead of 3PM UTC, so we have more time to fix any unplanned > issues. > > On Wed, Apr 15, 2020, at 9:51 AM, Oleg Nenashev wrote: > > I confirm my availability for tomorrow. Will also spend some time today to > merge changes into the Jenkins core so that the release includes some real > enhancements and fixes for Jenkins users. > > Regarding the schedule, it would be great to start the release process a > bit earlier so that we have more time to fix issues if any. E.g. we could > start the release Pipeline early so that artifacts are ready to packaging > when US wakes up. > > Best regards, > Oleg > > > > On Tuesday, April 14, 2020 at 9:04:03 PM UTC+2, Olblak wrote: > > Today, during the infra meeting, we discussed creating a new weekly > release from the new process this Thursday the 16th at 3PM UTC. > While there won't be major features, the purpose of this release is to > validate that everything is working as expected in a real scenario. > > > I am proposing the following agenda > > > * 3PM(UTC), I start the release job and we release directly to the master > branch and directly on repo.jenkins-ci.org/releases. > Currently the release job take approximatively 1h30 > > > * 4:30PM(UTC), or early if ready, I start the packaging process in order > to publish artifacts to pkg.jenkins.io > Currently, the packaging job take approximatively 15min to run and I > estimate the script sync.sh to take 15min to run > > * 5PM(UTC), we verify that the release is working correctly. > > During the whole all process we'll be on IRC, #jenkins-infra channel > > Again feel free to raise any concerns you may have. > > Cheers > > On Tue, Apr 14, 2020, at 4:44 PM, Oleg Nenashev wrote: > > Thanks a lot to Olivier and all other contributors who invested lots of > time to get it done! I am looking forward to finally get it over the line, > at least for weeklies. > > FWIW, the Sunday's weekly got delayed (due to Easter). We could make an > attempt to cut this release using the new flow without waiting for the next > weekly. > > Best regards, > Oleg > > > On Tue, Apr 14, 2020 at 4:32 PM Olblak <[email protected]> wrote: > > > Hi Everybody, > > This project is getting to the end and I am looking for a last round of > reviewers and testers before planning a weekly release with the new process. > I opened a pull request (link > <https://github.com/jenkins-infra/release/pull/53/files>) with > instructions about how to trigger a release but currently, it only covers > weekly releases. > Our first objective is to successfully release weekly versions from > https://release.ci.jenkins.io, <https://release.ci.jenkins.io> then we'll > focus on stable and security releases. > > For security releases, the way I see it: > 1) We add two security profile, one for weekly and the second one for the > stable in order to trigger a release from jenkinsci-cert/jenkins:master to > https://repo.jenkins-ci.org/releases/ > 2) Trigger packaging job(weekly or stable profile), > 3) We manually merge from jenkinsci-cert/jenkins:master to > jenkinsci/jenkins:master > > For stable releases: > 1) We update the stable > <https://github.com/jenkins-infra/release/blob/master/profile.d/stable> > profile branch to 'stable-<version>', variable JENKINS_GIT_BRANCH) > 2) We trigger stable release job > 3) We trigger stable packaging job > > Another topic that needs clarification is who should be able to trigger a > release from release.ci.jenkins.io and who will trigger the different > releases? > > Cheers > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Infrastructure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web, visit > https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com > <https://groups.google.com/d/msgid/jenkins-infra/8656f659-17ef-49f3-b265-950488fd9b45%40www.fastmail.com?utm_medium=email&utm_source=footer> > . > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLBfFoN7FYM-gFO8NXixcz6nduoLH_HP6ysEsdRrPEgQcA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/cb175d2d-b8f3-403c-a0d1-5c2ebfdbfea6%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-dev/cb175d2d-b8f3-403c-a0d1-5c2ebfdbfea6%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/460c46ba-3205-4748-a90e-f52e1da5f36b%40www.fastmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/460c46ba-3205-4748-a90e-f52e1da5f36b%40www.fastmail.com?utm_medium=email&utm_source=footer> > . > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Infrastructure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web, visit > https://groups.google.com/d/msgid/jenkins-infra/649dfada-eaa8-4169-9f96-e6173ee8e216%40www.fastmail.com > <https://groups.google.com/d/msgid/jenkins-infra/649dfada-eaa8-4169-9f96-e6173ee8e216%40www.fastmail.com?utm_medium=email&utm_source=footer> > . > -- Kohsuke Kawaguchi -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAN4CQ4zREpFFcCSW%2BUPUPs1Y2NUfwTAxQ6A2myGvYxWgiHhr4g%40mail.gmail.com.
