Hi Everybody, Quick feedback on the recent security release.
While we successfully managed to publish everything in time, I must admit that it was more painful than I expected. So what happened? Yesterday, as part of the release procedure we trigger the maven release job but we quickly got hit by timeout errors between the Jenkins agent and the controller. As we faced the same issue on infra.ci last week, we had an idea on how to mitigate that issue and with the help of Gareth and Damien we quickly apply the same mitigation that we applied to infra.ci which was switching from using a Jenkins tunnel to a WebSocket connection as defined in this PR <https://github.com/jenkins-infra/charts/pull/1030/files> so we could finish the first release part Today we met again with Daniel Beck to finalize the release, so we build and publish packages but this time we got hit by two additional issues. First, we had WebSockets connection timeout error that we solved by increasing that value from 30sec to 60sec as defined in this PR <https://github.com/jenkins-infra/charts/pull/1032/files> The second issue that hit us was that we couldn't upload windows artifacts from the Jenkins agent to pkg.jenkins.io using the ssh-agent. The problem seems to be related to the latest ssh-agent plugin version which deleted non-exec based agent factories according to the changelog <https://github.com/jenkinsci/ssh-agent-plugin/releases/tag/ssh-agent-1.22> :p. Thanks to Gareth recent work on building a custom Jenkins image with our plugins as I could quickly pin down the previous version in this PR <https://github.com/jenkins-infra/docker-jenkins-lts/pull/232> So we first focus on publishing packages for non-Windows distributions, then I published the windows ones. In this process, I also identified additional issues. 1) Our monitoring didn't detect that the latest Jenkins version wasn't available from get,jenkins.io, which seems to be regression. 2) I had to manually trigger mirrrobits mirror scan to enable them, I don't know yet why it didn't automatically scan mirrors. So what's next? Next week we'll update the AKS cluster used by the release environment hoping that it will solve all the network issues we currently have. Regarding the ssh-agent issue with the windows containers, I am looking for someone with better windows skills than mine :) to spare me some debugging time. Cheers, Olivier -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/1782f6e3-09bf-4dce-85fc-c15cd7372de0%40www.fastmail.com.
