I have two releases that I consider high-priority: github-branch-source and github-api .
Users have been able to rollback to the previous release to unblock themselves, but people who cannot rollback (new installations) remain blocked. On Friday, June 12, 2020 at 10:05:33 AM UTC-7, Oleg Nenashev wrote: > > Dear all, > > June 12 update: > > - We continue to work on the accounts migration and will share the > next update on Monday > - Jenkins releases are still blocked. If there are any emergency > releases you need to perform, please reply in this thread. > > Best regards, > Oleg Nenashev > > On Friday, June 12, 2020 at 6:00:32 PM UTC+2, Oleg Nenashev wrote: >> >> Hi Dave, >> >> This is an email from the *Step 2. We’ll reset every user password from >> the LDAP database*. This one includes a temporary password, and we >> expect users to change it after they login into the system. >> >> For those who wonder: Yes, the temporary password is sent in plain text >> as mentioned above. This is how our current password reset system is >> designed. As other projects, we have a decent amount of technical debt in >> our infrastructure which we gradually resolve. I have already added >> changing the account password reset flow to the outage retrospective list, >> an we will be reviewing what to do there after the outage is fully >> resolved. Apart from fixing it, migrating to a 3rd-party identity service >> is on the table for me (Linux Foundation or GitHub). If anyone is >> interested to participate and to improve the project, the Jenkins >> infrastructure team <https://www.jenkins.io/projects/infrastructure/> is >> always looking for more contributors! >> >> If anyone has concerns about such method and wants to use alternate >> channels for encrypted password transfer, please send us a message through >> the Jenkins Infrastructure mailing list from your email registered in >> Jenkins. In this email please provide your public GPG key so that we can >> reset a password again in a secure way. >> >> Best regards, >> Oleg >> >> On Friday, June 12, 2020 at 5:07:27 PM UTC+2, Dave Pedu wrote: >>> >>> Hello, >>> >>> I have received an email linking to this thread. However, it contains a >>> plaintext password for my account, despite this: >>> >>> > There will be no temporary password in these emails, but there will >>> be information pointing to this thread. >>> >>> Is this email legitimate or am I being phished? Screenshot attached. >>> >>> Thanks, >>> Dave >>> >>> On Thursday, June 11, 2020 at 3:07:01 AM UTC-7, Olblak wrote: >>>> >>>> Dear all, >>>> >>>> We are ready to proceed with restoration of the Jenkins account >>>> database. Today we are going to restore user LDAP accounts that were >>>> created since the First of February 2020 based on the data from Jenkins >>>> Jira and the repository Permission Manager metadata data. We will also >>>> reset passwords for all users registered in the database. >>>> >>>> Step 1. All users who lost their account will receive an email saying >>>> that their accounts were re-created. There will be no temporary password >>>> in >>>> these emails, but there will be information pointing to this thread. >>>> >>>> Step 2. We’ll reset every user password from the LDAP database, it is >>>> more than 100 000 users. Once done, you’ll receive an email telling you >>>> that your password was reset with a reason containing a link to this mail >>>> thread. >>>> >>>> Step 3. We will delete accounts of users who requested such deletion >>>> between February and June 2020. These users were restored from the backup, >>>> so we have to delete them again.The list of users is based on Jira tickets >>>> and private messages to the Jenkins Infra officer. If for some reason you >>>> notice that your account still exists, feel free to raise a ticket in >>>> Jenkins >>>> Jira <https://issues.jenkins-ci.org/> (project=INFRA, component=account >>>> ). >>>> >>>> Please do not hesitate to contact us using the #jenkins-infra channel >>>> on Freenode IRC or the Jenkins Infrastructure mailing list if you have any >>>> questions or suggestions. If you see a security issue related to the >>>> accounts, please follow the vulnerability reporting guidelines >>>> <https://www.jenkins.io/security/#reporting-vulnerabilities>. >>>> >>>> Best regards, >>>> >>>> Olivier Vernin && Jenkins Infrastructure Team >>>> >>>> >>>> On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote: >>>>> >>>>> Dear all, >>>>> >>>>> As you may have noticed, the release artifact uploads are currently >>>>> blocked in the Jenkins Artifactory instances ( >>>>> https://repo.jenkins-ci.org/). We are doing a security investigation >>>>> due to a partial user database loss on June 02. Today we blocked releases >>>>> to the Jenkins artifactory, and there also was a temporary outage of the >>>>> Artifactory downloads which was a collateral damage of the temporary >>>>> permissions. You can find more details about it in this Jenkins Infra >>>>> Thread >>>>> <https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE> >>>>> and in this Dev List thread >>>>> <https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ> >>>>> . >>>>> >>>>> Current status: >>>>> >>>>> - >>>>> >>>>> Downloads are restored for all artifacts on >>>>> https://repo.jenkins-ci.org/, Jenkins core historical releases, >>>>> Remoting library and Windows Service Wrapper which were among ones >>>>> reported >>>>> by Jenkins users. >>>>> - >>>>> >>>>> Uploads: Jenkins artifact uploads are blocked for the most of >>>>> Jenkins plugin maintainers and contributors. It affects releases of >>>>> Jenkins >>>>> plugins, Jenkins core and modules, developer tools and all libraries >>>>> hosted >>>>> on https://repo.jenkins-ci.org/. Incremental and Snapshot >>>>> deployments are not affected. >>>>> >>>>> >>>>> Quick summary: >>>>> >>>>> - >>>>> >>>>> Jun 02 - There was a Kubernetes Cluster outage on June 02. During >>>>> this outage we had to rebuild the cluster from scratch to get some >>>>> services >>>>> working again. >>>>> - >>>>> >>>>> Jun 02 - After the recovery we lost three months of LDAP changes. >>>>> It has happened due to the broken backup of the LDAP database. >>>>> - >>>>> >>>>> Jun 02 - We identified a number of potential security risks which >>>>> may be caused by the LDAP outage. Account overtake and malicious >>>>> upload was >>>>> one of the identified risks. FTR this issue is tracked as >>>>> SECURITY-1895 as >>>>> a follow-up to these discussions. Only the Security team members have >>>>> access to it, so I am not sharing a link here. >>>>> - >>>>> >>>>> Jun 09 - After the security risk was independently reported in >>>>> public by a plugin maintainer in the dev list thread >>>>> <https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg>, we >>>>> decided to block uploads of release artifacts to the Jenkins >>>>> Artifactory >>>>> instance. >>>>> - >>>>> >>>>> Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked >>>>> (plugins, Jenkins core and modules, developer tools, etc.). Downloads >>>>> of >>>>> some binaries were also blocked as an unexpected collateral damage. >>>>> Jenkins >>>>> core historical releases, Remoting library and Windows Service Wrapper >>>>> are >>>>> among the affected binaries >>>>> - >>>>> >>>>> Jun 09, 10AM UTC - We finished reviews of all artifact releases to >>>>> https://repo.jenkins-ci.org/, which happened between the infra >>>>> outage on June 02 and the blockage of the releases. There are no >>>>> maliciously uploaded artifacts. Note that the common plugin release >>>>> flow >>>>> requires access to GitHub in order to push the release commits, so a >>>>> malicious attacker would need to overtake both Jenkins and GitHub >>>>> accounts >>>>> of a single user to submit a legitimately-looking release. >>>>> - >>>>> >>>>> Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch >>>>> >>>>> <https://github.com/jenkins-infra/repository-permissions-updater/pull/1569> >>>>> >>>>> in the Repository Permission Updater was applied to prevent uploads. >>>>> Artifact uploads are still blocking >>>>> - >>>>> >>>>> Jun 09, 2PM UTC, based on repo.jenkins-ci.org and >>>>> issues.jenkins-ci.org data, we restored maintainers accounts. >>>>> >>>>> >>>>> Our next steps would be to communicate the issue to all maintainers >>>>> and contributors who might have been affected by the LDAP history loss. >>>>> We >>>>> will likely need to perform additional user verification steps for plugin >>>>> maintainers to ensure that there are no contributors affected by the >>>>> issues. Today at 3:30PM UTC we will also have a Jenkins >>>>> Infrastructure team meeting where this issue will be discussed in more >>>>> detail. This is a public meeting, and everyone is welcome to join. >>>>> Calendar >>>>> link >>>>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com> >>>>> >>>>> Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and >>>>> Security team members who contributed to this investigation. >>>>> >>>>> Best regards, >>>>> >>>>> Oleg Nenashev >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ab94c691-ab74-4eaa-a166-c2e16f8e2b28o%40googlegroups.com.
