Dear all, June 12 update:
- We continue to work on the accounts migration and will share the next update on Monday - Jenkins releases are still blocked. If there are any emergency releases you need to perform, please reply in this thread. Best regards, Oleg Nenashev On Friday, June 12, 2020 at 6:00:32 PM UTC+2, Oleg Nenashev wrote: > > Hi Dave, > > This is an email from the *Step 2. We’ll reset every user password from > the LDAP database*. This one includes a temporary password, and we expect > users to change it after they login into the system. > > For those who wonder: Yes, the temporary password is sent in plain text as > mentioned above. This is how our current password reset system is designed. > As other projects, we have a decent amount of technical debt in our > infrastructure which we gradually resolve. I have already added changing > the account password reset flow to the outage retrospective list, an we > will be reviewing what to do there after the outage is fully resolved. > Apart from fixing it, migrating to a 3rd-party identity service is on the > table for me (Linux Foundation or GitHub). If anyone is interested to > participate and to improve the project, the Jenkins infrastructure team > <https://www.jenkins.io/projects/infrastructure/> is always looking for > more contributors! > > If anyone has concerns about such method and wants to use alternate > channels for encrypted password transfer, please send us a message through > the Jenkins Infrastructure mailing list from your email registered in > Jenkins. In this email please provide your public GPG key so that we can > reset a password again in a secure way. > > Best regards, > Oleg > > On Friday, June 12, 2020 at 5:07:27 PM UTC+2, Dave Pedu wrote: >> >> Hello, >> >> I have received an email linking to this thread. However, it contains a >> plaintext password for my account, despite this: >> >> > There will be no temporary password in these emails, but there will be >> information pointing to this thread. >> >> Is this email legitimate or am I being phished? Screenshot attached. >> >> Thanks, >> Dave >> >> On Thursday, June 11, 2020 at 3:07:01 AM UTC-7, Olblak wrote: >>> >>> Dear all, >>> >>> We are ready to proceed with restoration of the Jenkins account >>> database. Today we are going to restore user LDAP accounts that were >>> created since the First of February 2020 based on the data from Jenkins >>> Jira and the repository Permission Manager metadata data. We will also >>> reset passwords for all users registered in the database. >>> >>> Step 1. All users who lost their account will receive an email saying >>> that their accounts were re-created. There will be no temporary password in >>> these emails, but there will be information pointing to this thread. >>> >>> Step 2. We’ll reset every user password from the LDAP database, it is >>> more than 100 000 users. Once done, you’ll receive an email telling you >>> that your password was reset with a reason containing a link to this mail >>> thread. >>> >>> Step 3. We will delete accounts of users who requested such deletion >>> between February and June 2020. These users were restored from the backup, >>> so we have to delete them again.The list of users is based on Jira tickets >>> and private messages to the Jenkins Infra officer. If for some reason you >>> notice that your account still exists, feel free to raise a ticket in >>> Jenkins >>> Jira <https://issues.jenkins-ci.org/> (project=INFRA, component=account >>> ). >>> >>> Please do not hesitate to contact us using the #jenkins-infra channel on >>> Freenode IRC or the Jenkins Infrastructure mailing list if you have any >>> questions or suggestions. If you see a security issue related to the >>> accounts, please follow the vulnerability reporting guidelines >>> <https://www.jenkins.io/security/#reporting-vulnerabilities>. >>> >>> Best regards, >>> >>> Olivier Vernin && Jenkins Infrastructure Team >>> >>> >>> On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote: >>>> >>>> Dear all, >>>> >>>> As you may have noticed, the release artifact uploads are currently >>>> blocked in the Jenkins Artifactory instances ( >>>> https://repo.jenkins-ci.org/). We are doing a security investigation >>>> due to a partial user database loss on June 02. Today we blocked releases >>>> to the Jenkins artifactory, and there also was a temporary outage of the >>>> Artifactory downloads which was a collateral damage of the temporary >>>> permissions. You can find more details about it in this Jenkins Infra >>>> Thread >>>> <https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE> >>>> and in this Dev List thread >>>> <https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ> >>>> . >>>> >>>> Current status: >>>> >>>> - >>>> >>>> Downloads are restored for all artifacts on >>>> https://repo.jenkins-ci.org/, Jenkins core historical releases, >>>> Remoting library and Windows Service Wrapper which were among ones >>>> reported >>>> by Jenkins users. >>>> - >>>> >>>> Uploads: Jenkins artifact uploads are blocked for the most of >>>> Jenkins plugin maintainers and contributors. It affects releases of >>>> Jenkins >>>> plugins, Jenkins core and modules, developer tools and all libraries >>>> hosted >>>> on https://repo.jenkins-ci.org/. Incremental and Snapshot >>>> deployments are not affected. >>>> >>>> >>>> Quick summary: >>>> >>>> - >>>> >>>> Jun 02 - There was a Kubernetes Cluster outage on June 02. During >>>> this outage we had to rebuild the cluster from scratch to get some >>>> services >>>> working again. >>>> - >>>> >>>> Jun 02 - After the recovery we lost three months of LDAP changes. >>>> It has happened due to the broken backup of the LDAP database. >>>> - >>>> >>>> Jun 02 - We identified a number of potential security risks which >>>> may be caused by the LDAP outage. Account overtake and malicious upload >>>> was >>>> one of the identified risks. FTR this issue is tracked as SECURITY-1895 >>>> as >>>> a follow-up to these discussions. Only the Security team members have >>>> access to it, so I am not sharing a link here. >>>> - >>>> >>>> Jun 09 - After the security risk was independently reported in >>>> public by a plugin maintainer in the dev list thread >>>> <https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg>, we >>>> decided to block uploads of release artifacts to the Jenkins >>>> Artifactory >>>> instance. >>>> - >>>> >>>> Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked >>>> (plugins, Jenkins core and modules, developer tools, etc.). Downloads >>>> of >>>> some binaries were also blocked as an unexpected collateral damage. >>>> Jenkins >>>> core historical releases, Remoting library and Windows Service Wrapper >>>> are >>>> among the affected binaries >>>> - >>>> >>>> Jun 09, 10AM UTC - We finished reviews of all artifact releases to >>>> https://repo.jenkins-ci.org/, which happened between the infra >>>> outage on June 02 and the blockage of the releases. There are no >>>> maliciously uploaded artifacts. Note that the common plugin release >>>> flow >>>> requires access to GitHub in order to push the release commits, so a >>>> malicious attacker would need to overtake both Jenkins and GitHub >>>> accounts >>>> of a single user to submit a legitimately-looking release. >>>> - >>>> >>>> Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch >>>> >>>> <https://github.com/jenkins-infra/repository-permissions-updater/pull/1569> >>>> >>>> in the Repository Permission Updater was applied to prevent uploads. >>>> Artifact uploads are still blocking >>>> - >>>> >>>> Jun 09, 2PM UTC, based on repo.jenkins-ci.org and >>>> issues.jenkins-ci.org data, we restored maintainers accounts. >>>> >>>> >>>> Our next steps would be to communicate the issue to all maintainers and >>>> contributors who might have been affected by the LDAP history loss. We >>>> will >>>> likely need to perform additional user verification steps for plugin >>>> maintainers to ensure that there are no contributors affected by the >>>> issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure >>>> team meeting where this issue will be discussed in more detail. This is a >>>> public meeting, and everyone is welcome to join. Calendar link >>>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com> >>>> >>>> Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and >>>> Security team members who contributed to this investigation. >>>> >>>> Best regards, >>>> >>>> Oleg Nenashev >>>> >>>> -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/edc41cb8-6fa0-4c44-93e3-04a0db0eb0d7o%40googlegroups.com.
