June 12th update:
- We are still working on the account migration
- Step 1 is completed, all users have been restored in the database
based on the data from Jenkins Jira and repository permissions updater.
- Step 2 is in progress. Tens of thousands users have already
received the password reset notifications, we had 2 batches of password
resets today. We will continue the migration tomorrow
- Step 3 - not started
- Plugin uploads are still blocked at the moment
- Tomorrow we plan to double-check the account resets for plugin
maintainers, and we will consider reenabling uploads after that
Best regards,
Oleg
On Thursday, June 11, 2020 at 12:07:01 PM UTC+2, Olblak wrote:
>
> Dear all,
>
> We are ready to proceed with restoration of the Jenkins account database.
> Today we are going to restore user LDAP accounts that were created since
> the First of February 2020 based on the data from Jenkins Jira and the
> repository Permission Manager metadata data. We will also reset passwords
> for all users registered in the database.
>
> Step 1. All users who lost their account will receive an email saying that
> their accounts were re-created. There will be no temporary password in
> these emails, but there will be information pointing to this thread.
>
> Step 2. We’ll reset every user password from the LDAP database, it is more
> than 100 000 users. Once done, you’ll receive an email telling you that
> your password was reset with a reason containing a link to this mail thread.
>
> Step 3. We will delete accounts of users who requested such deletion
> between February and June 2020. These users were restored from the backup,
> so we have to delete them again.The list of users is based on Jira tickets
> and private messages to the Jenkins Infra officer. If for some reason you
> notice that your account still exists, feel free to raise a ticket in Jenkins
> Jira <https://issues.jenkins-ci.org/> (project=INFRA, component=account).
>
> Please do not hesitate to contact us using the #jenkins-infra channel on
> Freenode IRC or the Jenkins Infrastructure mailing list if you have any
> questions or suggestions. If you see a security issue related to the
> accounts, please follow the vulnerability reporting guidelines
> <https://www.jenkins.io/security/#reporting-vulnerabilities>.
>
> Best regards,
>
> Olivier Vernin && Jenkins Infrastructure Team
>
>
> On Tuesday, 9 June 2020 17:00:25 UTC+2, Oleg Nenashev wrote:
>>
>> Dear all,
>>
>> As you may have noticed, the release artifact uploads are currently
>> blocked in the Jenkins Artifactory instances (
>> https://repo.jenkins-ci.org/). We are doing a security investigation due
>> to a partial user database loss on June 02. Today we blocked releases to
>> the Jenkins artifactory, and there also was a temporary outage of the
>> Artifactory downloads which was a collateral damage of the temporary
>> permissions. You can find more details about it in this Jenkins Infra
>> Thread
>> <https://groups.google.com/forum/#!topic/jenkins-infra/zRqdiyarLDE> and
>> in this Dev List thread
>> <https://groups.google.com/d/msg/jenkinsci-dev/juHejx8zfdg/xpySiv1_CQAJ>.
>>
>> Current status:
>>
>> -
>>
>> Downloads are restored for all artifacts on
>> https://repo.jenkins-ci.org/, Jenkins core historical releases,
>> Remoting library and Windows Service Wrapper which were among ones
>> reported
>> by Jenkins users.
>> -
>>
>> Uploads: Jenkins artifact uploads are blocked for the most of Jenkins
>> plugin maintainers and contributors. It affects releases of Jenkins
>> plugins, Jenkins core and modules, developer tools and all libraries
>> hosted
>> on https://repo.jenkins-ci.org/. Incremental and Snapshot deployments
>> are not affected.
>>
>>
>> Quick summary:
>>
>> -
>>
>> Jun 02 - There was a Kubernetes Cluster outage on June 02. During
>> this outage we had to rebuild the cluster from scratch to get some
>> services
>> working again.
>> -
>>
>> Jun 02 - After the recovery we lost three months of LDAP changes. It
>> has happened due to the broken backup of the LDAP database.
>> -
>>
>> Jun 02 - We identified a number of potential security risks which may
>> be caused by the LDAP outage. Account overtake and malicious upload was
>> one
>> of the identified risks. FTR this issue is tracked as SECURITY-1895 as a
>> follow-up to these discussions. Only the Security team members have
>> access
>> to it, so I am not sharing a link here.
>> -
>>
>> Jun 09 - After the security risk was independently reported in public
>> by a plugin maintainer in the dev list thread
>> <https://groups.google.com/g/jenkinsci-dev/c/juHejx8zfdg>, we decided
>> to block uploads of release artifacts to the Jenkins Artifactory instance.
>> -
>>
>> Jun 09, 8:50AM UTC - All uploads of release artifacts were blocked
>> (plugins, Jenkins core and modules, developer tools, etc.). Downloads of
>> some binaries were also blocked as an unexpected collateral damage.
>> Jenkins
>> core historical releases, Remoting library and Windows Service Wrapper
>> are
>> among the affected binaries
>> -
>>
>> Jun 09, 10AM UTC - We finished reviews of all artifact releases to
>> https://repo.jenkins-ci.org/, which happened between the infra outage
>> on June 02 and the blockage of the releases. There are no maliciously
>> uploaded artifacts. Note that the common plugin release flow requires
>> access to GitHub in order to push the release commits, so a malicious
>> attacker would need to overtake both Jenkins and GitHub accounts of a
>> single user to submit a legitimately-looking release.
>> -
>>
>> Jun 09, ~1PM UTC - Artifact downloads are restored, alternate patch
>>
>> <https://github.com/jenkins-infra/repository-permissions-updater/pull/1569>
>> in the Repository Permission Updater was applied to prevent uploads.
>> Artifact uploads are still blocking
>> -
>>
>> Jun 09, 2PM UTC, based on repo.jenkins-ci.org and
>> issues.jenkins-ci.org data, we restored maintainers accounts.
>>
>>
>> Our next steps would be to communicate the issue to all maintainers and
>> contributors who might have been affected by the LDAP history loss. We will
>> likely need to perform additional user verification steps for plugin
>> maintainers to ensure that there are no contributors affected by the
>> issues. Today at 3:30PM UTC we will also have a Jenkins Infrastructure
>> team meeting where this issue will be discussed in more detail. This is a
>> public meeting, and everyone is welcome to join. Calendar link
>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=dTJsaWoxN2xjZHFkajRsbmJlcWFiaXI5b2JfMjAyMDA2MDlUMTUzMDAwWiA0c3MxMmYwbXFyM3RicDF0MmZlMzY5c2xmNEBn&tmsrc=4ss12f0mqr3tbp1t2fe369slf4%40group.calendar.google.com>
>>
>> Thanks to Olivier Vernin, Daniel Beck and other Jenkins Infra and
>> Security team members who contributed to this investigation.
>>
>> Best regards,
>>
>> Oleg Nenashev
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/9f9cf846-f5b6-4906-87a7-6f2faf969c9fo%40googlegroups.com.
