Re: [arch-dev-public] GitLab switchover and SSO migration
On 5/17/20 7:22 PM, Sven-Hendrik Haase via arch-dev-public wrote: >> On a related note, will this impact projects that prefer email patch >> submissions in any way (except that they can now opt-in to GitLab too)? >> > > Yes and no: The current idea is to stop operating patchwork, the current > primary way for namcap, pacman, AUR, archiso to accept patches via > email. However, GitLab has some support for email-based collaboration > [2]. David, who recently became archiso maintainer, is fine with taking > archiso to GitLab and Allan is fine taking pacman dev over there as > well. It is my hope that the other projects who are currently primarily > relying on email patches will follow suit. It would be nice to > consolidate everything onto the same platform. We're in no hurry to shut > down patchwork right now though so we'll give everyone some time to > evaluate GitLab and if it turns out that it can't support some > much-desired development workflows, we'll re-evaluate. > > Cheers, > Sven > > [0] https://git.archlinux.org/svntogit/community.git/ > [1] https://git.archlinux.org/svntogit/packages.git/ > [2] > https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html#new-merge-request-by-email-core-only ISTR some discussion long in the past about how gitlab could "do it all", but I poked into it recently out of curiosity and the status seems to be more confusing than that. This email submission thing is imperfect, since as far as I've been able to tell it creates a per-user email address for users who already have a gitlab account, so no anonymous or ad-hoc submissions. And it is strictly one way -- i.e. it enters the gitlab silo and emails don't leave. You could, I guess, email a mailing list or the project maintainers manually, and bcc your secret email submission endpoint with its user API token address. That would, however, simply end up with the two discussions being completely siloed away from each other. I haven't been able to find discussion about gitlab cc'ing a mailing list for issue or patch discussion, or even archive purposes. There is of course email notification that doesn't invoke a mailing list, but that doesn't provide patch diffs for inline comments so it's not quite the same experience and I think you'll inevitably end up interacting with merge requests mainly via the gitlab website (leaving aside the question of "what if someone just really likes mailing lists"). -- Eli Schwartz Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] GitLab switchover and SSO migration
On 17.05.20 22:37, Lukas Fleischer via arch-dev-public wrote: > On Sun, 17 May 2020 at 14:39:25, Sven-Hendrik Haase via arch-dev-public wrote: >> As some of you know, we've been toying with two ideas for a while: >> Arch-wide centralized user management as well as using GitLab to >> consolidate some of our current services. The overall goal is manifold. >> In no particular order, the goals are to >> - make Arch more contributor friendly >> - provide more modern tools for ourselves >> - enable more automation >> - make Arch services more secure >> - make team management activities less error-prone and more streamlined > > Great! Thanks a lot for working on this! > >> After looking at various solutions, we eventually settled on Keycloak >> since it seemed like a modern, well-maintained, and secure piece of >> software. It allows us to enable logins for services via OpenID Connect >> and SAML which is likely the best coverage we could hope for. It also >> allows us to connect other social login providers such as github.com and >> gitlab.com and it supports Recaptcha, 2FA, and WebAuthn out of the box. >> The idea is to eventually transition all our online service as well as >> SSH keys to Keycloak to ease on-/offboarding and make it less error-prone. > > Is 2FA going to be opt-in? Will it be mandatory for members of the > staff? You mentioned increased security before but having a single > password also increases the attack surface in case a password is stolen. > The current idea is to make 2FA mandatory for all staff (because let's face it, we have a lot of staff and a single hacked staff account could cause A LOT of trouble). As of right now, only DevOps people are forced to use 2FA but this requirement will be expanded soon. For ease of collaboration, we're not planning to force outside contributors to use 2FA for the time being but we'll monitor that situation. >> 1) Transition as many staff-only services to Keycloak as possible. We >> looked at our current services and put up a table on the wiki that shows >> support per service [2]. >> Some of the services that we operate are deprecated or have >> functionality that is also provided by GitLab. In our current >> understanding, this concerns Flyspray, Kanboard, and Patchwork. Of >> those, Kanboard and some of the Flyspray projects will be our first >> targets to transition to GitLab. We'll continue using Flyspray for the >> time being for package bugs but will discontinue its use for all >> non-packaging bugs. The reason for this is that how we manage our bugs >> for packages is somewhat intertwined with the svn2git migration which is >> yet to be done and might dictate a different repo structure than what we >> would come up with currently and we don't want to block on this. This >> was also discussed in a recent DevOps meeting [3]. > > When saying "svn2git migration", are you referring to the already > existing svntogit repositories on git.archlinux.org or to the migration > of our main package VCS to Git? > I was talking about the latter point you mentioned: the permanent conversion of our repos from svn to git. The svntogit repos, on the other hand, can likely be easily migrated to GitLab. To those who don't know: This is a git-accessible read-only mirror for our svn package repos currently hosted at git.archlinux.org [0][1]. Maybe there's a complication I don't know about right now but I don't see why not. > Could you elaborate on how svn2git migration and Flyspray are > intertwined? I could not find any details in [3]. > We currently don't have a perfect grasp on what needs doing for svn2git seeing as that project was begun a few times and stopped just as often. I think the last idea was to have a git repo per package and then have a metadata repo gather a bunch of package metadata but I'm not sure this was ever perfectly fleshed out and I don't want to block this migration on that decision. If we _do_ go with one repo per package then we could track bugs for each package in its repo right there on GitLab and personally, I think this is what we should go with. In this way, the Flyspray migration of package bugs and the GitLab migration are intertwined since we have to know _where_ the bugs will go in the end. It does make sense to have package sources and package bugs together in the same repo. However, doing this also requires some more tooling from our side (you're not going to create 1 repos in GitLab by hand and keep them in sync if you decide to change something). Again, we don't want to block on that. Keeping Flyspray running for only package bugs but migrating the project issues will already improve the status quo. freswa had some opinions on that as well and a plan of attack. >> 2) We'd like to get rid of our own cgit instance at git.archlinux.org >> and transition our git hosting into GitLab. AUR git access will stay as >> it is due to its special shell magic. > You may already be aware of this but I'd like to clarify,
Re: [arch-dev-public] GitLab switchover and SSO migration
On Sun, 17 May 2020 at 17:38:33, Eli Schwartz via arch-dev-public wrote: > On 5/17/20 4:37 PM, Lukas Fleischer via arch-dev-public wrote: > > Email addresses of aurweb accounts have to be confirmed (and accounts > > without verified email addresses are not usable and can be filtered out > > by a simple SQL query). Old accounts have been purged in ~2014 and, to > > the best of my knowledge, there should not be any active accounts left > > that did not go through the email verification process. > > I think the other concern here was that after you've confirmed your > email address, you can modify it and it won't be re-verified. Good point. I would say that this should be considered a bug and should be fixed as soon as possible... Lukas
Re: [arch-dev-public] GitLab switchover and SSO migration
On 5/17/20 4:37 PM, Lukas Fleischer via arch-dev-public wrote: > Email addresses of aurweb accounts have to be confirmed (and accounts > without verified email addresses are not usable and can be filtered out > by a simple SQL query). Old accounts have been purged in ~2014 and, to > the best of my knowledge, there should not be any active accounts left > that did not go through the email verification process. I think the other concern here was that after you've confirmed your email address, you can modify it and it won't be re-verified. The same rule applies to flyspray since it will email you a code to complete registration, but doesn't use a verification link when you change it. The current state of the forum and the wiki do, however, require verifying changes of email address for both sites. :) -- Eli Schwartz Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] GitLab switchover and SSO migration
On Sun, 17 May 2020 at 14:39:25, Sven-Hendrik Haase via arch-dev-public wrote: > As some of you know, we've been toying with two ideas for a while: > Arch-wide centralized user management as well as using GitLab to > consolidate some of our current services. The overall goal is manifold. > In no particular order, the goals are to > - make Arch more contributor friendly > - provide more modern tools for ourselves > - enable more automation > - make Arch services more secure > - make team management activities less error-prone and more streamlined Great! Thanks a lot for working on this! > After looking at various solutions, we eventually settled on Keycloak > since it seemed like a modern, well-maintained, and secure piece of > software. It allows us to enable logins for services via OpenID Connect > and SAML which is likely the best coverage we could hope for. It also > allows us to connect other social login providers such as github.com and > gitlab.com and it supports Recaptcha, 2FA, and WebAuthn out of the box. > The idea is to eventually transition all our online service as well as > SSH keys to Keycloak to ease on-/offboarding and make it less error-prone. Is 2FA going to be opt-in? Will it be mandatory for members of the staff? You mentioned increased security before but having a single password also increases the attack surface in case a password is stolen. > 1) Transition as many staff-only services to Keycloak as possible. We > looked at our current services and put up a table on the wiki that shows > support per service [2]. > Some of the services that we operate are deprecated or have > functionality that is also provided by GitLab. In our current > understanding, this concerns Flyspray, Kanboard, and Patchwork. Of > those, Kanboard and some of the Flyspray projects will be our first > targets to transition to GitLab. We'll continue using Flyspray for the > time being for package bugs but will discontinue its use for all > non-packaging bugs. The reason for this is that how we manage our bugs > for packages is somewhat intertwined with the svn2git migration which is > yet to be done and might dictate a different repo structure than what we > would come up with currently and we don't want to block on this. This > was also discussed in a recent DevOps meeting [3]. When saying "svn2git migration", are you referring to the already existing svntogit repositories on git.archlinux.org or to the migration of our main package VCS to Git? Could you elaborate on how svn2git migration and Flyspray are intertwined? I could not find any details in [3]. > 2) We'd like to get rid of our own cgit instance at git.archlinux.org > and transition our git hosting into GitLab. AUR git access will stay as > it is due to its special shell magic. You may already be aware of this but I'd like to clarify, especially since the DevOps meeting notes [3] mention that "git.archlinux.org likely needs to be kept for svn2git and the AUR": the cgit instance for AUR packages is entirely separate from the cgit instance running git.archlinux.org --- shutting down git.archlinux.org should not impact aur.archlinux.org in any way. > We tried hard to come up with a good source of users to import into > Keycloak so that we could seed that database with a solid user base but > sadly it appears that there is no trusted source of users that we can > rely on. Potential candidates were the wiki, BBS, AUR but we ruled them > all out in their current state as none of them have always had email > verification and so we can't trust those emails to be the sole source of > truth. In order to still allow users to keep their old contributions in > cases where they can prove their identity via email, we'll build a new > small web application that allows them to connect their new Keycloak > identity to their other identities. For now, we seeded the Keycloak > database with the only known-good source of trusted emails: Staff from > Archweb. Email addresses of aurweb accounts have to be confirmed (and accounts without verified email addresses are not usable and can be filtered out by a simple SQL query). Old accounts have been purged in ~2014 and, to the best of my knowledge, there should not be any active accounts left that did not go through the email verification process. > On more goal we had is automatic github.com mirroring in some fashion. > We looked at creating a two-way github.com <-> GitLab mirror but that > setup can break easily in the case of force pushes and race conditions > and also would have us looking at both places for pull requests. It > seems simpler to us for the time being to have one-way mirroring from > our GitLab to github.com only and then allow github.com users to easily > collaborate on our GitLab via github.com social login. It's a little bit > more hassle for the users than collaborating directly on github.com but > it's a lot less hassle for us so it's perhaps the best compromise. One argument for users to
