Re: Information regarding upcoming Gitlab Migration
Hello Johan, On Tue, Apr 28, 2020 at 06:18:14PM +0200, Johan Ouwerkerk wrote: > I would like to propose that the sysadmin team pick the layout that is > easiest to *implement*, which I suspect is also the most like what we > have now, and that we live with that. Unless there is a pressing need, > like real hard data that we're losing contributors because they can't > find our repos or a sysadmin burden that would be excessively higher > than otherwise... let's keep things as simple and straightforward for > syadmin and tooling maintainers to implement during the Gitlab > migration. > > Let's worry about the perfect project layout once we've identified the > need. That way it's a lot easier to garner consensus for a practical > solution that isn't just a wishlist of all the features we would like > but which Gitlab may or may not have and which tooling is probably > completely unprepared for. Going with flat namespace right now and then making whole thing non-flat means double amount of work for sysadmins, once to migrate everything on invent, and then later to their final home. It is equal amount of work for contributors, once to migrate to invent URL and then migrate to new grouping. If setups needs changing, I'd rather change it now at once then later. > By all means disregard this stuff if the pressing need has already > been identified and I've either neglected to read e-mails properly or > wasn't aware for some other, possibly historical, reason. We have gotten a request for namespacing from projects on multiple occassion, in cgit our workaround has always been that we prefix the repo name with namespace- (i.e wikitolearn-courses-backend). While this works out with our current workflow, it is not really optimal. I've also mentioned various new contributor focused requirements which lead us to this proposal for structuring in previous emails. -- Bhushan Shah http://blog.bshah.in IRC Nick : bshah on Freenode GPG key fingerprint : 0AAC 775B B643 7A8D 9AF7 A3AC FE07 8411 7FBC E11D signature.asc Description: PGP signature
Re: Information regarding upcoming Gitlab Migration
On Monday, 27 April 2020 21:25:09 BST Albert Astals Cid wrote: > El dilluns, 27 d’abril de 2020, a les 13:58:02 CEST, Bhushan Shah va escriure: > > In part I am mostly re-iterating what Ben already mentioned in different > > messages. > > > > On Mon, Apr 27, 2020 at 12:38:42PM +0200, Aleix Pol wrote: > > > Does this mean that to clone it we'll have to go "git clone > > > kde:games/knetwalk" or something along the lines? > > > > Yes > > > > [Rest of message is with sysadmin hat off and as a developer] > > > > > If that's the case I'd much prefer if we didn't do this, at the moment > > > it's already uncomfortable for me to remember the URL for some of the > > > repos (e.g. is it sysadmin/ or not?), this will only increase the > > > problem and I personally don't see the advantage. > > > > I do agree that it maybe small inconvience, but let's be honest, most of > > us have been using kdesrc-build or some kind of automated tooling for > > building everything, apart from very rare case we never have to manually > > clone any of KDE repository, at least it is true for me personally. I am > > not sure about others. > > Please let's refrain from saying things like "most of us have been using > kdesrc-build" when you don't have any data to back that up. > > Cheers, > Albert Just adding my "i don't use kdesrc-build, and git clone kde:x everything myself" voice, here. Now, if a simple(ish) script can be created to make something akin to the kde: rewriting work, even if what it really does is to search gitlab and create a clone with the appropriate command, i could deal with that, but having the ability to simply ask for the project name is more than a little useful. -- ..dan / leinir.. http://leinir.dk/
Re: Information regarding upcoming Gitlab Migration
On Tue, Apr 28, 2020 at 1:09 PM Adriaan de Groot wrote: > > One actor is "tooling", as Albert has pointed out. Whatever the resulting > structure is, it needs to be communicated to tool authors on time for tools to > be updated, released, and rolled out for use. Tools mentioned so far: > - kdesrc-build > - i18n / scripty > - release scripts > The tools don't have An Opinion regarding the layout, they just need to be > updated. > To some extent, yes, but some assumptions are baked in at a quite fundamental level for e.g. kdesrc-build (if I understand the Perl correctly): - Uniqueness of project names. Cannot have two "documentation" projects being advertised through kde-build-metadata - There must be a single shared root prefix for all projects. This prefix may be the empty string, but the point is that repo paths advertised in kde-build-metadata must work from this single root prefix Any solution that wishes to preserve tooling must take these fundamental constraints as given or else risk that you cannot move until the whole world is fixed first. I'm obviously biased as someone who invested some time and effort in kdesrc-build, but the busfactor is a real concern here. E.g.: Michael clearly just hasn't had the time lately to spend on kdesrc-build and I don't think it's such a great idea for me to merge in major changes without him reviewing it first. > > A tool-like actor that I don't think has been mentioned so far is "existing > checkouts". I have a src/kde with all the bits I've looked at "recently". > There may even be some SVN checkouts there -- I'm willing to forget about > those. Surprising and annoying me every time I update those sometime in the > future is not good, but it's only going to annoy me once (per repo, so at most > 143 times for my clones). > I think that this is a bit of a lost cause. I think there's going to be some level of annoyance all around until the pain has finally died away because everybody made the upgrade. We can have a script to help with that but that will inherently assume quite a bit about how you set up your projects manually to begin with and it will still piss off a lot of people because we break their workflow no matter what. The main problem is not technical here, the main problem is that people who use that manual git clone workflow do so because they don't want to use the layers of tooling. So adding layers of tooling to "fix" the manual work flow is defeating the point, and therefore the pain is here to stay I think. At a fundamental the only way to avoid the pain is for the URLs to continue to work somehow, but I gather that is a completely unfeasible solution in terms of not driving sysadmin crazy. A fairly big one is that the fingerprint of the host's SSH key will be different. > > I'd be *vaguely* worried about existing crontabs and automatic jobs that we've > forgotten about, or which others have forgotten about. Those aren't fixable in > the face of any changes to repositories, anyway. > Yep. And I think for the exact same reason why a porting/transitioning script is probably not going help all that much -- too much diff in setup/config/versions/tools on the other end to deal with in a reasonable script. It's the main source of complexity in a tool like kdesrc-build and that has lot's of Perl to hammer systems into submission... :) > > Turning to human actors, who are the more interesting ones, > I think we're in a situation that this migration is going to cause pain for the human element no matter what. I also think that ultimately from a technical pov it mostly doesn't really matter. As someone else already mentioned: the Internet has search engines for that. What exact flavour of labeling/grouping/hierarchy we use, I think it is mostly bikeshedding: I would predict that at first this is going to be annoying for many if not nearly all contributors. After a while we've all made the upgrade (just a matter of tweaking your ~/.gitconfig & git remote set-url per repo) and then it's business as usual. So yes this is about the social aspect. Personally I think the pain is unavoidable for the individual contributor, but what seems to be getting lost in this discussion is that there *is* also an amount of pain that can be avoided. We can avoid pain for the sysadmin team by choosing whatever is easiest to maintain long term here. We can avoid pain for the maintainers of tooling by choosing whatever fits the tooling most easily. In particular we can avoid choosing an option that imposes additional parallel sysadmin work indefinitely, like e.g. dedicated mirror services to redirect stuff. I would like to propose that the sysadmin team pick the layout that is easiest to *implement*, which I suspect is also the most like what we have now, and that we live with that. Unless there is a pressing need, like real hard data that we're losing contributors because they can't find our repos or a sysadmin burden that would be excessively higher than
Re: Information regarding upcoming Gitlab Migration
Hi Adriaan, On Tue, Apr 28, 2020 at 01:08:33PM +0200, Adriaan de Groot wrote: > A tool-like actor that I don't think has been mentioned so far is "existing > checkouts". I have a src/kde with all the bits I've looked at "recently". > There may even be some SVN checkouts there -- I'm willing to forget about > those. Surprising and annoying me every time I update those sometime in the > future is not good, but it's only going to annoy me once (per repo, so at > most > 143 times for my clones). We have a plan to provide a script which can change remotes in existing clones. (It would take a top-level directory path for all your clones). > Perhaps it's the "old school", but kdesrc-build doesn't do anything for me. > I'm intermittently interested in the source of some random part of KDE -- > generally because it's mentioned on IRC -- and then I need that source where > I > can look at it. Whether it's 'git clone' or 'kdesrc-build --just-the-source- > please' doesn't matter much. > > If there's any compiling to be done, the less magic there is between me and > the compile, the better. > > So, yeah: `git clone kde:x` all the way, but I'm also not really invested in > the structure of the label x, or the precise configuration of kde:. > > > > Now, if a simple(ish) script can be created to make > > something akin to the kde: rewriting work, even if what it really does is to > > search gitlab and create a clone with the appropriate command, i could deal > > with that, but having the ability to simply ask for the project name is > > more than a little useful. > > I think we shouldn't underestimate how names are a social construct, though: > the current flat structure comes after a structured SVN naming epoch. But I'd > totes +1 a search-and-redirector, especially if it means I can write `git > clone kde:peruse` and the resulting .git/config has followed the redirects > and > whatnot and ended up with `url: kdeforreal:audio/peruse` One thing to remember is, you can't really have a dynamic insteadOf URL setup. Besides, even if we had a everything under KDE namespace I'd find a setup for this weird enough. Let's assume that everything is under invent.kde.org/kde (and invent.kde.org/sysadmin and invent.kde.org/websites as it is right now). Now you add a following snippet in gitconfig, [url "https://invent.kde.org/KDE/;] insteadOf = kde: [url "g...@invent.kde.org:KDE/"] pushInsteadOf = kde: Which is slightly modified version of the existing kde: URL helper shown here, https://community.kde.org/Sysadmin/GitKdeOrgManual#Let_Git_rewrite_URL_prefixes Now this is perfectly fine for the something which is kde/ so you can do git clone kde:krita and it would happily replace 'kde:' part with 'https://invent.kde.org/KDE/' and would clone 'https://invent.kde.org/KDE/krita' But, now things get interesting when you want to clone the docs-krita-org which is in the websites/ namespace. We will have to keep this things in separate namespace for policy reasons (and same goes for sysadmin stuff). How would you clone docs-krita-org? git clone kde:../websites/krita-org ? add a separate prefix for websites and sysadmin? Let's assume that you added snippet for sysadmin and websites, it's just 8 more lines after all, [url "https://invent.kde.org/websites/;] insteadOf = websites: [url "g...@invent.kde.org:websites/"] pushInsteadOf = websites: [url "https://invent.kde.org/sysadmin/;] insteadOf = sysadmin: [url "g...@invent.kde.org:sysadmin/"] pushInsteadOf = sysadmin: Does this solve all use-cases? I believe no, it still does not support the personal repositories of users. So you also need to add 4 more lines for your own user, and 4 for each user whose repository you want to clone. What our workflow with the kde: prefix so far had been really useful I agree, and I will miss it, but with departure of gitolite which allowed repositories at top-level, we can't easily support this, and we have to accept it. Perhaps solution suggested by Ben in his email could be useful instead and in addition generic invent: snippet which does not include any namespace. [url "https://invent.kde.org/;] insteadOf = invent: [url "g...@invent.kde.org:"] pushInsteadOf = invent: > (That said, bigflatlistofrepositories.kde.org .. or maybe call it > cgit.kde.org > .. could be a particular view onto gitlab which does flattening and search, > but only if there's people around to create it and maintain it) Just to clarify, sysadmins have no plan to support or maintain the cgit instance once we migrate to Gitlab. Thanks -- Bhushan Shah http://blog.bshah.in IRC Nick : bshah on Freenode GPG key fingerprint : 0AAC 775B B643 7A8D 9AF7 A3AC FE07 8411 7FBC E11D signature.asc Description: PGP signature
Re: Information regarding upcoming Gitlab Migration
On Tue, Apr 28, 2020 at 11:09 PM Adriaan de Groot wrote: > > There are a whole bunch of considerations and use-cases being discussed at > once in this thread, and Leinir's post made me think a bit about different > actors can interact with "the collection of repositories". > > One actor is "tooling", as Albert has pointed out. Whatever the resulting > structure is, it needs to be communicated to tool authors on time for tools to > be updated, released, and rolled out for use. Tools mentioned so far: > - kdesrc-build > - i18n / scripty > - release scripts > The tools don't have An Opinion regarding the layout, they just need to be > updated. > > A tool-like actor that I don't think has been mentioned so far is "existing > checkouts". I have a src/kde with all the bits I've looked at "recently". > There may even be some SVN checkouts there -- I'm willing to forget about > those. Surprising and annoying me every time I update those sometime in the > future is not good, but it's only going to annoy me once (per repo, so at most > 143 times for my clones). > > I'd be *vaguely* worried about existing crontabs and automatic jobs that we've > forgotten about, or which others have forgotten about. Those aren't fixable in > the face of any changes to repositories, anyway. > > > Turning to human actors, who are the more interesting ones, > > On 2020 prilula d. 28id 10:38:36 CEST Dan Leinir Turthra Jensen wrote: > > On Monday, 27 April 2020 21:25:09 BST Albert Astals Cid wrote: > > > El dilluns, 27 d’abril de 2020, a les 13:58:02 CEST, Bhushan Shah va > > > > On Mon, Apr 27, 2020 at 12:38:42PM +0200, Aleix Pol wrote: > > > > > Does this mean that to clone it we'll have to go "git clone > > > > > kde:games/knetwalk" or something along the lines? > > > > > > If that's the case I'd much prefer if we didn't do this, at the moment > > > > > it's already uncomfortable for me to remember the URL for some of the > > > > > repos (e.g. is it sysadmin/ or not?), this will only increase the > > > > > problem and I personally don't see the advantage. > > > Humans come in all shapes and sizes; here's one called Aleix who likes to > remember the name of a thing, one single label. (Ironic: this particular Aleix > is also labelled "Aleix Pol" and possibly "Aleix Pol Gonzales") The question > I'd ask is "in what **context** do you need to remember the URL of a repo?" > and for that matter: "why is 'knetwalk' an obvious thing to remember, and > 'games/knetwalk' not-obvious?" > > Context here can mean many things. In this thread we've had mentioned: > - people just browsing and wanting A Big List >Here a hierarchical approach means more clicks and navigating a tree, >rather than a flat structure. > - people browsing for a known label >Using ^F in a flat list or some search field (see also Ian Wadham's >message just now) doesn't seem *that* different to me, although I've >got to give ^F the benefit of speed and simplicity. > - developers sharing a task list and reviews > > .. probably more. Some of these roles have, depending on the chosen solution, > problems that can be solved with a *technical* addition > (bigflatlistofrepositories.kde.org .. or whatever), others are going to need a > social solution. > > > > Personally, I'm with Leinir here: > > > Just adding my "i don't use kdesrc-build, and git clone kde:x everything > > myself" voice, here. > > Perhaps it's the "old school", but kdesrc-build doesn't do anything for me. > I'm intermittently interested in the source of some random part of KDE -- > generally because it's mentioned on IRC -- and then I need that source where I > can look at it. Whether it's 'git clone' or 'kdesrc-build --just-the-source- > please' doesn't matter much. > > If there's any compiling to be done, the less magic there is between me and > the compile, the better. > > So, yeah: `git clone kde:x` all the way, but I'm also not really invested in > the structure of the label x, or the precise configuration of kde:. > > > > Now, if a simple(ish) script can be created to make > > something akin to the kde: rewriting work, even if what it really does is to > > search gitlab and create a clone with the appropriate command, i could deal > > with that, but having the ability to simply ask for the project name is > > more than a little useful. > > I think we shouldn't underestimate how names are a social construct, though: > the current flat structure comes after a structured SVN naming epoch. But I'd > totes +1 a search-and-redirector, especially if it means I can write `git > clone kde:peruse` and the resulting .git/config has followed the redirects and > whatnot and ended up with `url: kdeforreal:audio/peruse` Would some form of git alias/custom command script that works similar to the following be suitable? git kde-clone skrooge That script would then search the appropriate groups (ignoring any personal repositories including forks), find the necessary repository and perform the clone
Re: Information regarding upcoming Gitlab Migration
There are a whole bunch of considerations and use-cases being discussed at once in this thread, and Leinir's post made me think a bit about different actors can interact with "the collection of repositories". One actor is "tooling", as Albert has pointed out. Whatever the resulting structure is, it needs to be communicated to tool authors on time for tools to be updated, released, and rolled out for use. Tools mentioned so far: - kdesrc-build - i18n / scripty - release scripts The tools don't have An Opinion regarding the layout, they just need to be updated. A tool-like actor that I don't think has been mentioned so far is "existing checkouts". I have a src/kde with all the bits I've looked at "recently". There may even be some SVN checkouts there -- I'm willing to forget about those. Surprising and annoying me every time I update those sometime in the future is not good, but it's only going to annoy me once (per repo, so at most 143 times for my clones). I'd be *vaguely* worried about existing crontabs and automatic jobs that we've forgotten about, or which others have forgotten about. Those aren't fixable in the face of any changes to repositories, anyway. Turning to human actors, who are the more interesting ones, On 2020 prilula d. 28id 10:38:36 CEST Dan Leinir Turthra Jensen wrote: > On Monday, 27 April 2020 21:25:09 BST Albert Astals Cid wrote: > > El dilluns, 27 d’abril de 2020, a les 13:58:02 CEST, Bhushan Shah va > > > On Mon, Apr 27, 2020 at 12:38:42PM +0200, Aleix Pol wrote: > > > > Does this mean that to clone it we'll have to go "git clone > > > > kde:games/knetwalk" or something along the lines? > > > > If that's the case I'd much prefer if we didn't do this, at the moment > > > > it's already uncomfortable for me to remember the URL for some of the > > > > repos (e.g. is it sysadmin/ or not?), this will only increase the > > > > problem and I personally don't see the advantage. Humans come in all shapes and sizes; here's one called Aleix who likes to remember the name of a thing, one single label. (Ironic: this particular Aleix is also labelled "Aleix Pol" and possibly "Aleix Pol Gonzales") The question I'd ask is "in what **context** do you need to remember the URL of a repo?" and for that matter: "why is 'knetwalk' an obvious thing to remember, and 'games/knetwalk' not-obvious?" Context here can mean many things. In this thread we've had mentioned: - people just browsing and wanting A Big List Here a hierarchical approach means more clicks and navigating a tree, rather than a flat structure. - people browsing for a known label Using ^F in a flat list or some search field (see also Ian Wadham's message just now) doesn't seem *that* different to me, although I've got to give ^F the benefit of speed and simplicity. - developers sharing a task list and reviews .. probably more. Some of these roles have, depending on the chosen solution, problems that can be solved with a *technical* addition (bigflatlistofrepositories.kde.org .. or whatever), others are going to need a social solution. Personally, I'm with Leinir here: > Just adding my "i don't use kdesrc-build, and git clone kde:x everything > myself" voice, here. Perhaps it's the "old school", but kdesrc-build doesn't do anything for me. I'm intermittently interested in the source of some random part of KDE -- generally because it's mentioned on IRC -- and then I need that source where I can look at it. Whether it's 'git clone' or 'kdesrc-build --just-the-source- please' doesn't matter much. If there's any compiling to be done, the less magic there is between me and the compile, the better. So, yeah: `git clone kde:x` all the way, but I'm also not really invested in the structure of the label x, or the precise configuration of kde:. > Now, if a simple(ish) script can be created to make > something akin to the kde: rewriting work, even if what it really does is to > search gitlab and create a clone with the appropriate command, i could deal > with that, but having the ability to simply ask for the project name is > more than a little useful. I think we shouldn't underestimate how names are a social construct, though: the current flat structure comes after a structured SVN naming epoch. But I'd totes +1 a search-and-redirector, especially if it means I can write `git clone kde:peruse` and the resulting .git/config has followed the redirects and whatnot and ended up with `url: kdeforreal:audio/peruse` (That said, bigflatlistofrepositories.kde.org .. or maybe call it cgit.kde.org .. could be a particular view onto gitlab which does flattening and search, but only if there's people around to create it and maintain it) [ade] signature.asc Description: This is a digitally signed message part.
