Re: Update on Status of Gitlab Migration
On Sun, Apr 12, 2020 at 11:04 AM Johan Ouwerkerk wrote: > > On Sun, Apr 12, 2020 at 12:49 AM Johan Ouwerkerk > wrote: > > > > Yes the only reason why a cleanup script might be needed is if the > > logical path used to express the repo in dependency information > > changes at the same time. E.g. suppose a `frameworks/kf5foo` gets > > remapped to `frameworks/kf5/foo` or something like that. In that case > > unless you use the flat repository layout, kdesrc-build would try to > > clone a new `frameworks/kf5/foo` repo, leaving your old > > `frameworks/kf5foo` to consume some wasted disk space. > > > > This is obviously a poor example, but the same problem occurs if > something moves from playground to extragear. Basically if the > `projectpath` YAML key changes. I had been considering adding the Gitlab Project ID number to the YAML metadata files as a way of allowing us to track projects through their whole lifetime. > > Regards, > > - Johan Cheers, Ben
Re: Update on Status of Gitlab Migration
On Sun, Apr 12, 2020 at 12:49 AM Johan Ouwerkerk wrote: > > Yes the only reason why a cleanup script might be needed is if the > logical path used to express the repo in dependency information > changes at the same time. E.g. suppose a `frameworks/kf5foo` gets > remapped to `frameworks/kf5/foo` or something like that. In that case > unless you use the flat repository layout, kdesrc-build would try to > clone a new `frameworks/kf5/foo` repo, leaving your old > `frameworks/kf5foo` to consume some wasted disk space. > This is obviously a poor example, but the same problem occurs if something moves from playground to extragear. Basically if the `projectpath` YAML key changes. Regards, - Johan
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 11:03 PM Michael Pyne wrote: > > On Sat, Apr 11, 2020 at 09:25:11PM +0200, Johan Ouwerkerk wrote: > > > > > A cleanup script could be handy. I think kdesrc-build will > > automatically pick up new repo paths from metadata and that should > > work transparently, but the old clones may get left behind as well. > > People who use the kdesrc-build option to ignore KDE repo structure > > shouldn't be affected at all. > > (...) > > All of the kde: repositories use the kde:foo syntax, where the 'foo' > comes from the 'repopath' parameter of the sysadmin/repo-metadata YAML > files. > Yes the only reason why a cleanup script might be needed is if the logical path used to express the repo in dependency information changes at the same time. E.g. suppose a `frameworks/kf5foo` gets remapped to `frameworks/kf5/foo` or something like that. In that case unless you use the flat repository layout, kdesrc-build would try to clone a new `frameworks/kf5/foo` repo, leaving your old `frameworks/kf5foo` to consume some wasted disk space. > > We may need to do on-the-fly conversion of the kde: repo paths if they > won't be expressible as 'kde:foo' in the future, but we should have the > information needed to do this in kdesrc-build to make this happen > on-the-fly. > Yes, this should be fairly straight forward: we could do a `git remote set-url` based on what the repo metadata tells us before updating a local clone. In fact: we could build this right now and sell it as "automagically recover your upstream". :) I might try to hack something up tomorrow or monday for that. Regards, - Johan
Re: Notice of upcoming changes to the behaviour of the anongit network
On Sat, Apr 11, 2020 at 10:14:38PM +1200, Ben Cooksley wrote: > Hi all, > > As part of the preparations for the move to Gitlab, and the rewrite of > our anongit tooling, one of the things we have looked into is how the > anongit network in general operates. > > As part of this, it has been observed that the git:// protocol is > unencrypted, and thus vulnerable to intercept and manipulation by > hostile actors. > > We have therefore decided that support for the git:// protocol to > access KDE Git repositories will cease following our migration to > Gitlab. > > Going forward, all anonymous access should take place instead over > https, which is encrypted, and has the added benefit of offering > support for redirects (should those be needed) For kdesrc-build users, as Johan Ouwerkerk noted on the other Gitlab thread, kdesrc-build since January 2020 has already switched over to using https for KDE-based source repositories in anticipation of this change. Thanks to Ben and the sysadmin team for coordinating ahead of time on this, it's allowed us to have it deployed for 3 months now and we haven't heard of any breakage as a result of this change. One important note is that if you have any git repositories that you have manually checked out using KDE's anongit, you may need to either manually adjust it to use a https:// git remote (if it uses git:// now), or adjust it to use a kde: remote (as explained at https://community.kde.org/Sysadmin/GitKdeOrgManual#Let_Git_rewrite_URL_prefixes). kdesrc-build configures git to understand this "kde:" prefix by default. Regards, - Michael Pyne
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 09:25:11PM +0200, Johan Ouwerkerk wrote: > On Sat, Apr 11, 2020 at 8:39 PM Ben Cooksley wrote: > > > > Yes, the hostname git.kde.org will be fully retired as part of this > > transition. > > > > From my understanding kdesrc-build will automatically pick this up > > once we update sysadmin/repo-metadata to show the new repository > > paths. > > This is something we'll confirm with mpyne though to ensure we can > > make the cutover as smooth as possible. > > > > Just to be clear, my understanding based on reading the > `Updated/Git.pm` module is that KDE repo paths are abstracted via > ~/.gitconfig URL remapping using `insteadOf`and `pushInsteadOf`. > Currently the code manipulates the user's ~/.gitconfig to bind the > correct mappings to the `kde:` prefix (this happens even before > cloning sysadmin repos for metadata). > > So if my understanding of the code is correct, the entire switch over > is transparent provided that kdesrc-build is updated beforehand to set > the updated value for `pushInsteadOf`. We already have the same > mechanism in place in kdesrc-build for ensuring that people use > https://anongit.kde.org instead of git://anongit.kde.org when > cloning/fetching. Yeah, when Ben asked me a couple of months back about it, this was the same conclusion I reached after reviewing the existing code. But I need to note that the way this works right now is that a module is cloned via a URL such as 'kde:juk' (not kde:kde/kdemultimedia/juk!). This is transparent to how git operates when setup with ~/.gitconfig, so you won't notice it in `git remote -v`, you need to actually look at the repository's .git/config, or run `git config --local --get remote.origin.url` in the source directory, to see whether it uses a kde: URL or a full URL. Either way, once the switchover happens, then *in theory* it can be as easy as running kdesrc-build once (to update ~/.gitconfig) and from there Git will rewrite to the updated URL automatically. We could add the switchover logic before that and guard it with a date check, that way we can do some testing early. > > Depending on how things look we may also make available a script that > > will update the configuration of a repository to reflect both the > > change in hostname as well as the change in path. > > > > A cleanup script could be handy. I think kdesrc-build will > automatically pick up new repo paths from metadata and that should > work transparently, but the old clones may get left behind as well. > People who use the kdesrc-build option to ignore KDE repo structure > shouldn't be affected at all. I don't know that we'll even necessarily need a cleanup script (though it couldn't hurt). In my case, my entire source repository contains only one repository directly referencing anongit (or git.kde.org), all others are non-KDE or kde: 1 git://anongit.kde.org/scratch/ 1 git://cmake.org/ 16 git://code.qt.io/ 1 git://git.freedesktop.org/ 1 git://git.gnupg.org/ 3 git://github.com/ 23 https://code.qt.io/ 7 https://github.com/ 1 https://gitlab.com/ 344 kde: All of the kde: repositories use the kde:foo syntax, where the 'foo' comes from the 'repopath' parameter of the sysadmin/repo-metadata YAML files. We may need to do on-the-fly conversion of the kde: repo paths if they won't be expressible as 'kde:foo' in the future, but we should have the information needed to do this in kdesrc-build to make this happen on-the-fly. Regards, - Michael Pyne
Re: Update on Status of Gitlab Migration
El sáb., 11 de abr. de 2020 a la(s) 16:26, Johan Ouwerkerk (jm.ouwerk...@gmail.com) escribió: > > On Sat, Apr 11, 2020 at 8:39 PM Ben Cooksley wrote: > > > > Yes, the hostname git.kde.org will be fully retired as part of this > > transition. > > > > From my understanding kdesrc-build will automatically pick this up > > once we update sysadmin/repo-metadata to show the new repository > > paths. > > This is something we'll confirm with mpyne though to ensure we can > > make the cutover as smooth as possible. > > > > Just to be clear, my understanding based on reading the > `Updated/Git.pm` module is that KDE repo paths are abstracted via > ~/.gitconfig URL remapping using `insteadOf`and `pushInsteadOf`. > Currently the code manipulates the user's ~/.gitconfig to bind the > correct mappings to the `kde:` prefix (this happens even before > cloning sysadmin repos for metadata). > > So if my understanding of the code is correct, the entire switch over > is transparent provided that kdesrc-build is updated beforehand to set > the updated value for `pushInsteadOf`. We already have the same > mechanism in place in kdesrc-build for ensuring that people use > https://anongit.kde.org instead of git://anongit.kde.org when > cloning/fetching. Changing .gitconfig won't be enough, per-repo changes will still be needed (although kdesrc-build could be updated to do those for you too). Currently if a project moves to gitlab, you need to change g...@git.kde.org:kdenlive to g...@invent.kde.org:kde/kdenlive (note the kde/ addition). If that was all, in principle you could use insteadOf to map kde: to "g...@invent.kde.org:kde/". But depending on future discussions about structure, it's possible that eg. kcoreaddons will end up moving to frameworks/kcoreaddons rather than kde/kcoreaddons. -- Nicolás
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 8:39 PM Ben Cooksley wrote: > > Yes, the hostname git.kde.org will be fully retired as part of this > transition. > > From my understanding kdesrc-build will automatically pick this up > once we update sysadmin/repo-metadata to show the new repository > paths. > This is something we'll confirm with mpyne though to ensure we can > make the cutover as smooth as possible. > Just to be clear, my understanding based on reading the `Updated/Git.pm` module is that KDE repo paths are abstracted via ~/.gitconfig URL remapping using `insteadOf`and `pushInsteadOf`. Currently the code manipulates the user's ~/.gitconfig to bind the correct mappings to the `kde:` prefix (this happens even before cloning sysadmin repos for metadata). So if my understanding of the code is correct, the entire switch over is transparent provided that kdesrc-build is updated beforehand to set the updated value for `pushInsteadOf`. We already have the same mechanism in place in kdesrc-build for ensuring that people use https://anongit.kde.org instead of git://anongit.kde.org when cloning/fetching. > > Depending on how things look we may also make available a script that > will update the configuration of a repository to reflect both the > change in hostname as well as the change in path. > A cleanup script could be handy. I think kdesrc-build will automatically pick up new repo paths from metadata and that should work transparently, but the old clones may get left behind as well. People who use the kdesrc-build option to ignore KDE repo structure shouldn't be affected at all. Regards, - Johan
Re: Update on Status of Gitlab Migration
El 11 abr. 2020, a la(s) 08:31, Johan Ouwerkerk escribió: > > On Sat, Apr 11, 2020 at 11:36 AM Ben Cooksley wrote: >> Should anyone have any questions on the above, please let us know. > > Does the migration also mean that `git.kde.org` push URL will be > retired and would need to be remapped to `invent.kde.org`? > > In that case, it would be good to have a grace period after the > initial migration to Gitlab so kdesrc-build (etc.) could be updated > before the cut off date to perform this migration automatically for > the user. I expect such a grace period would not need to last very > long because the feature would be trivial to implement. How would it work during the "grace period"? Keeping an outdated read-only mirror on the old URL? I have done some research into redirecting or remapping from the old URL to the new one so we can keep it working for a longer period of time, and it's harder than it seems... It can be done but I need to be convinced that it's actually necessary / worth the effort. -- Nicolás KDE Sysadmin Team
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 11:31 PM Johan Ouwerkerk wrote: > > On Sat, Apr 11, 2020 at 11:36 AM Ben Cooksley wrote: > > > > Should anyone have any questions on the above, please let us know. > > > > Does the migration also mean that `git.kde.org` push URL will be > retired and would need to be remapped to `invent.kde.org`? Yes, the hostname git.kde.org will be fully retired as part of this transition. >From my understanding kdesrc-build will automatically pick this up once we update sysadmin/repo-metadata to show the new repository paths. This is something we'll confirm with mpyne though to ensure we can make the cutover as smooth as possible. Depending on how things look we may also make available a script that will update the configuration of a repository to reflect both the change in hostname as well as the change in path. > > In that case, it would be good to have a grace period after the > initial migration to Gitlab so kdesrc-build (etc.) could be updated > before the cut off date to perform this migration automatically for > the user. I expect such a grace period would not need to last very > long because the feature would be trivial to implement. > > Regards, > > - Johan Cheers, Ben
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 11:36 AM Ben Cooksley wrote: > > Should anyone have any questions on the above, please let us know. > Does the migration also mean that `git.kde.org` push URL will be retired and would need to be remapped to `invent.kde.org`? In that case, it would be good to have a grace period after the initial migration to Gitlab so kdesrc-build (etc.) could be updated before the cut off date to perform this migration automatically for the user. I expect such a grace period would not need to last very long because the feature would be trivial to implement. Regards, - Johan
Re: Update on Status of Gitlab Migration
On Sat, Apr 11, 2020 at 11:36 AM Ben Cooksley wrote: > > To help assist with this, it would be appreciated if all holders of a > KDE Developer account could please login on our Gitlab instance (at > https://invent.kde.org/) and ensure they are listed as a 'Developer' > of the KDE group. You can do this by checking the 'Groups' tab of your > Profile on Gitlab. > This is a more direct URL that goes straight to the members page of the KDE group: https://invent.kde.org/groups/kde/-/group_members If you are logged in on invent.kde.org you should be able to use the search box to find yourself (by identity.kde.org name, in my case). Regards, - Johan
Notice of upcoming changes to the behaviour of the anongit network
Hi all, As part of the preparations for the move to Gitlab, and the rewrite of our anongit tooling, one of the things we have looked into is how the anongit network in general operates. As part of this, it has been observed that the git:// protocol is unencrypted, and thus vulnerable to intercept and manipulation by hostile actors. We have therefore decided that support for the git:// protocol to access KDE Git repositories will cease following our migration to Gitlab. Going forward, all anonymous access should take place instead over https, which is encrypted, and has the added benefit of offering support for redirects (should those be needed) Should anyone have any questions regarding this, please let us know. Thanks, Ben Cooksley KDE Sysadmin
Re: Looking for training topics for Akademy
Dnia niedziela, 5 kwietnia 2020 23:42:03 CEST Adam Szopa pisze: > Hi, > I've created a task on Phabricator (https://phabricator.kde.org/T12924) with > a example list of topics for training's that could happen during the next > Akademy. > Please check it out and add any topics that you feel would be a good fit for > the community, in both technical and non-technical categories. > Keep in mind that the list is just a suggestion, so there is no guarantee > that a given topic will have a training session during Akademy. > On the other hand, popular training topics that won't fit into the Akademy > schedule cold be conducted on a different date. > I'll leave the ticket open for adding your proposals for about a week, after > that I'll switch to estimating the popularity, viability etc of those on > the list. > > - Adam Hi, Just a quick reminder, that you can still propose topics that you'd like to be covered by training's at Akademy here: https://phabricator.kde.org/T12924 . - Adam
Re: Update on Status of Gitlab Migration
Hi, I just want to express my big thanks for all the work that went into this! Thanks to all people involved in making this happen and maintaining/improving all our other infrastructure. Besides, happy Easter and become/stay healthy! Greetings Christoph On 2020-04-11 11:35, Ben Cooksley wrote: Good morning Community, I'm pleased to report that this week we reached a major milestone, with all the necessary technical components now being in place on our side for our migration to Gitlab to take place. This includes the replacement of the tooling that supports the anongit network, as well as implementation of a system to sync changes from Identity to Gitlab in real time. As part of this, the anongit network following the migration to Gitlab should pick up new repositories, along with changes to default branches, project descriptions, repository names and their path, and the deletion of repositories within 10 seconds of the change taking place on Gitlab (effectively making it real time) We are currently looking to confirm how repositories will be organised on Gitlab, after which we should be able to confirm how and when we perform the migration. Please note that only code hosting and code review will be transitioning at this time - CI and tasks will follow later on. Existing reviews on Phabricator will not be imported to Gitlab as part of this process and will need to be closed out on Phabricator. To help assist with this, it would be appreciated if all holders of a KDE Developer account could please login on our Gitlab instance (at https://invent.kde.org/) and ensure they are listed as a 'Developer' of the KDE group. You can do this by checking the 'Groups' tab of your Profile on Gitlab. Should you not be listed as a Developer of the group, and you have previously logged into Gitlab, please visit KDE Identity and make a change to your details there, which will trigger a sync of your account to Gitlab. Should anyone have any questions on the above, please let us know. Thanks, Ben Cooksley KDE Sysadmin -- Ignorance is bliss... https://cullmann.io | https://kate-editor.org
Update on Status of Gitlab Migration
Good morning Community, I'm pleased to report that this week we reached a major milestone, with all the necessary technical components now being in place on our side for our migration to Gitlab to take place. This includes the replacement of the tooling that supports the anongit network, as well as implementation of a system to sync changes from Identity to Gitlab in real time. As part of this, the anongit network following the migration to Gitlab should pick up new repositories, along with changes to default branches, project descriptions, repository names and their path, and the deletion of repositories within 10 seconds of the change taking place on Gitlab (effectively making it real time) We are currently looking to confirm how repositories will be organised on Gitlab, after which we should be able to confirm how and when we perform the migration. Please note that only code hosting and code review will be transitioning at this time - CI and tasks will follow later on. Existing reviews on Phabricator will not be imported to Gitlab as part of this process and will need to be closed out on Phabricator. To help assist with this, it would be appreciated if all holders of a KDE Developer account could please login on our Gitlab instance (at https://invent.kde.org/) and ensure they are listed as a 'Developer' of the KDE group. You can do this by checking the 'Groups' tab of your Profile on Gitlab. Should you not be listed as a Developer of the group, and you have previously logged into Gitlab, please visit KDE Identity and make a change to your details there, which will trigger a sync of your account to Gitlab. Should anyone have any questions on the above, please let us know. Thanks, Ben Cooksley KDE Sysadmin
