Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Mon, Mar 23, 2020 at 6:53 AM Albert Astals Cid wrote: > > El diumenge, 22 de març de 2020, a les 16:12:04 CET, Ben Cooksley va escriure: > > On Mon, Mar 23, 2020 at 12:49 AM Albert Astals Cid wrote: > > > > > > El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va > > > escriure: > > > > Note however that images based upon Fedora or anything that shares > > > > it's lineage (including CentOS and it's derivates) is strictly > > > > prohibited and won't be accepted for inclusion. > > > > > > I still find this highly annoying since Fedora seems to be the only > > > distro providing an almost full stack of mingw packages so for example i > > > could add easily a mingw/windows CI to QCA using fedora but i can't > > > because, for some reason i forgot, you are very unhappy with them. > > > > > > > ... > > > > This is why Fedora and all associated derivatives are banned from our > > systems. > > Is there a way they can be ever forgiven or is your plan to ban Fedora > forever? It would be reasonably difficult for them to be forgiven for their transgressions i'm afraid, but not impossible. It would need to be started from their side however, and they would need to reform the way they operate in addition to apologise for their severe failures. > > > > > > Cheers, > > > Albert > > > > With regards to MingW, the better way to do this would be by using > > native MingW rather than cross compiling. Craft already uses MingW for > > some dependencies, and Krita's binary factory builds use it as well so > > most of the infrastructure for this is already in position. > > But that's something that requires magic and thus sysadmin to do it, a > Fedora+mingw gitlab CI is something i can do myself. Not too much magic - all the information on install locations can be found within Craft / Craftmaster. Once you have that you would just need to customise a pipeline template for Mingw builds. > > But If you're volunteering to do a Windows gitlab CI for QCA I'll take it :) Not at this stage - aside from you mentioning it here, and Krita needing to use MingW for their builds I haven't come across any other KDE project that was after/interested in MingW on Windows CI. (I'm curious though as to why MingW on Windows is relevant given the GCC on Linux coverage and MSVC on Windows that we already have) > > Cheers, > Albert Cheers, Ben > > > > > Regards, > > Ben > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > - Johan > > > > > > > > Cheers, > > > > Ben > > > > > > > > > > > > > > > > > > > > > >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
El diumenge, 22 de març de 2020, a les 16:12:04 CET, Ben Cooksley va escriure: > On Mon, Mar 23, 2020 at 12:49 AM Albert Astals Cid wrote: > > > > El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va > > escriure: > > > Note however that images based upon Fedora or anything that shares > > > it's lineage (including CentOS and it's derivates) is strictly > > > prohibited and won't be accepted for inclusion. > > > > I still find this highly annoying since Fedora seems to be the only distro > > providing an almost full stack of mingw packages so for example i could add > > easily a mingw/windows CI to QCA using fedora but i can't because, for some > > reason i forgot, you are very unhappy with them. > > > > ... > > This is why Fedora and all associated derivatives are banned from our systems. Is there a way they can be ever forgiven or is your plan to ban Fedora forever? > > > Cheers, > > Albert > > With regards to MingW, the better way to do this would be by using > native MingW rather than cross compiling. Craft already uses MingW for > some dependencies, and Krita's binary factory builds use it as well so > most of the infrastructure for this is already in position. But that's something that requires magic and thus sysadmin to do it, a Fedora+mingw gitlab CI is something i can do myself. But If you're volunteering to do a Windows gitlab CI for QCA I'll take it :) Cheers, Albert > > Regards, > Ben > > > > > > > > > > > > > > Regards, > > > > > > > > - Johan > > > > > > Cheers, > > > Ben > > > > > > > > > > > >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Mon, Mar 23, 2020 at 12:41 AM Johan Ouwerkerk wrote: > > On Sun, Mar 22, 2020 at 3:20 AM Ben Cooksley wrote: > > > > We already do have a repository of artifacts :) > > You can find the public view of this at > > https://build-artifacts.kde.org/production/ > > > > > > > We already use Virtual Machines for both FreeBSD and Windows. > > > > The main problem here is that there is no way of dynamically > > provisioning full VMs without building an entire Openstack type > > system, which is just too much overhead for the number of builders we > > have. > > > > First of all, thanks for your patience in explaining to me what must > be obvious to you. I really appreciate it. > > > > > Shouldn't this kind of experimentation be done on developers systems > > rather than the CI system? > > Please do remember that CI resources are not infinite and at times can > > be quite constrained. > > > > My main query here though is what would be achieved by allowing this > > sort of thing that the existing system does not allow for? > > (Ignoring the fact we don't cover feature branches - something which > > is in the long term pipeline) > > > > Pure experimentation should definitely be done on developers' systems. > But changes to your CI cannot be fully validated on your own machine > beforehand: you need to verify that it works on the actual CI > builders. Ideally without breaking the CI for anyone else. > > So the main gain here is that projects can improve, evaluate and > mature their CI efforts independently. Examples of things we currently > don't have but some projects might benefit from are: licensing > compliance checking (reuse), dependency scanning (e.g.: snyk) and > mutation testing (e.g.: stryker). > Sounds reasonable enough. My main concern here is people trying to replicate builds under their $distroOfChoice which doesn't really add value to the system but which does increase resource consumption. > I'm not saying that this couldn't be achieved today with a lot of help > and guidance from the sysadmins. But I think this does impose a lot of > overhead, especially while such changes are still experimental and do > not fully work yet in CI or need tweaking for prettier output or > whatever. In fact, your reply to my other question kind of indicates > as much: > > > > > > > Possibly projects would also submit custom images for consideration to > > > that registry. > > > > This isn't something that is terribly easy to do within our existing > > Jenkins setup - but is theoretically possible. It requires quite a bit > > of provisioning work on the Jenkins side. > > > > The CI scripts would also need some adapting to make this sort of > > setup efficient to operate with them if you were to use them (and you > > probably do want to use them for running tests). > > A good portion of this adapting will done as part of moving the CI > > system to Gitlab. > > Does this mean that we will be encouraged to configure `image` > settings in .gitlab-ci.yml? If so, that basically gives me what I > really wanted here. All the actual images I might ever need could be > submitted for review via MR to a repository on invent as and when I > would need them. I would prefer if we could avoid the proliferation of too many images if at all possible, however there may be scope for this where they don't overlap functionality wise with our existing images (a different Qt version does not overlap, but providing builds using a different distribution for a version of Qt already covered definitely would overlap) > > > > > Note however that images based upon Fedora or anything that shares > > it's lineage (including CentOS and it's derivates) is strictly > > prohibited and won't be accepted for inclusion. > > > > Personally I'm more inclined towards using Debian myself, but out of > interest what is the reason for banning Fedora and CentOS images? Does > this still apply when the move to Gitlab CI is completed? I've detailed this ban in my response to Albert. The ban applies to all parts of our infrastructure, including Gitlab, and is indefinite in duration. Images based upon Debian, Ubuntu and SUSE are perfectly acceptable. > > Regards, > > - Johan Cheers, Ben
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Mon, Mar 23, 2020 at 12:49 AM Albert Astals Cid wrote: > > El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va escriure: > > Note however that images based upon Fedora or anything that shares > > it's lineage (including CentOS and it's derivates) is strictly > > prohibited and won't be accepted for inclusion. > > I still find this highly annoying since Fedora seems to be the only distro > providing an almost full stack of mingw packages so for example i could add > easily a mingw/windows CI to QCA using fedora but i can't because, for some > reason i forgot, you are very unhappy with them. > I'm unhappy with them due to how they handled the complete disaster that was a significant version update to a core system library (libc I think) which they did in a stable, released distribution. As one would expect, trying to do such an update broke things, in particular it broke the BDB libraries, which are used by amongst other things, their package manager (DNF). Once this became apparent (which didn't take long) not only did they not withdraw the update to libc/glibc, they pressed ahead with a series of hotfixes to bdb and the package manager. Unfortunately these were reliant on the underlying file system supporting a particular set of POSIX specified functionality, which overlayfs (the file system used by Docker) didn't support. The end result of all of this is that it was impossible for us to generate a new CI image. While this is already bad, this was then amplified by the reaction on the Fedora side, where the DNF developers ignored our (and others) report of the issue they said was fixed still being broken. Two weeks later, and still being ignored, I complained to the Fedora Engineering Steering Committee (Fesco) concerning this, and after a few days someone asked someone else to rebuild the base Fedora Docker image - which pulled in some fixes that resolved the issue finally and let us generate an image. Following this my complaint was closed as "resolved", even though they had not put in place any safeguards to prevent this from happening again, nor had they addressed the failure of the maintainers of one of the core components of their distribution to respond to users regarding something that was totally broken. Unsurprisingly, within a few days the DNF developers pushed yet another update that broke the build of the image again. I returned to Fesco, asking for the push privileges of the DNF developers to be suspended as they had broken it yet again. This finally resulted in the DNF developers responding (funny that!), but not with an apology or anything along those lines. Instead, they engaged in a ranting attack. In this they made various errornous claims (including that we were using an out of date Fedora image, which was not the case and I had provided logs earlier which demonstrated this which it was obvious they had failed to read) and were in general extremely hostile. Following this there was some back and forth and the issue was scheduled for discussion at the next Fesco online meeting (apparently they only make decisions once a week at these meetings). This time for this meeting came and went, with no comment made on the issue. Following further investigation I found out the meeting had been cancelled due to a lack of attendence, which had not been communicated to us. It was at this point that I had finally had enough of Fedora (having previously had to deal with their internal politics over the packaging of QtWebEngine, which meant we ended up having to use Qt 5.8 rather than the Qt 5.9 which we had initially targeted as the 5.9 packaging was blocked) and dumped them for SUSE, the images that we continue to use today. For those wondering, my complaint against the DNF developers was dismissed and closed as "they were responding now" with a one line message posted through an IRC Bot about a week later when they held their next meeting. Suffice to say, I don't want anything to do with Fedora ever again. It was an extremely stressful 6 weeks or so in all, with developers chasing up when we could get some additional packages installed and us being unable to deliver because Fedora had broken things. This is why Fedora and all associated derivatives are banned from our systems. > Cheers, > Albert With regards to MingW, the better way to do this would be by using native MingW rather than cross compiling. Craft already uses MingW for some dependencies, and Krita's binary factory builds use it as well so most of the infrastructure for this is already in position. Regards, Ben > > > > > > > > > Regards, > > > > > > - Johan > > > > Cheers, > > Ben > > > > > >
CI mail filtering (was Re: Manner in which kde-gtk-config development is conducted)
On 21/03/2020 22:38, Ben Cooksley wrote: On Sun, Mar 22, 2020 at 3:08 AM David Edmundson wrote: You're absolutely right that mistakes were made and have reason to be frustrated. kde-gtk-config is now maintained by new developers. Plasma has a new influx of new people which is good to see and something we need to foster carefully. Overall these new devs are doing a super job and we want to encourage them. Ultimately there are two parties at fault: - brand new developers who didn't know the rules. KDE has a lot of rules and they're certainly not all written down in a consistent location. - the more "senior" Plasma people (me, Kai, Aleix, etc) who do know the rules, not paying due attention to something that's now under Plasma's umbrella which means that the repository is no longer eligible to form part of a KDE release module and should be moved to Playground I think this is an overreaction that punishes the wrong people. Users. The reaction is intended to force the hand of Plasma as a collective group to pay attention to the notifications from the CI system - which it delivers to the plasma-devel mailing list. I know that some people do pay attention to these, but it is evident that others do not or have active filtering in place to ensure they don't see them. The traffic of CI mail can be a bit daunting, and failure mails could simply go unnoticed, so I suggest some "reverse filtering", i.e. I've just created a filter in Thunderbird: From: CI System X-Jenkins-Results: FAILURE (in TBird you can create customise message headers and add a new one). I don't care as much about success reports, (and I haven't yet dug in the mails to create a filter to cull mails about successful builds that have some unit tests failure). I suggest that if you commit regulary to a KDE repo to make a habit of checking the results of the unit tests on the CI every now and then (jenkins has a nice web interface). Unit tests could pass locally and fail on the CI for some reason. [...] My, beginner-dev, 2 p's, -- Ahmad Samir
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va escriure: > Note however that images based upon Fedora or anything that shares > it's lineage (including CentOS and it's derivates) is strictly > prohibited and won't be accepted for inclusion. I still find this highly annoying since Fedora seems to be the only distro providing an almost full stack of mingw packages so for example i could add easily a mingw/windows CI to QCA using fedora but i can't because, for some reason i forgot, you are very unhappy with them. Cheers, Albert > > > > > Regards, > > > > - Johan > > Cheers, > Ben >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Sun, Mar 22, 2020 at 3:20 AM Ben Cooksley wrote: > > We already do have a repository of artifacts :) > You can find the public view of this at > https://build-artifacts.kde.org/production/ > > > We already use Virtual Machines for both FreeBSD and Windows. > > The main problem here is that there is no way of dynamically > provisioning full VMs without building an entire Openstack type > system, which is just too much overhead for the number of builders we > have. > First of all, thanks for your patience in explaining to me what must be obvious to you. I really appreciate it. > > Shouldn't this kind of experimentation be done on developers systems > rather than the CI system? > Please do remember that CI resources are not infinite and at times can > be quite constrained. > > My main query here though is what would be achieved by allowing this > sort of thing that the existing system does not allow for? > (Ignoring the fact we don't cover feature branches - something which > is in the long term pipeline) > Pure experimentation should definitely be done on developers' systems. But changes to your CI cannot be fully validated on your own machine beforehand: you need to verify that it works on the actual CI builders. Ideally without breaking the CI for anyone else. So the main gain here is that projects can improve, evaluate and mature their CI efforts independently. Examples of things we currently don't have but some projects might benefit from are: licensing compliance checking (reuse), dependency scanning (e.g.: snyk) and mutation testing (e.g.: stryker). I'm not saying that this couldn't be achieved today with a lot of help and guidance from the sysadmins. But I think this does impose a lot of overhead, especially while such changes are still experimental and do not fully work yet in CI or need tweaking for prettier output or whatever. In fact, your reply to my other question kind of indicates as much: > > > > Possibly projects would also submit custom images for consideration to > > that registry. > > This isn't something that is terribly easy to do within our existing > Jenkins setup - but is theoretically possible. It requires quite a bit > of provisioning work on the Jenkins side. > > The CI scripts would also need some adapting to make this sort of > setup efficient to operate with them if you were to use them (and you > probably do want to use them for running tests). > A good portion of this adapting will done as part of moving the CI > system to Gitlab. Does this mean that we will be encouraged to configure `image` settings in .gitlab-ci.yml? If so, that basically gives me what I really wanted here. All the actual images I might ever need could be submitted for review via MR to a repository on invent as and when I would need them. > > Note however that images based upon Fedora or anything that shares > it's lineage (including CentOS and it's derivates) is strictly > prohibited and won't be accepted for inclusion. > Personally I'm more inclined towards using Debian myself, but out of interest what is the reason for banning Fedora and CentOS images? Does this still apply when the move to Gitlab CI is completed? Regards, - Johan
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Sun, Mar 22, 2020 at 12:00 PM Johan Ouwerkerk wrote: > > On Sat, Mar 21, 2020 at 10:27 PM Ben Cooksley wrote: > > > > On Sun, Mar 22, 2020 at 3:27 AM Johan Ouwerkerk > > wrote: > > > > > > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley wrote: > > > > > > > > Comments welcome. Please note that simply fixing the dependency > > > > breakage in this case is not enough to resolve this - there are > > > > underlying issues which need to be addressed here. > > > > > > > > Regards, > > > > Ben Cooksley > > > > KDE Sysadmin > > > > > > I cannot comment as to whether or not this is a pattern of behaviour > > > or just a few isolated instances. From a technical perspective I feel > > > there are two (additional) underlying issues worth addressing here: > > > > > > 1. This could be prevented for the most part by having CI run before, > > > and not after the fact. I.e. prior to merging code. > > > > This will happen once we are on Gitlab. > > > > > 2. Different projects have different CI needs, and it would help if a > > > project could safely manage their CI environment "on their own" as > > > much as possible. The current system requires a lot of daunting > > > (possibly otherwise unnecessary) complexity purely to manage the fact > > > that a builder image will be used not just for one project but for > > > perhaps the whole of KDE. > > > > The global builder image doesn't cause too many issues fortunately and > > isn't the problem here. Our current setup actually has the flexibility > > to use different images - we're just constrained in terms of setting > > this up by Jenkins as there is a bunch of tooling needed to provision > > jobs. > > > > A good proportion of the current "complexity" is due to our use of > > Jenkins. The remainder is due to needing to provide the most recent > > version of a given KDE project as a dependency, along with the > > infrastructure for tests to execute successfully (which is quite a bit > > more than just 'make test') > > > > The need to use the most recent version of a KDE project means either: > > a) You have to have all of those projects use the same image as well; or > > b) You have a special job to build all of those projects that don't > > use the same image (which is what the 'Dependency Build' jobs do); or > > c) You incorporate the project within the base system itself > > > > Option (c) is not possible on Windows or FreeBSD as those platforms do > > not support dynamic provisioning of images - meaning that all KDE > > projects have to share the same machines and therefore must be > > provided via our mechanisms (options a and b above). > > > > In the case of Plasma, baking in say Frameworks to the image would > > mean a full image rebuild everytime you needed something new in a > > Framework. Please note that kwindowsystem and plasma-framework are > > both Frameworks. Suffice to say, that won't work - you would be > > rebuilding that image multiple times a week at a minimum. > > > > The only way to manage the build environment for something like Plasma > > is to do it on a Product level, taking all of it's member projects > > into account. This is what we do currently - while the actual Docker > > image may happen to be shared with Frameworks and Applications, that > > isn't required and could be seperated within our existing > > architecture. > > > > Or use option (d): a repository of artifacts which you push to/pull > from. The problem I see with that right now is: we would need "package > management" and our projects use mostly plain CMake which by itself > doesn't understand this concept. So the standard tooling for our > projects can only check for declared dependencies, but cannot try to > install them locally. As I understand it, that is effectively what a > good chunk of the CI tooling sort of replicates anyway: a custom built > package management solution for KDE CI. We already do have a repository of artifacts :) You can find the public view of this at https://build-artifacts.kde.org/production/ That is how options (a) and (b) work - so you are right that we have a mini package management solution of our own. This is why we have the dependency metadata files. > > Out of interest, apart from the amount of work it might take what > would be the main blocker for using VMs and things like Vagrant boxes > for FreeBSD as the next best thing to containers? I'd ask the same for > Windows, but I imagine the answer is licensing costs. > We already use Virtual Machines for both FreeBSD and Windows. The main problem here is that there is no way of dynamically provisioning full VMs without building an entire Openstack type system, which is just too much overhead for the number of builders we have. > > > > That isn't CI. That is CD - as it results in an artifact for use on > > developers/testers/end user systems. > > > > In the sense that if you were to push to a package repository it > entails a deployment. But building binaries by itself obviously isn't, >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
El sáb., 21 de mar. de 2020 a la(s) 20:00, Johan Ouwerkerk (jm.ouwerk...@gmail.com) escribió: > Out of interest, apart from the amount of work it might take what > would be the main blocker for using VMs and things like Vagrant boxes > for FreeBSD as the next best thing to containers? I'd ask the same for > Windows, but I imagine the answer is licensing costs. We already use VMs for that. We obviously can't run a FreeBSD or Windows in a container in a Linux host. -- Nicolás
CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Sat, Mar 21, 2020 at 10:27 PM Ben Cooksley wrote: > > On Sun, Mar 22, 2020 at 3:27 AM Johan Ouwerkerk > wrote: > > > > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley wrote: > > > > > > Comments welcome. Please note that simply fixing the dependency > > > breakage in this case is not enough to resolve this - there are > > > underlying issues which need to be addressed here. > > > > > > Regards, > > > Ben Cooksley > > > KDE Sysadmin > > > > I cannot comment as to whether or not this is a pattern of behaviour > > or just a few isolated instances. From a technical perspective I feel > > there are two (additional) underlying issues worth addressing here: > > > > 1. This could be prevented for the most part by having CI run before, > > and not after the fact. I.e. prior to merging code. > > This will happen once we are on Gitlab. > > > 2. Different projects have different CI needs, and it would help if a > > project could safely manage their CI environment "on their own" as > > much as possible. The current system requires a lot of daunting > > (possibly otherwise unnecessary) complexity purely to manage the fact > > that a builder image will be used not just for one project but for > > perhaps the whole of KDE. > > The global builder image doesn't cause too many issues fortunately and > isn't the problem here. Our current setup actually has the flexibility > to use different images - we're just constrained in terms of setting > this up by Jenkins as there is a bunch of tooling needed to provision > jobs. > > A good proportion of the current "complexity" is due to our use of > Jenkins. The remainder is due to needing to provide the most recent > version of a given KDE project as a dependency, along with the > infrastructure for tests to execute successfully (which is quite a bit > more than just 'make test') > > The need to use the most recent version of a KDE project means either: > a) You have to have all of those projects use the same image as well; or > b) You have a special job to build all of those projects that don't > use the same image (which is what the 'Dependency Build' jobs do); or > c) You incorporate the project within the base system itself > > Option (c) is not possible on Windows or FreeBSD as those platforms do > not support dynamic provisioning of images - meaning that all KDE > projects have to share the same machines and therefore must be > provided via our mechanisms (options a and b above). > > In the case of Plasma, baking in say Frameworks to the image would > mean a full image rebuild everytime you needed something new in a > Framework. Please note that kwindowsystem and plasma-framework are > both Frameworks. Suffice to say, that won't work - you would be > rebuilding that image multiple times a week at a minimum. > > The only way to manage the build environment for something like Plasma > is to do it on a Product level, taking all of it's member projects > into account. This is what we do currently - while the actual Docker > image may happen to be shared with Frameworks and Applications, that > isn't required and could be seperated within our existing > architecture. > Or use option (d): a repository of artifacts which you push to/pull from. The problem I see with that right now is: we would need "package management" and our projects use mostly plain CMake which by itself doesn't understand this concept. So the standard tooling for our projects can only check for declared dependencies, but cannot try to install them locally. As I understand it, that is effectively what a good chunk of the CI tooling sort of replicates anyway: a custom built package management solution for KDE CI. Out of interest, apart from the amount of work it might take what would be the main blocker for using VMs and things like Vagrant boxes for FreeBSD as the next best thing to containers? I'd ask the same for Windows, but I imagine the answer is licensing costs. > > That isn't CI. That is CD - as it results in an artifact for use on > developers/testers/end user systems. > In the sense that if you were to push to a package repository it entails a deployment. But building binaries by itself obviously isn't, even if you expose these as downloadable artifacts. > > The infrastructure for Android builds has grown organically, and I > suspect we are at a point where we do need to take a step back and ask > ourselves if there is a better way of doing this that is more > maintainable. The dependencies bit in particular definitely needs > work. > Agreed. And I understand that there is always something to be said for working code now, rather than possibly working something maybe sometime in the future. > > CI is definitely not easy. Especially not when you are dealing with an > organisation the size of KDE, where projects depend on the latest > development version of other projects. > Oh I agree. I'm not definitely not claiming that any of this is easy or obvious or that what I throw out here will not
Re: Manner in which kde-gtk-config development is conducted
On Sun, Mar 22, 2020 at 3:27 AM Johan Ouwerkerk wrote: > > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley wrote: > > > > Comments welcome. Please note that simply fixing the dependency > > breakage in this case is not enough to resolve this - there are > > underlying issues which need to be addressed here. > > > > Regards, > > Ben Cooksley > > KDE Sysadmin > > I cannot comment as to whether or not this is a pattern of behaviour > or just a few isolated instances. From a technical perspective I feel > there are two (additional) underlying issues worth addressing here: > > 1. This could be prevented for the most part by having CI run before, > and not after the fact. I.e. prior to merging code. This will happen once we are on Gitlab. > 2. Different projects have different CI needs, and it would help if a > project could safely manage their CI environment "on their own" as > much as possible. The current system requires a lot of daunting > (possibly otherwise unnecessary) complexity purely to manage the fact > that a builder image will be used not just for one project but for > perhaps the whole of KDE. The global builder image doesn't cause too many issues fortunately and isn't the problem here. Our current setup actually has the flexibility to use different images - we're just constrained in terms of setting this up by Jenkins as there is a bunch of tooling needed to provision jobs. A good proportion of the current "complexity" is due to our use of Jenkins. The remainder is due to needing to provide the most recent version of a given KDE project as a dependency, along with the infrastructure for tests to execute successfully (which is quite a bit more than just 'make test') The need to use the most recent version of a KDE project means either: a) You have to have all of those projects use the same image as well; or b) You have a special job to build all of those projects that don't use the same image (which is what the 'Dependency Build' jobs do); or c) You incorporate the project within the base system itself Option (c) is not possible on Windows or FreeBSD as those platforms do not support dynamic provisioning of images - meaning that all KDE projects have to share the same machines and therefore must be provided via our mechanisms (options a and b above). In the case of Plasma, baking in say Frameworks to the image would mean a full image rebuild everytime you needed something new in a Framework. Please note that kwindowsystem and plasma-framework are both Frameworks. Suffice to say, that won't work - you would be rebuilding that image multiple times a week at a minimum. The only way to manage the build environment for something like Plasma is to do it on a Product level, taking all of it's member projects into account. This is what we do currently - while the actual Docker image may happen to be shared with Frameworks and Applications, that isn't required and could be seperated within our existing architecture. > > As for running the CI beforehand: by this I mean that it is relatively > easy to 'break the build' one way or another and for this not to be > caught during review, especially if a change is complex and has been > in development for a while with many revisions. Certainly this is a > particularly severe case of breaking the build, but the same should > also apply to e.g. tests that start to fail. > > On the day job I find that it is absolutely routine for commits to be > pushed to feature branches for review which don't even pass the CI > sanity check. This is not because people are lazy, but because of the > perennial pitfall of "works on my machine" (local config vs. remote), > and sometimes because people want to get early review on code > structure for example. > > I think it would help massively for CI to be run on feature branches > prior to merging as a pre-requirement for merging. This should be > automatic and not require any special care on the part of reviewers, > and the result of the CI should be immediately visible as part of the > review tooling (workflow UI/UX). After that it is mostly a process of > unlearning to commit to master directly. This should be fairly easy to > implement for invent.kde.org, in the sense that Gitlab offers such CI > as a feature out of the box and it is fully integrated with the MR > workflow. > That is correct, and the plans we've discussed for CI on Gitlab (invent.kde.org) do plan on allowing for CI on feature branches. At this time, we are on Jenkins, and it is extremely non-trivial and therefore not possible to support feature branches in this manner. > Regarding your point about changing dependencies and the need for > communication to manage the CI environment, to the extent that this > breaks the build this could be simplified if it were easier to manage > the build environment yourself (from a project perspective). This > brings to my second point: I think it would be desirable for projects > to be able to cater for their CI
Re: Manner in which kde-gtk-config development is conducted
On Sun, Mar 22, 2020 at 3:08 AM David Edmundson wrote: > > You're absolutely right that mistakes were made and have reason to be > frustrated. > > kde-gtk-config is now maintained by new developers. > Plasma has a new influx of new people which is good to see and > something we need to foster carefully. > > Overall these new devs are doing a super job and we want to encourage them. > > Ultimately there are two parties at fault: > - brand new developers who didn't know the rules. KDE has a lot of > rules and they're certainly not all written down in a consistent > location. > > - the more "senior" Plasma people (me, Kai, Aleix, etc) who do know > the rules, not paying due attention to something that's now under > Plasma's umbrella > > >which means that the repository is no longer eligible to form > >part of a KDE release module and should be moved to Playground > > I think this is an overreaction that punishes the wrong people. Users. The reaction is intended to force the hand of Plasma as a collective group to pay attention to the notifications from the CI system - which it delivers to the plasma-devel mailing list. I know that some people do pay attention to these, but it is evident that others do not or have active filtering in place to ensure they don't see them. > > I think the tone of this email sends a strong message, and is in > proportion, but any action would be too much. > Ultimately if devs break a CI, its the devs that'll suffer because > they won't have a CI. > > >Comments welcome. > > I hear pre-commit CI will be a thing in the future? If so this > specific problem will address itself. Once we are on Gitlab yes, we will get this. > > David Cheers, Ben
Re: Manner in which kde-gtk-config development is conducted
On Sat, Mar 21, 2020, 10:27 AM Johan Ouwerkerk wrote: > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley wrote: > > > > Comments welcome. Please note that simply fixing the dependency > > breakage in this case is not enough to resolve this - there are > > underlying issues which need to be addressed here. > > > > Regards, > > Ben Cooksley > > KDE Sysadmin > > I cannot comment as to whether or not this is a pattern of behaviour > or just a few isolated instances. From a technical perspective I feel > there are two (additional) underlying issues worth addressing here: > > 1. This could be prevented for the most part by having CI run before, > and not after the fact. I.e. prior to merging code. > 2. Different projects have different CI needs, and it would help if a > project could safely manage their CI environment "on their own" as > much as possible. The current system requires a lot of daunting > (possibly otherwise unnecessary) complexity purely to manage the fact > that a builder image will be used not just for one project but for > perhaps the whole of KDE. > In regards to custom images kdiff3 has been using one since before invent.kde.org came into being. The repo is mirrored on gitlab. Now that invent.kde.org is up that setup runs the official ci as well. Since my custom image is based on bionic it doesn't have the lastest kf5. This quite recently cought a doc change made by some else that broke generation of docs on older doctools versions. Both KDE's official CI and my personal machine had the necessary kf5 version to build it and gave no error. Using custom docker images is a great way of catching this kind of "works for my setup issue". >
Re: Manner in which kde-gtk-config development is conducted
First up: I would like to apologize for submitting and landing
https://phabricator.kde.org/D28076 and
https://phabricator.kde.org/D28086 without properly checking that
everything was behaving as it should.
For the initial breakage of https://phabricator.kde.org/D28076, I
failed to check that it built from a clean directory after changing
CMake files.
For https://phabricator.kde.org/D28086, I erroneously thought that
giomm was already a dependency of kde-gtk-config because I didn't
notice that ${GTK_GIOMM_LIBRARY} in
https://cgit.kde.org/kde-gtk-config.git/tree/gtkproxies/CMakeLists.txt#n12
wasn't defined anywhere.
> No such communication was done in this case. Once again this is unacceptable.
While we didn't notify y'all about the new dependency, I thought this
was already a dependency as stated above and didn't think much of it.
I will keep this in mind for when I do intentionally add new
dependencies.
> The first violation was a compilation failure introduced following the
> commit of changes in https://phabricator.kde.org/D28076. This failure
> was completely ignored by all involved developers, including the
> Plasma community in general until I raised the matter by commenting on
> the original review - several days after the fact.
I can understand your frustration with us approving and committing
faulty code, but we didn't ignore it. While I failed to revert my
changes after both Mikhail and I noticed that it failed to build from
source only a few hours after landing, we were aware and discussing it
in the thread.
> Comments welcome. Please note that simply fixing the dependency
> breakage in this case is not enough to resolve this - there are
> underlying issues which need to be addressed here.
I feel as long as everyone involved understands what went wrong with
these two patches, we should avoid this problem again in the future.
Thanks for understanding,
Carson Black [ jan Pontaoski ]
Am Fr., 20. März 2020 um 20:32 Uhr schrieb Ben Cooksley :
>
> Hi all,
>
> In recent days the repository `kde-gtk-config` has had a string of
> issues with the CI system which raises serious questions concerning
> the development practices of this project.
>
> The first violation was a compilation failure introduced following the
> commit of changes in https://phabricator.kde.org/D28076. This failure
> was completely ignored by all involved developers, including the
> Plasma community in general until I raised the matter by commenting on
> the original review - several days after the fact.
>
> This is unacceptable - part of the agreed rules for all code committed
> to KDE repositories is that it compiles.
>
> The second violation, which took place only a few hours after the
> initial breakage was corrected, was an unannounced change to the
> dependencies of the project which took place as a result of code
> reviewed in https://phabricator.kde.org/D28086
>
> As has been discussed on these lists many times in the past, it is a
> requirement for changes to the external dependencies of projects to be
> communicated to the maintainers of the CI system in advance of the
> change itself being made.
>
> No such communication was done in this case. Once again this is unacceptable.
>
> I therefore conclude that the development of kde-gtk-config is being
> conducted in a matter which is not consistent with that of a KDE
> project, which means that the repository is no longer eligible to form
> part of a KDE release module and should be moved to Playground.
>
> Comments welcome. Please note that simply fixing the dependency
> breakage in this case is not enough to resolve this - there are
> underlying issues which need to be addressed here.
>
> Regards,
> Ben Cooksley
> KDE Sysadmin
Re: Manner in which kde-gtk-config development is conducted
On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley wrote: > > Comments welcome. Please note that simply fixing the dependency > breakage in this case is not enough to resolve this - there are > underlying issues which need to be addressed here. > > Regards, > Ben Cooksley > KDE Sysadmin I cannot comment as to whether or not this is a pattern of behaviour or just a few isolated instances. From a technical perspective I feel there are two (additional) underlying issues worth addressing here: 1. This could be prevented for the most part by having CI run before, and not after the fact. I.e. prior to merging code. 2. Different projects have different CI needs, and it would help if a project could safely manage their CI environment "on their own" as much as possible. The current system requires a lot of daunting (possibly otherwise unnecessary) complexity purely to manage the fact that a builder image will be used not just for one project but for perhaps the whole of KDE. As for running the CI beforehand: by this I mean that it is relatively easy to 'break the build' one way or another and for this not to be caught during review, especially if a change is complex and has been in development for a while with many revisions. Certainly this is a particularly severe case of breaking the build, but the same should also apply to e.g. tests that start to fail. On the day job I find that it is absolutely routine for commits to be pushed to feature branches for review which don't even pass the CI sanity check. This is not because people are lazy, but because of the perennial pitfall of "works on my machine" (local config vs. remote), and sometimes because people want to get early review on code structure for example. I think it would help massively for CI to be run on feature branches prior to merging as a pre-requirement for merging. This should be automatic and not require any special care on the part of reviewers, and the result of the CI should be immediately visible as part of the review tooling (workflow UI/UX). After that it is mostly a process of unlearning to commit to master directly. This should be fairly easy to implement for invent.kde.org, in the sense that Gitlab offers such CI as a feature out of the box and it is fully integrated with the MR workflow. Regarding your point about changing dependencies and the need for communication to manage the CI environment, to the extent that this breaks the build this could be simplified if it were easier to manage the build environment yourself (from a project perspective). This brings to my second point: I think it would be desirable for projects to be able to cater for their CI needs on their own as much as (safely) possible. I understand why you would want to avoid proliferation of too many CI setups but as the same time having a single image that contains everything + kitchen sink also leads to problems. In particular it becomes hard to ensure the image and versions of tooling or pre-installed libraries are compatible with every project, to keep this up to date and to ensure that this all remains consistent with the CI aims of a particular project. For instance a KF5 framework might want to validate that they can build and pass tests against *both* an ancient version of Qt (for stability promises) and the most recent Qt release as well (and possibly even Qt from master to get a heads up of forwards compatibility issues). A Plasma Mobile app might have rather different CI needs: it matters that both 64 and 32 bit builds are produced, both as Android apps and flatpaks. A project might want to validate that it builds against both master versions of KF5 frameworks and those more typically found on a typical distro. This sort of per-project complexity is hard to deal with in one big builder image, but something we ideally would be able to pull off for all our projects as a matter of routine. I think we already run into the complexity problems pretty hard with the current CI setup in that sense: taking binary factory as an example, it is quite obvious that the JSON blob encoding dependency information for Android apps is something that came out of a need for "yet another" layer of abstraction to manage CI needs of many different projects, and that this is hard to grasp, comprehend fully let alone maintain: there is an ever-growing list of ad-hoc helper scripts needed to keep the JSON blob manageable, and even so it is not very easy to follow if a project has to pull in multiple dependencies. For comparison I would ask: how many people understand how all of the following actually work: - The various instructions in Docker files - The related resources copied into the images, in particular the shell scripts - The Python scripts that do much of the heavy lifting - The Jenkins groovy DSL scripts that actually make this work on Jenkins - The repository metadata, and dependency information files - The various data files that encode crucial config data/build
Re: Manner in which kde-gtk-config development is conducted
You're absolutely right that mistakes were made and have reason to be frustrated. kde-gtk-config is now maintained by new developers. Plasma has a new influx of new people which is good to see and something we need to foster carefully. Overall these new devs are doing a super job and we want to encourage them. Ultimately there are two parties at fault: - brand new developers who didn't know the rules. KDE has a lot of rules and they're certainly not all written down in a consistent location. - the more "senior" Plasma people (me, Kai, Aleix, etc) who do know the rules, not paying due attention to something that's now under Plasma's umbrella >which means that the repository is no longer eligible to form >part of a KDE release module and should be moved to Playground I think this is an overreaction that punishes the wrong people. Users. I think the tone of this email sends a strong message, and is in proportion, but any action would be too much. Ultimately if devs break a CI, its the devs that'll suffer because they won't have a CI. >Comments welcome. I hear pre-commit CI will be a thing in the future? If so this specific problem will address itself. David
Please watch build statuses of dependent projects (was: Re: Manner in which kde-gtk-config development is conducted)
In data sabato 21 marzo 2020 01:31:40 CET, Ben Cooksley ha scritto: > Plasma community in general until I raised the matter by commenting on > the original review - several days after the fact. I take the opportunity to ask reviewers who review changes to watch for the impact of such changes on other dependent projects within KDE. Case in point: kaccounts-integration changes (which were reviewed, mind you, so this is not a blind push or something reckless) which broke building / dependent projects several times over the course of the past week. This is something that even a pre-review CI won't help with, so lxr and careful approaches are the way to go. -- Luca Beltrame - KDE Forums team GPG key ID: A29D259B signature.asc Description: This is a digitally signed message part.
