Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Sat, Mar 28, 2020 at 12:44 PM Kevin Kofler wrote: > > Ben Cooksley wrote: > > I'm unhappy with them due to how they handled the complete disaster > > that was a significant version update to a core system library (libc I > > think) which they did in a stable, released distribution. > > I cannot really speak for that part, but for the following part: > > > It was at this point that I had finally had enough of Fedora (having > > previously had to deal with their internal politics over the packaging > > of QtWebEngine, which meant we ended up having to use Qt 5.8 rather > > than the Qt 5.9 which we had initially targeted as the 5.9 packaging > > was blocked) and dumped them for SUSE, the images that we continue to > > use today. > > the technical issue back then was that QtWebEngine was heavily patched in > Fedora, in particular, in order to support x86 machines without SSE2. (Yes, > these exist.) All the patches, including the huge no-sse2 patch, had to be > ported from release to release by one person that happened to be me. > (Needless to say, I was strictly opposed to pushing an update removing > support for some hardware to a stable release, as I hope you can understand. > Hence, rebasing the no-sse2 patch was a blocker.) As soon as the patch > rebases were ready, the package update was submitted. These days, > QtWebEngine is not as heavily patched anymore, and in particular, we no > longer attempt to support x86 without SSE2, because Fedora has dropped > support for it distrowide. > > In addition, I have since resigned as the primary maintainer of QtWebEngine. > You will occasionally find me committing some fixes, but I am no longer the > point of contact nor the one doing most of the work. QtWebEngine in Fedora > is now maintained by Rex Dieter. And the comaintainer who attempted (and > failed) to do the QtWebEngine 5.9 upgrade at your request has since left > Fedora entirely. So the political issue should also be resolved to your > liking. > > I hereby formally apologize for any extra work the delays in getting Qt 5.9 > out (and the possible miscommunications around them – I do not know what the > former comaintainer may have promised to you without consulting me first) > may have caused to you. It will never happen the same way again. Thank you Kevin. The issues with Qt 5.9 were relatively minor in the scheme of things, but it is nice to know how things have changed since then none the less. > > Kevin Kofler > Cheers, Ben
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
Ben Cooksley wrote: > I'm unhappy with them due to how they handled the complete disaster > that was a significant version update to a core system library (libc I > think) which they did in a stable, released distribution. I cannot really speak for that part, but for the following part: > It was at this point that I had finally had enough of Fedora (having > previously had to deal with their internal politics over the packaging > of QtWebEngine, which meant we ended up having to use Qt 5.8 rather > than the Qt 5.9 which we had initially targeted as the 5.9 packaging > was blocked) and dumped them for SUSE, the images that we continue to > use today. the technical issue back then was that QtWebEngine was heavily patched in Fedora, in particular, in order to support x86 machines without SSE2. (Yes, these exist.) All the patches, including the huge no-sse2 patch, had to be ported from release to release by one person that happened to be me. (Needless to say, I was strictly opposed to pushing an update removing support for some hardware to a stable release, as I hope you can understand. Hence, rebasing the no-sse2 patch was a blocker.) As soon as the patch rebases were ready, the package update was submitted. These days, QtWebEngine is not as heavily patched anymore, and in particular, we no longer attempt to support x86 without SSE2, because Fedora has dropped support for it distrowide. In addition, I have since resigned as the primary maintainer of QtWebEngine. You will occasionally find me committing some fixes, but I am no longer the point of contact nor the one doing most of the work. QtWebEngine in Fedora is now maintained by Rex Dieter. And the comaintainer who attempted (and failed) to do the QtWebEngine 5.9 upgrade at your request has since left Fedora entirely. So the political issue should also be resolved to your liking. I hereby formally apologize for any extra work the delays in getting Qt 5.9 out (and the possible miscommunications around them – I do not know what the former comaintainer may have promised to you without consulting me first) may have caused to you. It will never happen the same way again. Kevin Kofler
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Tue, Mar 24, 2020 at 12:55 AM Thomas Baumgart wrote: > > On Montag, 23. März 2020 09:32:26 CET Ben Cooksley wrote: > > > On Mon, Mar 23, 2020 at 6:53 AM Albert Astals Cid wrote: > > [...] > > > > But If you're volunteering to do a Windows gitlab CI for QCA I'll take it > > > :) > > > > Not at this stage - aside from you mentioning it here, and Krita > > needing to use MingW for their builds I haven't come across any other > > KDE project that was after/interested in MingW on Windows CI. > > (I'm curious though as to why MingW on Windows is relevant given the > > GCC on Linux coverage and MSVC on Windows that we already have) > > Isn't in KMyMoney MingW the requirement to have AqBanking build since it does > not build with MSVC? At least that is my current understanding of the matter. > Or do I misinterpret something here? I ignored the Binary Factory for the purposes of my above comment, but yes - KMyMoney and a few others do depend on projects that can only be built under MingW. > > -- > > Regards > > Thomas Baumgart Cheers, Ben > > https://www.signal.org/ Signal, the better WhatsApp > - > Be kind whenever possible. It is always possible. -- Dalai Lama > -
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Montag, 23. März 2020 09:32:26 CET Ben Cooksley wrote: > On Mon, Mar 23, 2020 at 6:53 AM Albert Astals Cid wrote: [...] > > But If you're volunteering to do a Windows gitlab CI for QCA I'll take it :) > > Not at this stage - aside from you mentioning it here, and Krita > needing to use MingW for their builds I haven't come across any other > KDE project that was after/interested in MingW on Windows CI. > (I'm curious though as to why MingW on Windows is relevant given the > GCC on Linux coverage and MSVC on Windows that we already have) Isn't in KMyMoney MingW the requirement to have AqBanking build since it does not build with MSVC? At least that is my current understanding of the matter. Or do I misinterpret something here? -- Regards Thomas Baumgart https://www.signal.org/ Signal, the better WhatsApp - Be kind whenever possible. It is always possible. -- Dalai Lama - signature.asc Description: This is a digitally signed message part.
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Mon, Mar 23, 2020 at 6:53 AM Albert Astals Cid wrote: > > El diumenge, 22 de març de 2020, a les 16:12:04 CET, Ben Cooksley va escriure: > > On Mon, Mar 23, 2020 at 12:49 AM Albert Astals Cid wrote: > > > > > > El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va > > > escriure: > > > > Note however that images based upon Fedora or anything that shares > > > > it's lineage (including CentOS and it's derivates) is strictly > > > > prohibited and won't be accepted for inclusion. > > > > > > I still find this highly annoying since Fedora seems to be the only > > > distro providing an almost full stack of mingw packages so for example i > > > could add easily a mingw/windows CI to QCA using fedora but i can't > > > because, for some reason i forgot, you are very unhappy with them. > > > > > > > ... > > > > This is why Fedora and all associated derivatives are banned from our > > systems. > > Is there a way they can be ever forgiven or is your plan to ban Fedora > forever? It would be reasonably difficult for them to be forgiven for their transgressions i'm afraid, but not impossible. It would need to be started from their side however, and they would need to reform the way they operate in addition to apologise for their severe failures. > > > > > > Cheers, > > > Albert > > > > With regards to MingW, the better way to do this would be by using > > native MingW rather than cross compiling. Craft already uses MingW for > > some dependencies, and Krita's binary factory builds use it as well so > > most of the infrastructure for this is already in position. > > But that's something that requires magic and thus sysadmin to do it, a > Fedora+mingw gitlab CI is something i can do myself. Not too much magic - all the information on install locations can be found within Craft / Craftmaster. Once you have that you would just need to customise a pipeline template for Mingw builds. > > But If you're volunteering to do a Windows gitlab CI for QCA I'll take it :) Not at this stage - aside from you mentioning it here, and Krita needing to use MingW for their builds I haven't come across any other KDE project that was after/interested in MingW on Windows CI. (I'm curious though as to why MingW on Windows is relevant given the GCC on Linux coverage and MSVC on Windows that we already have) > > Cheers, > Albert Cheers, Ben > > > > > Regards, > > Ben > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > - Johan > > > > > > > > Cheers, > > > > Ben > > > > > > > > > > > > > > > > > > > > > >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
El diumenge, 22 de març de 2020, a les 16:12:04 CET, Ben Cooksley va escriure: > On Mon, Mar 23, 2020 at 12:49 AM Albert Astals Cid wrote: > > > > El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va > > escriure: > > > Note however that images based upon Fedora or anything that shares > > > it's lineage (including CentOS and it's derivates) is strictly > > > prohibited and won't be accepted for inclusion. > > > > I still find this highly annoying since Fedora seems to be the only distro > > providing an almost full stack of mingw packages so for example i could add > > easily a mingw/windows CI to QCA using fedora but i can't because, for some > > reason i forgot, you are very unhappy with them. > > > > ... > > This is why Fedora and all associated derivatives are banned from our systems. Is there a way they can be ever forgiven or is your plan to ban Fedora forever? > > > Cheers, > > Albert > > With regards to MingW, the better way to do this would be by using > native MingW rather than cross compiling. Craft already uses MingW for > some dependencies, and Krita's binary factory builds use it as well so > most of the infrastructure for this is already in position. But that's something that requires magic and thus sysadmin to do it, a Fedora+mingw gitlab CI is something i can do myself. But If you're volunteering to do a Windows gitlab CI for QCA I'll take it :) Cheers, Albert > > Regards, > Ben > > > > > > > > > > > > > > Regards, > > > > > > > > - Johan > > > > > > Cheers, > > > Ben > > > > > > > > > > > >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Mon, Mar 23, 2020 at 12:41 AM Johan Ouwerkerk wrote: > > On Sun, Mar 22, 2020 at 3:20 AM Ben Cooksley wrote: > > > > We already do have a repository of artifacts :) > > You can find the public view of this at > > https://build-artifacts.kde.org/production/ > > > > > > > We already use Virtual Machines for both FreeBSD and Windows. > > > > The main problem here is that there is no way of dynamically > > provisioning full VMs without building an entire Openstack type > > system, which is just too much overhead for the number of builders we > > have. > > > > First of all, thanks for your patience in explaining to me what must > be obvious to you. I really appreciate it. > > > > > Shouldn't this kind of experimentation be done on developers systems > > rather than the CI system? > > Please do remember that CI resources are not infinite and at times can > > be quite constrained. > > > > My main query here though is what would be achieved by allowing this > > sort of thing that the existing system does not allow for? > > (Ignoring the fact we don't cover feature branches - something which > > is in the long term pipeline) > > > > Pure experimentation should definitely be done on developers' systems. > But changes to your CI cannot be fully validated on your own machine > beforehand: you need to verify that it works on the actual CI > builders. Ideally without breaking the CI for anyone else. > > So the main gain here is that projects can improve, evaluate and > mature their CI efforts independently. Examples of things we currently > don't have but some projects might benefit from are: licensing > compliance checking (reuse), dependency scanning (e.g.: snyk) and > mutation testing (e.g.: stryker). > Sounds reasonable enough. My main concern here is people trying to replicate builds under their $distroOfChoice which doesn't really add value to the system but which does increase resource consumption. > I'm not saying that this couldn't be achieved today with a lot of help > and guidance from the sysadmins. But I think this does impose a lot of > overhead, especially while such changes are still experimental and do > not fully work yet in CI or need tweaking for prettier output or > whatever. In fact, your reply to my other question kind of indicates > as much: > > > > > > > Possibly projects would also submit custom images for consideration to > > > that registry. > > > > This isn't something that is terribly easy to do within our existing > > Jenkins setup - but is theoretically possible. It requires quite a bit > > of provisioning work on the Jenkins side. > > > > The CI scripts would also need some adapting to make this sort of > > setup efficient to operate with them if you were to use them (and you > > probably do want to use them for running tests). > > A good portion of this adapting will done as part of moving the CI > > system to Gitlab. > > Does this mean that we will be encouraged to configure `image` > settings in .gitlab-ci.yml? If so, that basically gives me what I > really wanted here. All the actual images I might ever need could be > submitted for review via MR to a repository on invent as and when I > would need them. I would prefer if we could avoid the proliferation of too many images if at all possible, however there may be scope for this where they don't overlap functionality wise with our existing images (a different Qt version does not overlap, but providing builds using a different distribution for a version of Qt already covered definitely would overlap) > > > > > Note however that images based upon Fedora or anything that shares > > it's lineage (including CentOS and it's derivates) is strictly > > prohibited and won't be accepted for inclusion. > > > > Personally I'm more inclined towards using Debian myself, but out of > interest what is the reason for banning Fedora and CentOS images? Does > this still apply when the move to Gitlab CI is completed? I've detailed this ban in my response to Albert. The ban applies to all parts of our infrastructure, including Gitlab, and is indefinite in duration. Images based upon Debian, Ubuntu and SUSE are perfectly acceptable. > > Regards, > > - Johan Cheers, Ben
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Mon, Mar 23, 2020 at 12:49 AM Albert Astals Cid wrote: > > El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va escriure: > > Note however that images based upon Fedora or anything that shares > > it's lineage (including CentOS and it's derivates) is strictly > > prohibited and won't be accepted for inclusion. > > I still find this highly annoying since Fedora seems to be the only distro > providing an almost full stack of mingw packages so for example i could add > easily a mingw/windows CI to QCA using fedora but i can't because, for some > reason i forgot, you are very unhappy with them. > I'm unhappy with them due to how they handled the complete disaster that was a significant version update to a core system library (libc I think) which they did in a stable, released distribution. As one would expect, trying to do such an update broke things, in particular it broke the BDB libraries, which are used by amongst other things, their package manager (DNF). Once this became apparent (which didn't take long) not only did they not withdraw the update to libc/glibc, they pressed ahead with a series of hotfixes to bdb and the package manager. Unfortunately these were reliant on the underlying file system supporting a particular set of POSIX specified functionality, which overlayfs (the file system used by Docker) didn't support. The end result of all of this is that it was impossible for us to generate a new CI image. While this is already bad, this was then amplified by the reaction on the Fedora side, where the DNF developers ignored our (and others) report of the issue they said was fixed still being broken. Two weeks later, and still being ignored, I complained to the Fedora Engineering Steering Committee (Fesco) concerning this, and after a few days someone asked someone else to rebuild the base Fedora Docker image - which pulled in some fixes that resolved the issue finally and let us generate an image. Following this my complaint was closed as "resolved", even though they had not put in place any safeguards to prevent this from happening again, nor had they addressed the failure of the maintainers of one of the core components of their distribution to respond to users regarding something that was totally broken. Unsurprisingly, within a few days the DNF developers pushed yet another update that broke the build of the image again. I returned to Fesco, asking for the push privileges of the DNF developers to be suspended as they had broken it yet again. This finally resulted in the DNF developers responding (funny that!), but not with an apology or anything along those lines. Instead, they engaged in a ranting attack. In this they made various errornous claims (including that we were using an out of date Fedora image, which was not the case and I had provided logs earlier which demonstrated this which it was obvious they had failed to read) and were in general extremely hostile. Following this there was some back and forth and the issue was scheduled for discussion at the next Fesco online meeting (apparently they only make decisions once a week at these meetings). This time for this meeting came and went, with no comment made on the issue. Following further investigation I found out the meeting had been cancelled due to a lack of attendence, which had not been communicated to us. It was at this point that I had finally had enough of Fedora (having previously had to deal with their internal politics over the packaging of QtWebEngine, which meant we ended up having to use Qt 5.8 rather than the Qt 5.9 which we had initially targeted as the 5.9 packaging was blocked) and dumped them for SUSE, the images that we continue to use today. For those wondering, my complaint against the DNF developers was dismissed and closed as "they were responding now" with a one line message posted through an IRC Bot about a week later when they held their next meeting. Suffice to say, I don't want anything to do with Fedora ever again. It was an extremely stressful 6 weeks or so in all, with developers chasing up when we could get some additional packages installed and us being unable to deliver because Fedora had broken things. This is why Fedora and all associated derivatives are banned from our systems. > Cheers, > Albert With regards to MingW, the better way to do this would be by using native MingW rather than cross compiling. Craft already uses MingW for some dependencies, and Krita's binary factory builds use it as well so most of the infrastructure for this is already in position. Regards, Ben > > > > > > > > > Regards, > > > > > > - Johan > > > > Cheers, > > Ben > > > > > >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
El diumenge, 22 de març de 2020, a les 3:19:57 CET, Ben Cooksley va escriure: > Note however that images based upon Fedora or anything that shares > it's lineage (including CentOS and it's derivates) is strictly > prohibited and won't be accepted for inclusion. I still find this highly annoying since Fedora seems to be the only distro providing an almost full stack of mingw packages so for example i could add easily a mingw/windows CI to QCA using fedora but i can't because, for some reason i forgot, you are very unhappy with them. Cheers, Albert > > > > > Regards, > > > > - Johan > > Cheers, > Ben >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Sun, Mar 22, 2020 at 3:20 AM Ben Cooksley wrote: > > We already do have a repository of artifacts :) > You can find the public view of this at > https://build-artifacts.kde.org/production/ > > > We already use Virtual Machines for both FreeBSD and Windows. > > The main problem here is that there is no way of dynamically > provisioning full VMs without building an entire Openstack type > system, which is just too much overhead for the number of builders we > have. > First of all, thanks for your patience in explaining to me what must be obvious to you. I really appreciate it. > > Shouldn't this kind of experimentation be done on developers systems > rather than the CI system? > Please do remember that CI resources are not infinite and at times can > be quite constrained. > > My main query here though is what would be achieved by allowing this > sort of thing that the existing system does not allow for? > (Ignoring the fact we don't cover feature branches - something which > is in the long term pipeline) > Pure experimentation should definitely be done on developers' systems. But changes to your CI cannot be fully validated on your own machine beforehand: you need to verify that it works on the actual CI builders. Ideally without breaking the CI for anyone else. So the main gain here is that projects can improve, evaluate and mature their CI efforts independently. Examples of things we currently don't have but some projects might benefit from are: licensing compliance checking (reuse), dependency scanning (e.g.: snyk) and mutation testing (e.g.: stryker). I'm not saying that this couldn't be achieved today with a lot of help and guidance from the sysadmins. But I think this does impose a lot of overhead, especially while such changes are still experimental and do not fully work yet in CI or need tweaking for prettier output or whatever. In fact, your reply to my other question kind of indicates as much: > > > > Possibly projects would also submit custom images for consideration to > > that registry. > > This isn't something that is terribly easy to do within our existing > Jenkins setup - but is theoretically possible. It requires quite a bit > of provisioning work on the Jenkins side. > > The CI scripts would also need some adapting to make this sort of > setup efficient to operate with them if you were to use them (and you > probably do want to use them for running tests). > A good portion of this adapting will done as part of moving the CI > system to Gitlab. Does this mean that we will be encouraged to configure `image` settings in .gitlab-ci.yml? If so, that basically gives me what I really wanted here. All the actual images I might ever need could be submitted for review via MR to a repository on invent as and when I would need them. > > Note however that images based upon Fedora or anything that shares > it's lineage (including CentOS and it's derivates) is strictly > prohibited and won't be accepted for inclusion. > Personally I'm more inclined towards using Debian myself, but out of interest what is the reason for banning Fedora and CentOS images? Does this still apply when the move to Gitlab CI is completed? Regards, - Johan
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Sun, Mar 22, 2020 at 12:00 PM Johan Ouwerkerk wrote: > > On Sat, Mar 21, 2020 at 10:27 PM Ben Cooksley wrote: > > > > On Sun, Mar 22, 2020 at 3:27 AM Johan Ouwerkerk > > wrote: > > > > > > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley wrote: > > > > > > > > Comments welcome. Please note that simply fixing the dependency > > > > breakage in this case is not enough to resolve this - there are > > > > underlying issues which need to be addressed here. > > > > > > > > Regards, > > > > Ben Cooksley > > > > KDE Sysadmin > > > > > > I cannot comment as to whether or not this is a pattern of behaviour > > > or just a few isolated instances. From a technical perspective I feel > > > there are two (additional) underlying issues worth addressing here: > > > > > > 1. This could be prevented for the most part by having CI run before, > > > and not after the fact. I.e. prior to merging code. > > > > This will happen once we are on Gitlab. > > > > > 2. Different projects have different CI needs, and it would help if a > > > project could safely manage their CI environment "on their own" as > > > much as possible. The current system requires a lot of daunting > > > (possibly otherwise unnecessary) complexity purely to manage the fact > > > that a builder image will be used not just for one project but for > > > perhaps the whole of KDE. > > > > The global builder image doesn't cause too many issues fortunately and > > isn't the problem here. Our current setup actually has the flexibility > > to use different images - we're just constrained in terms of setting > > this up by Jenkins as there is a bunch of tooling needed to provision > > jobs. > > > > A good proportion of the current "complexity" is due to our use of > > Jenkins. The remainder is due to needing to provide the most recent > > version of a given KDE project as a dependency, along with the > > infrastructure for tests to execute successfully (which is quite a bit > > more than just 'make test') > > > > The need to use the most recent version of a KDE project means either: > > a) You have to have all of those projects use the same image as well; or > > b) You have a special job to build all of those projects that don't > > use the same image (which is what the 'Dependency Build' jobs do); or > > c) You incorporate the project within the base system itself > > > > Option (c) is not possible on Windows or FreeBSD as those platforms do > > not support dynamic provisioning of images - meaning that all KDE > > projects have to share the same machines and therefore must be > > provided via our mechanisms (options a and b above). > > > > In the case of Plasma, baking in say Frameworks to the image would > > mean a full image rebuild everytime you needed something new in a > > Framework. Please note that kwindowsystem and plasma-framework are > > both Frameworks. Suffice to say, that won't work - you would be > > rebuilding that image multiple times a week at a minimum. > > > > The only way to manage the build environment for something like Plasma > > is to do it on a Product level, taking all of it's member projects > > into account. This is what we do currently - while the actual Docker > > image may happen to be shared with Frameworks and Applications, that > > isn't required and could be seperated within our existing > > architecture. > > > > Or use option (d): a repository of artifacts which you push to/pull > from. The problem I see with that right now is: we would need "package > management" and our projects use mostly plain CMake which by itself > doesn't understand this concept. So the standard tooling for our > projects can only check for declared dependencies, but cannot try to > install them locally. As I understand it, that is effectively what a > good chunk of the CI tooling sort of replicates anyway: a custom built > package management solution for KDE CI. We already do have a repository of artifacts :) You can find the public view of this at https://build-artifacts.kde.org/production/ That is how options (a) and (b) work - so you are right that we have a mini package management solution of our own. This is why we have the dependency metadata files. > > Out of interest, apart from the amount of work it might take what > would be the main blocker for using VMs and things like Vagrant boxes > for FreeBSD as the next best thing to containers? I'd ask the same for > Windows, but I imagine the answer is licensing costs. > We already use Virtual Machines for both FreeBSD and Windows. The main problem here is that there is no way of dynamically provisioning full VMs without building an entire Openstack type system, which is just too much overhead for the number of builders we have. > > > > That isn't CI. That is CD - as it results in an artifact for use on > > developers/testers/end user systems. > > > > In the sense that if you were to push to a package repository it > entails a deployment. But building binaries by itself obviously isn't, >
Re: CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
El sáb., 21 de mar. de 2020 a la(s) 20:00, Johan Ouwerkerk (jm.ouwerk...@gmail.com) escribió: > Out of interest, apart from the amount of work it might take what > would be the main blocker for using VMs and things like Vagrant boxes > for FreeBSD as the next best thing to containers? I'd ask the same for > Windows, but I imagine the answer is licensing costs. We already use VMs for that. We obviously can't run a FreeBSD or Windows in a container in a Linux host. -- Nicolás
CI talk (Was: re: Manner in which kde-gtk-config development is conducted)
On Sat, Mar 21, 2020 at 10:27 PM Ben Cooksley wrote: > > On Sun, Mar 22, 2020 at 3:27 AM Johan Ouwerkerk > wrote: > > > > On Sat, Mar 21, 2020 at 1:32 AM Ben Cooksley wrote: > > > > > > Comments welcome. Please note that simply fixing the dependency > > > breakage in this case is not enough to resolve this - there are > > > underlying issues which need to be addressed here. > > > > > > Regards, > > > Ben Cooksley > > > KDE Sysadmin > > > > I cannot comment as to whether or not this is a pattern of behaviour > > or just a few isolated instances. From a technical perspective I feel > > there are two (additional) underlying issues worth addressing here: > > > > 1. This could be prevented for the most part by having CI run before, > > and not after the fact. I.e. prior to merging code. > > This will happen once we are on Gitlab. > > > 2. Different projects have different CI needs, and it would help if a > > project could safely manage their CI environment "on their own" as > > much as possible. The current system requires a lot of daunting > > (possibly otherwise unnecessary) complexity purely to manage the fact > > that a builder image will be used not just for one project but for > > perhaps the whole of KDE. > > The global builder image doesn't cause too many issues fortunately and > isn't the problem here. Our current setup actually has the flexibility > to use different images - we're just constrained in terms of setting > this up by Jenkins as there is a bunch of tooling needed to provision > jobs. > > A good proportion of the current "complexity" is due to our use of > Jenkins. The remainder is due to needing to provide the most recent > version of a given KDE project as a dependency, along with the > infrastructure for tests to execute successfully (which is quite a bit > more than just 'make test') > > The need to use the most recent version of a KDE project means either: > a) You have to have all of those projects use the same image as well; or > b) You have a special job to build all of those projects that don't > use the same image (which is what the 'Dependency Build' jobs do); or > c) You incorporate the project within the base system itself > > Option (c) is not possible on Windows or FreeBSD as those platforms do > not support dynamic provisioning of images - meaning that all KDE > projects have to share the same machines and therefore must be > provided via our mechanisms (options a and b above). > > In the case of Plasma, baking in say Frameworks to the image would > mean a full image rebuild everytime you needed something new in a > Framework. Please note that kwindowsystem and plasma-framework are > both Frameworks. Suffice to say, that won't work - you would be > rebuilding that image multiple times a week at a minimum. > > The only way to manage the build environment for something like Plasma > is to do it on a Product level, taking all of it's member projects > into account. This is what we do currently - while the actual Docker > image may happen to be shared with Frameworks and Applications, that > isn't required and could be seperated within our existing > architecture. > Or use option (d): a repository of artifacts which you push to/pull from. The problem I see with that right now is: we would need "package management" and our projects use mostly plain CMake which by itself doesn't understand this concept. So the standard tooling for our projects can only check for declared dependencies, but cannot try to install them locally. As I understand it, that is effectively what a good chunk of the CI tooling sort of replicates anyway: a custom built package management solution for KDE CI. Out of interest, apart from the amount of work it might take what would be the main blocker for using VMs and things like Vagrant boxes for FreeBSD as the next best thing to containers? I'd ask the same for Windows, but I imagine the answer is licensing costs. > > That isn't CI. That is CD - as it results in an artifact for use on > developers/testers/end user systems. > In the sense that if you were to push to a package repository it entails a deployment. But building binaries by itself obviously isn't, even if you expose these as downloadable artifacts. > > The infrastructure for Android builds has grown organically, and I > suspect we are at a point where we do need to take a step back and ask > ourselves if there is a better way of doing this that is more > maintainable. The dependencies bit in particular definitely needs > work. > Agreed. And I understand that there is always something to be said for working code now, rather than possibly working something maybe sometime in the future. > > CI is definitely not easy. Especially not when you are dealing with an > organisation the size of KDE, where projects depend on the latest > development version of other projects. > Oh I agree. I'm not definitely not claiming that any of this is easy or obvious or that what I throw out here will not
