Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu Feb 29, 2024 at 12:55 AM EET, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining > a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and > build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible > 'scenarios' > mechanism for subsystem-specific extensions. > > tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg, > but don't forget to check the "Motivation for this work" below. Your feedback, > whether a simple thumbs up or down, is crucial to determine if it is > worthwhile > to pursue this initiative. > > GitLab is an Open Source platform that includes integrated CI/CD. The pipeline > provided in this patch is designed to work out-of-the-box with any GitLab > instance, including the gitlab.com Free Tier. If you reach the limits of the > Free Tier, consider using community instances like > https://gitlab.freedesktop.org/. > Alternatively, you can set up a local runner for more flexibility. The > bootstrap-gitlab-runner.sh script included with this patch simplifies this > process, enabling you to run tests on your preferred infrastructure, including > your own machine. > > For detailed information, please refer to the documentation included in the > patch, or check the rendered version here: > https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html > . > > > Motivation for this Work > > > We all know tests are a major topic in the community, so let's mention the > specificities of this approach: > > 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has > an > user-friendly interface. Our experience with the upstream DRM-CI in the kernel > tree (see this blog post > [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] > ) > has provided insights into how such a system can benefit the wider community. > > 2. **Distributed Infrastructure:** > The proposed GitLab-CI pipeline is designed with a distributed infrastructure > model, being possible to run in any gitlab instance. > > 3. **Reduce regressions:** Fostering a culture where people habitually run > validated tests and post their results can prevent many issues in post-merge > tests. > > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines > such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. > > 5. **Ownership of QA:** > Discrepancies between kernel code and outdated tests often lead to > misattributed > failures, complicating regression tracking. This issue, often arising from > neglected or deprioritized test updates, creates uncertainty about the source > of > failures. Adopting an "always green pipeline" approach, as detailed in this > patch's documentation, encourages timely maintenance and validation of tests. > This ensures that testing accurately reflects the current state of the kernel, > thereby improving the effectiveness of our QA processes. > > Additionally, if we discover that this method isn't working for us, we can > easily remove it from the codebase, as it is primarily contained within the > ci/ > folder. Not to criticize but I can do tests I ever want with either Github or Gitlab simply by bootstrapping BuildRoot on top of whatever the CI runner has. So I essentially need just enough deps to make a BR build, and that's it. And e.g. could run x86 tests on ARM ISA runner with zero issues. And can even have emulated TPM chip in the QEMU VM by building swtpm. I had this for some time running actually Gitlab runner. It does not currently build QEMU but then it also did that: https://gitlab.com/jarkkojs/linux-tpmdd-test Essentially just executing this sequence: git clone https://gitlab.com/jarkkojs/linux-tpmdd-test.git cd linux-tpmdd-test cmake -Bbuild && make -Cbuild buildroot-prepare make -Cbuild/buildroot/build build/buildroot/build/images/run-tests.sh I use TCL's "expect" to make conclusions from the output :-) I'm assuming that this has a bigger point that I can understand right now but makes me a bit puzzled given that it is quite trivial problem to my understanding (if you want to pursue to it). Like one work week maybe but not more than that... Especially it feels weird that it needs kernel to be patched at all and when I did read the motivation but it has sort of whitepaperish stuff that does not really explain me the edge of this
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 02/03/2024 10:48 pm, Gustavo Padovan wrote: On Friday, March 01, 2024 18:56 -03, Guillaume Tucker wrote: On 29/02/2024 17:28, Nicolas Dufresne wrote: Hi, Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : On 2/29/24 2:20 PM, Guillaume Tucker wrote: Hello, On 28/02/2024 23:55, Helen Koike wrote: Dear Kernel Community, This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a basic test pipeline triggered by code pushes to a GitLab-CI instance. This initial version includes static checks (checkpatch and smatch for now) and build tests across various architectures and configurations. It leverages an integrated cache for efficient build times and introduces a flexible 'scenarios' mechanism for subsystem-specific extensions. This sounds like a nice starting point to me as an additional way to run tests upstream. I have one particular question as I see a pattern through the rest of the email, please see below. [...] 4. **Collaborative Testing Environment:** The kernel community is already engaged in numerous testing efforts, including various GitLab-CI pipelines such as DRM-CI, which I maintain, along with other solutions like KernelCI and BPF-CI. This proposal is designed to further stimulate contributions to the evolving testing landscape. Our goal is to establish a comprehensive suite of common tools and files. [...] **Leveraging External Test Labs:** We can extend our testing to external labs, similar to what DRM-CI currently does. This includes: - Lava labs - Bare metal labs - Using KernelCI-provided labs **Other integrations** - Submit results to KCIDB [...] **Join Our Slack Channel:** We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . Feel free to join and contribute to the conversation. The KernelCI team has weekly calls where we also discuss the GitLab-CI pipeline. **Acknowledgments:** A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - and KernelCI community for their valuable feedback and support in this proposal. Where does this fit on the KernelCI roadmap? I see it mentioned a few times but it's not entirely clear whether this initiative is an independent one or in some way linked to KernelCI. Say, are you planning to use the kci tool, new API, compiler toolchains, user-space and Docker images etc? Or, are KernelCI plans evolving to follow this move? I would say this is an important part of KernelCI the project, considering its aim to improve testing and CI in the kernel. It's not a part of KernelCI the service as it is right now, although I would say it would be good to have ability to submit KernelCI jobs from GitLab CI and pull results in the same pipeline, as we discussed earlier. Right, I think this needs a bit of disambiguation. The legacy KernelCI system from the Linaro days several years ago is really a service on its own like the many other CIs out there. However, the new KernelCI API and related tooling (kci command line, new web dashboard, modular runtime design etc.) is not that. It's about addressing all the community requirements and that includes being able to run a same test manually in a shell, or in a VM, or automatically from GitLab CI or using a main generic pipeline hosted by KernelCI itself. With this approach, there's no distinction between "the project" and "the service", and as we discussed before there shouldn't even be a distinction with KCIDB. Just KernelCI. However I don't really see this happening, unless I'm missing a part of the story or some upcoming announcement with an updated roadmap. For some reason the old and established paradigm seems unshakeable. The new KernelCI implementation is starting to look just like a refresh of the old one with newer components - which is a huge missed opportunity to really change things IMHO. Calling that a missed opportunity is a subjective perspective about the latest developments in KernelCI. The system implementation is one level less important than the actual kernel community engagement the project can generate. If one asks people around, the lack of community engagement with KernelCI is evident. Well I would argue that community engagement and technical development work side-by-side, not as a hierarchy. You can't run Android phones or data centers with community engagement, and you can't write the code without the people. I was enquiring about this in particluar because I'm preparing a LF webinar, so I've started another thread to avoid spamming this one as it's really a side topic: https://lore.kernel.org/all/71f59a56-aef3-4bae-867b-769a0cdd1...@gtucker.io/T/#u However, after the recent leadership change in the project there is a growing effort to bring the kernel community closer to the KernelCI project with a renewed focus on high quality test results, clean regression reporting, among other things. Then, with an increased number of community members
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 29/02/2024 17:28, Nicolas Dufresne wrote: > Hi, > > Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : >> On 2/29/24 2:20 PM, Guillaume Tucker wrote: >>> Hello, >>> >>> On 28/02/2024 23:55, Helen Koike wrote: Dear Kernel Community, This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a basic test pipeline triggered by code pushes to a GitLab-CI instance. This initial version includes static checks (checkpatch and smatch for now) and build tests across various architectures and configurations. It leverages an integrated cache for efficient build times and introduces a flexible 'scenarios' mechanism for subsystem-specific extensions. >>> >>> This sounds like a nice starting point to me as an additional way >>> to run tests upstream. I have one particular question as I see a >>> pattern through the rest of the email, please see below. >>> >>> [...] >>> 4. **Collaborative Testing Environment:** The kernel community is already engaged in numerous testing efforts, including various GitLab-CI pipelines such as DRM-CI, which I maintain, along with other solutions like KernelCI and BPF-CI. This proposal is designed to further stimulate contributions to the evolving testing landscape. Our goal is to establish a comprehensive suite of common tools and files. >>> >>> [...] >>> **Leveraging External Test Labs:** We can extend our testing to external labs, similar to what DRM-CI currently does. This includes: - Lava labs - Bare metal labs - Using KernelCI-provided labs **Other integrations** - Submit results to KCIDB >>> >>> [...] >>> **Join Our Slack Channel:** We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . Feel free to join and contribute to the conversation. The KernelCI team has weekly calls where we also discuss the GitLab-CI pipeline. **Acknowledgments:** A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - and KernelCI community for their valuable feedback and support in this proposal. >>> >>> Where does this fit on the KernelCI roadmap? >>> >>> I see it mentioned a few times but it's not entirely clear >>> whether this initiative is an independent one or in some way >>> linked to KernelCI. Say, are you planning to use the kci tool, >>> new API, compiler toolchains, user-space and Docker images etc? >>> Or, are KernelCI plans evolving to follow this move? >> >> I would say this is an important part of KernelCI the project, considering >> its >> aim to improve testing and CI in the kernel. It's not a part of KernelCI the >> service as it is right now, although I would say it would be good to have >> ability to submit KernelCI jobs from GitLab CI and pull results in the same >> pipeline, as we discussed earlier. Right, I think this needs a bit of disambiguation. The legacy KernelCI system from the Linaro days several years ago is really a service on its own like the many other CIs out there. However, the new KernelCI API and related tooling (kci command line, new web dashboard, modular runtime design etc.) is not that. It's about addressing all the community requirements and that includes being able to run a same test manually in a shell, or in a VM, or automatically from GitLab CI or using a main generic pipeline hosted by KernelCI itself. With this approach, there's no distinction between "the project" and "the service", and as we discussed before there shouldn't even be a distinction with KCIDB. Just KernelCI. However I don't really see this happening, unless I'm missing a part of the story or some upcoming announcement with an updated roadmap. For some reason the old and established paradigm seems unshakeable. The new KernelCI implementation is starting to look just like a refresh of the old one with newer components - which is a huge missed opportunity to really change things IMHO. This may sound like a bit of a tangent, facilitating GitLab CI for the upstream kernel is of course significant progress in any case - no question about that. My comment is more about why it's being driven hand-in-hand with KernelCI in what seems like a diverging direction from KernelCI's announced plans. Why push for a GitLab-centered orchestration when there's a more universal solution being proposed by the project? I would find it easier to understand - and I sense I'm not the only one here reading the thread - if KernelCI wasn't mentioned that many times in the cover letter and if the scripts didn't have KCI_* in so many places, basically if this was clearly an independent initiative such as KUnit, 0-day or regzbot. > I'd like to add that both CI have a different purpose in the Linux project. > This > CI work is a pre-merge verification. Everyone needs to run checkpatch and >
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Friday, March 01, 2024 18:56 -03, Guillaume Tucker wrote: > On 29/02/2024 17:28, Nicolas Dufresne wrote: > > Hi, > > > > Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : > >> On 2/29/24 2:20 PM, Guillaume Tucker wrote: > >>> Hello, > >>> > >>> On 28/02/2024 23:55, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, > defining a > basic test pipeline triggered by code pushes to a GitLab-CI instance. > This > initial version includes static checks (checkpatch and smatch for now) > and build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible > 'scenarios' > mechanism for subsystem-specific extensions. > >>> > >>> This sounds like a nice starting point to me as an additional way > >>> to run tests upstream. I have one particular question as I see a > >>> pattern through the rest of the email, please see below. > >>> > >>> [...] > >>> > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI > pipelines such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to > the > evolving testing landscape. Our goal is to establish a comprehensive > suite of > common tools and files. > >>> > >>> [...] > >>> > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI > currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB > >>> > >>> [...] > >>> > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team > has > weekly calls where we also discuss the GitLab-CI pipeline. > > **Acknowledgments:** > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from > Red Hat - > and KernelCI community for their valuable feedback and support in this > proposal. > >>> > >>> Where does this fit on the KernelCI roadmap? > >>> > >>> I see it mentioned a few times but it's not entirely clear > >>> whether this initiative is an independent one or in some way > >>> linked to KernelCI. Say, are you planning to use the kci tool, > >>> new API, compiler toolchains, user-space and Docker images etc? > >>> Or, are KernelCI plans evolving to follow this move? > >> > >> I would say this is an important part of KernelCI the project, considering > >> its > >> aim to improve testing and CI in the kernel. It's not a part of KernelCI > >> the > >> service as it is right now, although I would say it would be good to have > >> ability to submit KernelCI jobs from GitLab CI and pull results in the > >> same > >> pipeline, as we discussed earlier. > > Right, I think this needs a bit of disambiguation. The legacy > KernelCI system from the Linaro days several years ago is really > a service on its own like the many other CIs out there. However, > the new KernelCI API and related tooling (kci command line, new > web dashboard, modular runtime design etc.) is not that. It's > about addressing all the community requirements and that includes > being able to run a same test manually in a shell, or in a VM, or > automatically from GitLab CI or using a main generic pipeline > hosted by KernelCI itself. With this approach, there's no > distinction between "the project" and "the service", and as we > discussed before there shouldn't even be a distinction with > KCIDB. Just KernelCI. > > However I don't really see this happening, unless I'm missing a > part of the story or some upcoming announcement with an updated > roadmap. For some reason the old and established paradigm seems > unshakeable. The new KernelCI implementation is starting to look > just like a refresh of the old one with newer components - which > is a huge missed opportunity to really change things IMHO. Calling that a missed opportunity is a subjective perspective about the latest developments in KernelCI. The system implementation is one level less important than the actual kernel community engagement the project can generate. If one asks people around, the lack of community engagement with KernelCI is evident. However, after the recent leadership change in the project there is a growing effort to bring the kernel community closer to the KernelCI project with a renewed focus on high quality test results, clean regression reporting, among other things. Then, with an increased number of community
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
Hi, Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : > On 2/29/24 2:20 PM, Guillaume Tucker wrote: > > Hello, > > > > On 28/02/2024 23:55, Helen Koike wrote: > > > Dear Kernel Community, > > > > > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, > > > defining a > > > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > > > initial version includes static checks (checkpatch and smatch for now) > > > and build > > > tests across various architectures and configurations. It leverages an > > > integrated cache for efficient build times and introduces a flexible > > > 'scenarios' > > > mechanism for subsystem-specific extensions. > > > > This sounds like a nice starting point to me as an additional way > > to run tests upstream. I have one particular question as I see a > > pattern through the rest of the email, please see below. > > > > [...] > > > > > 4. **Collaborative Testing Environment:** The kernel community is already > > > engaged in numerous testing efforts, including various GitLab-CI > > > pipelines such > > > as DRM-CI, which I maintain, along with other solutions like KernelCI and > > > BPF-CI. This proposal is designed to further stimulate contributions to > > > the > > > evolving testing landscape. Our goal is to establish a comprehensive > > > suite of > > > common tools and files. > > > > [...] > > > > > **Leveraging External Test Labs:** > > > We can extend our testing to external labs, similar to what DRM-CI > > > currently > > > does. This includes: > > > - Lava labs > > > - Bare metal labs > > > - Using KernelCI-provided labs > > > > > > **Other integrations** > > > - Submit results to KCIDB > > > > [...] > > > > > **Join Our Slack Channel:** > > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > > > https://kernelci.slack.com/ . > > > Feel free to join and contribute to the conversation. The KernelCI team > > > has > > > weekly calls where we also discuss the GitLab-CI pipeline. > > > > > > **Acknowledgments:** > > > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from > > > Red Hat - > > > and KernelCI community for their valuable feedback and support in this > > > proposal. > > > > Where does this fit on the KernelCI roadmap? > > > > I see it mentioned a few times but it's not entirely clear > > whether this initiative is an independent one or in some way > > linked to KernelCI. Say, are you planning to use the kci tool, > > new API, compiler toolchains, user-space and Docker images etc? > > Or, are KernelCI plans evolving to follow this move? > > I would say this is an important part of KernelCI the project, considering > its > aim to improve testing and CI in the kernel. It's not a part of KernelCI the > service as it is right now, although I would say it would be good to have > ability to submit KernelCI jobs from GitLab CI and pull results in the same > pipeline, as we discussed earlier. I'd like to add that both CI have a different purpose in the Linux project. This CI work is a pre-merge verification. Everyone needs to run checkpatch and smatch, this is automating it (and will catch those that forgot or ran it incorrectly). But it can go further by effectively testing specific patches on real hardware (with pretty narrow filters). It will help catch submission issues earlier, and reduce kernelCI regression rate. As a side effect, kernelCI infra will endup catching the "integration" issues, which are the issue as a result of simultenous changes in different trees. They are also often more complex and benefit from the bisection capabilities. kernelCI tests are also a lot more intensive, they usually covers everything, but they bundle multiple changes per run. The pre-merge tests will be reduced to what seems meaningful for the changes. Its important to understand that pre- merge CI have a time cost, and we need to make sure CI time does not exceed the merge window period. Nicolas
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 2/29/24 01:07, Laurent Pinchart wrote: On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: **Join Our Slack Channel:** We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . Feel free to join and contribute to the conversation. The KernelCI team has weekly calls where we also discuss the GitLab-CI pipeline. Could we communicate using free software please ? Furthermore, it's not possible to create an account on that slack instance unless you have an e-mail address affiliated with a small number of companies (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go for me. Yes, it's not ideal that we use closed-source software for communication, but FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with a Google account, I'd be able to see and approve your attempt too. Otherwise, our video meetings are generally open to everyone and run in Jitsi. Nick
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
Hi Laurent, Helen, On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > **Join Our Slack Channel:** > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > > https://kernelci.slack.com/ . > > Feel free to join and contribute to the conversation. The KernelCI team has > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > possible to create an account on that slack instance unless you have an > e-mail address affiliated with a small number of companies > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > for me. I agree. There is no shortage of good alternatives either: we have IRC, Matrix and XMPP for instance. Just pick one of them. -- Regards, Sakari Ailus
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 2/29/24 2:25 PM, Laurent Pinchart wrote: On Thu, Feb 29, 2024 at 02:20:41PM +0200, Laurent Pinchart wrote: On Thu, Feb 29, 2024 at 12:53:38PM +0100, Guillaume Tucker wrote: On 29/02/2024 12:41, Mark Brown wrote: On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: Of course. You're also welcome to join the #kernelci channel on libera.chat. Isn't that a bit pointless if it's no the main IM channel ? It *was* the original channel and still gets some usage (mostly started by me admittedly since I've never joined slack for a bunch of reasons that make it hassle), IIRC the Slack was started because there were some interns who had trouble figuring out IRC and intermittent connectivity but people seem to have migrated. In fact it was initially created for the members of the Linux Foundation project only, which is why registration is moderated for emails that don't have a domain linked to a member (BTW not any Google account will just work e.g. @gmail.com is moderated, only @google.com for Google employees isn't). And yes IRC is the "least common denominator" chat platform. Maybe having a bridge between the main Slack channel and IRC would help. If the gitlab CI pipeline proposal wants to be considered for inclusion in the kernel, I think it needs to switch to a free software solution for its *main* communication channels. And to clarify, I didn't meant the kernel CI project, but only the gitlab CI pipeline for the Linux kernel project. I don't know how tightly integrated the two projects are though. They're not tightly integrated so far. However, we're exploring the idea of letting GitLab CI submit jobs to KernelCI and get results as a part of the pipeline. Helen already joined the #kernelci channel, and we will maintain a presence there for the GitLab CI project. Nick
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
Hello, On 28/02/2024 23:55, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining > a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and > build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible > 'scenarios' > mechanism for subsystem-specific extensions. This sounds like a nice starting point to me as an additional way to run tests upstream. I have one particular question as I see a pattern through the rest of the email, please see below. [...] > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines > such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. [...] > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB [...] > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team has > weekly calls where we also discuss the GitLab-CI pipeline. > > **Acknowledgments:** > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red > Hat - > and KernelCI community for their valuable feedback and support in this > proposal. Where does this fit on the KernelCI roadmap? I see it mentioned a few times but it's not entirely clear whether this initiative is an independent one or in some way linked to KernelCI. Say, are you planning to use the kci tool, new API, compiler toolchains, user-space and Docker images etc? Or, are KernelCI plans evolving to follow this move? Thanks, Guillaume
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 2/29/24 1:19 PM, Laurent Pinchart wrote: On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: On 2/29/24 11:34 AM, Laurent Pinchart wrote: On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: On 2/29/24 01:07, Laurent Pinchart wrote: On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: **Join Our Slack Channel:** We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . Feel free to join and contribute to the conversation. The KernelCI team has weekly calls where we also discuss the GitLab-CI pipeline. Could we communicate using free software please ? Furthermore, it's not possible to create an account on that slack instance unless you have an e-mail address affiliated with a small number of companies (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go for me. Yes, it's not ideal that we use closed-source software for communication, but FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with a Google account, I'd be able to see and approve your attempt too. I don't use Google accounts to authenticate to third-party services, sorry. And in any case, that won't make slack free software. Of course. You're also welcome to join the #kernelci channel on libera.chat. Isn't that a bit pointless if it's no the main IM channel ? Yes, it's not ideal, but if more people come there, more discussions will happen there too. Nick
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 2/29/24 11:34 AM, Laurent Pinchart wrote: On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: On 2/29/24 01:07, Laurent Pinchart wrote: On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: **Join Our Slack Channel:** We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . Feel free to join and contribute to the conversation. The KernelCI team has weekly calls where we also discuss the GitLab-CI pipeline. Could we communicate using free software please ? Furthermore, it's not possible to create an account on that slack instance unless you have an e-mail address affiliated with a small number of companies (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go for me. Yes, it's not ideal that we use closed-source software for communication, but FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with a Google account, I'd be able to see and approve your attempt too. I don't use Google accounts to authenticate to third-party services, sorry. And in any case, that won't make slack free software. Of course. You're also welcome to join the #kernelci channel on libera.chat. It's much quieter there, though. Nick
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 2/29/24 2:20 PM, Guillaume Tucker wrote: Hello, On 28/02/2024 23:55, Helen Koike wrote: Dear Kernel Community, This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a basic test pipeline triggered by code pushes to a GitLab-CI instance. This initial version includes static checks (checkpatch and smatch for now) and build tests across various architectures and configurations. It leverages an integrated cache for efficient build times and introduces a flexible 'scenarios' mechanism for subsystem-specific extensions. This sounds like a nice starting point to me as an additional way to run tests upstream. I have one particular question as I see a pattern through the rest of the email, please see below. [...] 4. **Collaborative Testing Environment:** The kernel community is already engaged in numerous testing efforts, including various GitLab-CI pipelines such as DRM-CI, which I maintain, along with other solutions like KernelCI and BPF-CI. This proposal is designed to further stimulate contributions to the evolving testing landscape. Our goal is to establish a comprehensive suite of common tools and files. [...] **Leveraging External Test Labs:** We can extend our testing to external labs, similar to what DRM-CI currently does. This includes: - Lava labs - Bare metal labs - Using KernelCI-provided labs **Other integrations** - Submit results to KCIDB [...] **Join Our Slack Channel:** We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . Feel free to join and contribute to the conversation. The KernelCI team has weekly calls where we also discuss the GitLab-CI pipeline. **Acknowledgments:** A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - and KernelCI community for their valuable feedback and support in this proposal. Where does this fit on the KernelCI roadmap? I see it mentioned a few times but it's not entirely clear whether this initiative is an independent one or in some way linked to KernelCI. Say, are you planning to use the kci tool, new API, compiler toolchains, user-space and Docker images etc? Or, are KernelCI plans evolving to follow this move? I would say this is an important part of KernelCI the project, considering its aim to improve testing and CI in the kernel. It's not a part of KernelCI the service as it is right now, although I would say it would be good to have ability to submit KernelCI jobs from GitLab CI and pull results in the same pipeline, as we discussed earlier. Nick
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On 29/02/2024 12:41, Mark Brown wrote: > On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: >> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > >>> Of course. You're also welcome to join the #kernelci channel on libera.chat. > >> Isn't that a bit pointless if it's no the main IM channel ? > > It *was* the original channel and still gets some usage (mostly started > by me admittedly since I've never joined slack for a bunch of reasons > that make it hassle), IIRC the Slack was started because there were some > interns who had trouble figuring out IRC and intermittent connectivity > but people seem to have migrated. In fact it was initially created for the members of the Linux Foundation project only, which is why registration is moderated for emails that don't have a domain linked to a member (BTW not any Google account will just work e.g. @gmail.com is moderated, only @google.com for Google employees isn't). And yes IRC is the "least common denominator" chat platform. Maybe having a bridge between the main Slack channel and IRC would help. Guillaume
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu, Feb 29, 2024 at 02:20:41PM +0200, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 12:53:38PM +0100, Guillaume Tucker wrote: > > On 29/02/2024 12:41, Mark Brown wrote: > > > On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: > > >> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > > > > > >>> Of course. You're also welcome to join the #kernelci channel on > > >>> libera.chat. > > > > > >> Isn't that a bit pointless if it's no the main IM channel ? > > > > > > It *was* the original channel and still gets some usage (mostly started > > > by me admittedly since I've never joined slack for a bunch of reasons > > > that make it hassle), IIRC the Slack was started because there were some > > > interns who had trouble figuring out IRC and intermittent connectivity > > > but people seem to have migrated. > > > > In fact it was initially created for the members of the Linux > > Foundation project only, which is why registration is moderated > > for emails that don't have a domain linked to a member (BTW not > > any Google account will just work e.g. @gmail.com is moderated, > > only @google.com for Google employees isn't). > > > > And yes IRC is the "least common denominator" chat platform. > > Maybe having a bridge between the main Slack channel and IRC > > would help. > > If the gitlab CI pipeline proposal wants to be considered for inclusion > in the kernel, I think it needs to switch to a free software solution > for its *main* communication channels. And to clarify, I didn't meant the kernel CI project, but only the gitlab CI pipeline for the Linux kernel project. I don't know how tightly integrated the two projects are though. -- Regards, Laurent Pinchart
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu, Feb 29, 2024 at 12:53:38PM +0100, Guillaume Tucker wrote: > On 29/02/2024 12:41, Mark Brown wrote: > > On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: > >> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > > > >>> Of course. You're also welcome to join the #kernelci channel on > >>> libera.chat. > > > >> Isn't that a bit pointless if it's no the main IM channel ? > > > > It *was* the original channel and still gets some usage (mostly started > > by me admittedly since I've never joined slack for a bunch of reasons > > that make it hassle), IIRC the Slack was started because there were some > > interns who had trouble figuring out IRC and intermittent connectivity > > but people seem to have migrated. > > In fact it was initially created for the members of the Linux > Foundation project only, which is why registration is moderated > for emails that don't have a domain linked to a member (BTW not > any Google account will just work e.g. @gmail.com is moderated, > only @google.com for Google employees isn't). > > And yes IRC is the "least common denominator" chat platform. > Maybe having a bridge between the main Slack channel and IRC > would help. If the gitlab CI pipeline proposal wants to be considered for inclusion in the kernel, I think it needs to switch to a free software solution for its *main* communication channels. -- Regards, Laurent Pinchart
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > > Of course. You're also welcome to join the #kernelci channel on libera.chat. > Isn't that a bit pointless if it's no the main IM channel ? It *was* the original channel and still gets some usage (mostly started by me admittedly since I've never joined slack for a bunch of reasons that make it hassle), IIRC the Slack was started because there were some interns who had trouble figuring out IRC and intermittent connectivity but people seem to have migrated. signature.asc Description: PGP signature
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > On 2/29/24 11:34 AM, Laurent Pinchart wrote: > > On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: > >> On 2/29/24 01:07, Laurent Pinchart wrote: > >>> On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team > has > weekly calls where we also discuss the GitLab-CI pipeline. > >>> > >>> Could we communicate using free software please ? Furthermore, it's not > >>> possible to create an account on that slack instance unless you have an > >>> e-mail address affiliated with a small number of companies > >>> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > >>> for me. > >> > >> Yes, it's not ideal that we use closed-source software for communication, > >> but > >> FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. > >> with > >> a Google account, I'd be able to see and approve your attempt too. > > > > I don't use Google accounts to authenticate to third-party services, > > sorry. And in any case, that won't make slack free software. > > Of course. You're also welcome to join the #kernelci channel on libera.chat. Isn't that a bit pointless if it's no the main IM channel ? > It's much quieter there, though. -- Regards, Laurent Pinchart
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu, Feb 29, 2024 at 09:39:01AM +, Sakari Ailus wrote: > On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > > > https://kernelci.slack.com/ . > > > Feel free to join and contribute to the conversation. The KernelCI team > > > has > > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > > possible to create an account on that slack instance unless you have an > > e-mail address affiliated with a small number of companies > > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > > for me. > I agree. There is no shortage of good alternatives either: we have IRC, > Matrix and XMPP for instance. Just pick one of them. And indeed KernelCI does actually already have #kernelci on libera.chat. signature.asc Description: PGP signature
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: > On 2/29/24 01:07, Laurent Pinchart wrote: > > On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > >> **Join Our Slack Channel:** > >> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > >> https://kernelci.slack.com/ . > >> Feel free to join and contribute to the conversation. The KernelCI team has > >> weekly calls where we also discuss the GitLab-CI pipeline. > > > > Could we communicate using free software please ? Furthermore, it's not > > possible to create an account on that slack instance unless you have an > > e-mail address affiliated with a small number of companies > > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > > for me. > > Yes, it's not ideal that we use closed-source software for communication, but > FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. > with > a Google account, I'd be able to see and approve your attempt too. I don't use Google accounts to authenticate to third-party services, sorry. And in any case, that won't make slack free software. > Otherwise, our video meetings are generally open to everyone and run in Jitsi. -- Regards, Laurent Pinchart
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > Chat Discussions > > > > > > For those interested in further discussions: > > > > **Join Our Slack Channel:** > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance > > https://kernelci.slack.com/ . > > Feel free to join and contribute to the conversation. The KernelCI team has > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > possible to create an account on that slack instance unless you have an > e-mail address affiliated with a small number of companies > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > for me. Yeah, and that list looks super restrictive and arbitrary. Like, microsoft is there but kernel.org isn't? I'm sure there's a reason, but we should at the very least open it to everyone. Maxime signature.asc Description: PGP signature
Re: [PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
Hi Helen, I appreciate the amount of work you've put into this, to improve testing of the kernel as a whole. I'll need more time to answer, but please see below for a quick comment already. On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining > a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and > build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible > 'scenarios' > mechanism for subsystem-specific extensions. > > tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg, > but don't forget to check the "Motivation for this work" below. Your feedback, > whether a simple thumbs up or down, is crucial to determine if it is > worthwhile > to pursue this initiative. > > GitLab is an Open Source platform that includes integrated CI/CD. The pipeline > provided in this patch is designed to work out-of-the-box with any GitLab > instance, including the gitlab.com Free Tier. If you reach the limits of the > Free Tier, consider using community instances like > https://gitlab.freedesktop.org/. > Alternatively, you can set up a local runner for more flexibility. The > bootstrap-gitlab-runner.sh script included with this patch simplifies this > process, enabling you to run tests on your preferred infrastructure, including > your own machine. > > For detailed information, please refer to the documentation included in the > patch, or check the rendered version here: > https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html > . > > > Motivation for this Work > > > We all know tests are a major topic in the community, so let's mention the > specificities of this approach: > > 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has > an > user-friendly interface. Our experience with the upstream DRM-CI in the kernel > tree (see this blog post > [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] > ) > has provided insights into how such a system can benefit the wider community. > > 2. **Distributed Infrastructure:** > The proposed GitLab-CI pipeline is designed with a distributed infrastructure > model, being possible to run in any gitlab instance. > > 3. **Reduce regressions:** Fostering a culture where people habitually run > validated tests and post their results can prevent many issues in post-merge > tests. > > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines > such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. > > 5. **Ownership of QA:** > Discrepancies between kernel code and outdated tests often lead to > misattributed > failures, complicating regression tracking. This issue, often arising from > neglected or deprioritized test updates, creates uncertainty about the source > of > failures. Adopting an "always green pipeline" approach, as detailed in this > patch's documentation, encourages timely maintenance and validation of tests. > This ensures that testing accurately reflects the current state of the kernel, > thereby improving the effectiveness of our QA processes. > > Additionally, if we discover that this method isn't working for us, we can > easily remove it from the codebase, as it is primarily contained within the > ci/ > folder. > > > Future Work > === > > **Expanding Static Checks:** > We have the opportunity to integrate a variety of static analysis tools, > including: > - dtbs_checks > - sparse > - yamllint > - dt-doc-validate > - coccicheck > > **Adding Userspace Tests on VMs:** > To further our testing, we can implement userspace tests that run on virtual > machines (VMs), such as: > - kselftests > - kunit tests > - Subsystem-specific tests, customizable in the scenarios. > > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB > > **Lightweight Implementation for All Developers:** > We aim to design these tests to be lightweight, ensuring developers with > limited > computing resources can still run essential tests. Resource-intensive tests > can > be set to trigger manually, rather than automatically, to accommodate diverse > development
[PATCH 0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
Dear Kernel Community, This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a basic test pipeline triggered by code pushes to a GitLab-CI instance. This initial version includes static checks (checkpatch and smatch for now) and build tests across various architectures and configurations. It leverages an integrated cache for efficient build times and introduces a flexible 'scenarios' mechanism for subsystem-specific extensions. tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg, but don't forget to check the "Motivation for this work" below. Your feedback, whether a simple thumbs up or down, is crucial to determine if it is worthwhile to pursue this initiative. GitLab is an Open Source platform that includes integrated CI/CD. The pipeline provided in this patch is designed to work out-of-the-box with any GitLab instance, including the gitlab.com Free Tier. If you reach the limits of the Free Tier, consider using community instances like https://gitlab.freedesktop.org/. Alternatively, you can set up a local runner for more flexibility. The bootstrap-gitlab-runner.sh script included with this patch simplifies this process, enabling you to run tests on your preferred infrastructure, including your own machine. For detailed information, please refer to the documentation included in the patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html . Motivation for this Work We all know tests are a major topic in the community, so let's mention the specificities of this approach: 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an user-friendly interface. Our experience with the upstream DRM-CI in the kernel tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] ) has provided insights into how such a system can benefit the wider community. 2. **Distributed Infrastructure:** The proposed GitLab-CI pipeline is designed with a distributed infrastructure model, being possible to run in any gitlab instance. 3. **Reduce regressions:** Fostering a culture where people habitually run validated tests and post their results can prevent many issues in post-merge tests. 4. **Collaborative Testing Environment:** The kernel community is already engaged in numerous testing efforts, including various GitLab-CI pipelines such as DRM-CI, which I maintain, along with other solutions like KernelCI and BPF-CI. This proposal is designed to further stimulate contributions to the evolving testing landscape. Our goal is to establish a comprehensive suite of common tools and files. 5. **Ownership of QA:** Discrepancies between kernel code and outdated tests often lead to misattributed failures, complicating regression tracking. This issue, often arising from neglected or deprioritized test updates, creates uncertainty about the source of failures. Adopting an "always green pipeline" approach, as detailed in this patch's documentation, encourages timely maintenance and validation of tests. This ensures that testing accurately reflects the current state of the kernel, thereby improving the effectiveness of our QA processes. Additionally, if we discover that this method isn't working for us, we can easily remove it from the codebase, as it is primarily contained within the ci/ folder. Future Work === **Expanding Static Checks:** We have the opportunity to integrate a variety of static analysis tools, including: - dtbs_checks - sparse - yamllint - dt-doc-validate - coccicheck **Adding Userspace Tests on VMs:** To further our testing, we can implement userspace tests that run on virtual machines (VMs), such as: - kselftests - kunit tests - Subsystem-specific tests, customizable in the scenarios. **Leveraging External Test Labs:** We can extend our testing to external labs, similar to what DRM-CI currently does. This includes: - Lava labs - Bare metal labs - Using KernelCI-provided labs **Other integrations** - Submit results to KCIDB **Lightweight Implementation for All Developers:** We aim to design these tests to be lightweight, ensuring developers with limited computing resources can still run essential tests. Resource-intensive tests can be set to trigger manually, rather than automatically, to accommodate diverse development environments. Chat Discussions For those interested in further discussions: **Join Our Slack Channel:** We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . Feel free to join and contribute to the conversation. The KernelCI team has weekly calls where we also discuss the GitLab-CI pipeline. **Acknowledgments:** A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - and KernelCI community for their
