On Fri, Aug 25, 2023 at 1:47 AM Scarlett Moore < scarlett.gately.mo...@gmail.com> wrote:
> Hi all, > Hi Scarlett, > First I apologize for starting this discussion without a clear intent. > I thought it was a simple request as flatpak manifests are already in > repos, why not snapcraft files. > I think we're all pretty okay with Snapcraft files being in the repositories, however the expectation was that this would result in us getting CI for Snaps. This was the gap that was not communicated well. > > SO first here are the problems I am trying to solve. > > a) I ( or any future snapcrafter snapping KDE applications ) would > like a closer relationship with developers and their apps/ snaps. > Makes sense. To give a bit of background to how we ended up with .kde-ci.yml, .flatpak-manifest.yml and soon to be .craft.ini files in our repositories, one of the big issues we've faced with CI prior to Gitlab was how to version things like dependencies and other settings. Storing this information outside the repository ( within the CI system itself ) means we have to carry around all the variants for the various versions in use, and ensure that we design in support to handle that. This never worked particularly well, and with Gitlab having support for an unlimited set of branch variants, a different approach was needed - hence why they ended up going into the repository itself (which has a number of other benefits as well, like letting developers update it all in one MR when making a change that changes dependencies, etc) This only works though when the necessary recipe (.flatpak-manifest.yml, .kde-ci.yml) refers to the sources in the folder that it sits in though - and not to some other copy such as a tarball. > b) Automated builds and publishing via launchpad. > 1) Launchpad does not support our current all in one repo in folders > setup. > 2) Our current method for builds sends off the snapcraft files to > launchpad via a public remote build in which launchpad creates a > temporary snap recipe, does the build, and sends back the snaps and > our CI then uploads them to the store. This is clunky at best for many > reasons. Those temporary snap recipes are last on the priority list > for launchpad builders and this results in many failed builds due to > timeouts. In this thread there are complaints about traffic ( this > setup creates loads of traffic sending the files to launchpad and the > receiving sometimes large snaps in return, and then again sending them > out to the store! Having launchpad do all this would reduce traffic > immensely. > This is quite a different topic though, and is where I think things have gotten confused. For Flatpaks, Android APKs, Windows appx we'll be making use of a special process to allow for Continuous Delivery processes to be triggered from Gitlab. It's yet to be fully determined how they'll be triggered. > 3) Support for more architectures. Having launchpad handle the > builds/upload would allow for many architectures without any effort on > our end. Many users ( especially of our SDK ) have requested this. > c) Eventually CI UI testing on the snaps. > > I do not have a clear plan on how I am to achieve the technical path > as of yet as I am in the R&D phase and using blinken as my test > subject. So far my testing has positive results on launchpad mirroring > the github repo and automatic builds on changes which looks like this: > branch -> publish store channel > master -> edge > release/xx.xx -> candidate > They would stay in candidate until the actual release happens, then as > I do now - I test the stable release snap and release to the stable > channel. Obviously this process is time consuming and I would like to > automate it via CI UI testing. > > I am getting many conflicts in this discussion as to building per > commit or release only. With launchpad doing all the builds / > publishing the KDE infrastructure will have no impact as I have > launchpad polling the github mirror. So the answer here is I will be > doing both. > > I did not know flatpak manifests are duplicated ( on build-bot and in > repos? ) Is there something in place that syncs them? > The snap store does have a similiar feature with github builds but > this would require github repos for the snapcraft files and I am not > sure duplication is the best path forward? Especially since the > launchpad method above works. > For Flatpak, we have our ones that are used for CI and nightly builds. There are also release versions though that live in the Flathub repositories and publish to Flathub. I'm not aware of anything to "sync" these. > > I am not sure I understand the need to build a snap on our CI that no > one can touch? Seems using the launchpad API to check the status would > be easier, but if the docker build is a requirement to move forward, > it is supported https://forum.snapcraft.io/t/build-on-docker/4158 You would only need to build a Snap within Docker if you were going to do CI. It seems however that your objective is not CI - it is CD - so that entire point can be ignored. > > > > I hope this clears up any questions or concerns. > Scarlett > Cheers, Ben > > > On Thu, Aug 24, 2023 at 3:33 AM Ben Cooksley <bcooks...@kde.org> wrote: > > > > On Thu, Aug 24, 2023 at 5:00 AM Albert Astals Cid <aa...@kde.org> wrote: > >> > >> El dimecres, 23 d’agost de 2023, a les 18:29:10 (CEST), Scarlett Moore > va escriure: > >> > After some testing and various complicated options, the easiest I have > >> > found is importing our github mirror in launchpad ( this avoids the > >> > polling on kde servers ) and we can create our necessary snap recipes > >> > and KDE infrastructure is unaffected save the already in place github > >> > mirror. Launchpad does everything else ( building and uploading to > >> > store ) I see no reason not to move forward as KDE infrastructure will > >> > have much less involvement in the snap lifecycle aside from the > >> > snapcraft files being in the application repository. As mentioned > >> > before this will allow developers closer involvement with snap > >> > developers. > >> > >> Honestly I'm still a bit lost on what is you are trying to do regarding > Snap and the KDE repos :D > > > > > > I have to admit I am similarly confused. > > > > I thought this was originally about CI, which is why I proposed tying > this in through .gitlab-ci.yml. > > Using that mechanism if this is really about final release builds > doesn't make much sense. > > > > Having a look at what was committed to Blinken's repository though, i'm > not sure there is much that would stop this being actual CI (assuming > someone got it working within Docker)? > > > >> > >> Are we going for CI or CD? > >> > >> CI -> snap is compiled on each commit and MR to make sure we don't > break the snap build > >> > >> CD -> We have an almost-automated way of doing snap "release" builds. > >> > >> > >> To give some examples: > >> > >> This is arianna's KDE flatpak CI > >> https://invent.kde.org/graphics/arianna/-/jobs/1142328 > >> It builds arianna using flatpak with each commit but the artifacts > are "unused" > >> > >> > >> This is Okular's KDE Windows CI > >> https://invent.kde.org/graphics/okular/-/jobs/1142178 > >> It builds each commit to make sure the Windows build doesn't get > broken > >> > >> > >> This is Okular's KDE Windows CD > >> https://binary-factory.kde.org/job/Okular_Release_win64/ > >> It builds release Okular nightly in a way that is a release build > and can be easily uploaded to the Windows Store > >> > >> > >> This is Okular flathub CD > >> https://github.com/flathub/org.kde.okular > >> https://buildbot.flathub.org/#/apps/org.kde.okular > >> It is a convenient way to generate new builds each time a stable > release happens > >> > >> > >> I see you committed this > >> https://invent.kde.org/education/blinken/-/blob/master/.snapcraft.yaml > >> > >> > >> What is that going to give us? > >> > >> > >> Cheers, > >> Albert > > > > > > Cheers, > > Ben > > > >> > >> > >> > >> > Thanks, > >> > Scarlett > >> > > >> > On Sun, Aug 20, 2023 at 7:20 PM Justin Zobel <justin.zo...@gmail.com> > wrote: > >> > > I think there is a lot of misunderstanding here. > >> > > > >> > > In GitLab CI only test builds are done and artifacts are kept so > people > >> > > can test an AppImage or Flatpak without having to compile locally. > >> > > > >> > > Stable releases are done by the Release Team via scripts to > tarballs. > >> > > Flatpaks are done via Flathub and are done (usually) automatically > via > >> > > the Flatpak External Data Checker. > >> > > > >> > > Hopefully this clarifies the situation. > >> > > > >> > > On 21/8/23 08:48, Scarlett Moore wrote: > >> > > > On Sun, Aug 20, 2023, 4:14 PM Julius Künzel > >> > > > > >> > > > <jk.kde...@smartlab.uber.space> wrote: > >> > > > To me it seems this discussion is quite abstract and there are > >> > > > several misunderstandings because not everybody knows every > detail > >> > > > of snap packaging and/or the KDE infrastructure (me neither). > >> > > > > >> > > > I understand that from an organizational perspective most > people > >> > > > like the idea of having the files in the repos, but have > technical > >> > > > doubts. Hence, I wonder whether it would be a good idea to > take > >> > > > one KDE app to try and showcase this suggestion? > >> > > > > >> > > > Cheers, > >> > > > Julius > >> > > > > >> > > > Sounds like a great idea to me. > >> > > > Scarlett > >> > > > > >> > > > 20.08.2023 17:55:24 Laura David Hurka < > david.hu...@mailbox.org>: > >> > > > > On Sunday, August 20, 2023 12:47:10 PM CEST Ben Cooksley > wrote: > >> > > > >> On Sun, Aug 20, 2023 at 12:43 PM Scarlett Moore < > >> > > > >> > >> > > > >> scarlett.gately.mo...@gmail.com> wrote: > >> > > > >>> Only on release! We will not be building from master! We > don't > >> > > > > >> > > > want > >> > > > > >> > > > >>> unstable snaps. > >> > > > >>> Thanks, > >> > > > >>> Scarlett > >> > > > >> > >> > > > >> In that particular case the jobs should be manually > triggered > >> > > > >> only. > >> > > > >> > >> > > > >> Gitlab CI is really made for building artifacts for a given > >> > > > > >> > > > commit rather > >> > > > > >> > > > >> than for a specified version though, so this is definitely > >> > > > > >> > > > going to be a > >> > > > > >> > > > >> case of things not fitting quite right. > >> > > > >> > >> > > > >> Cheers, > >> > > > >> Ben > >> > > > >> [...] > >> > > > > > >> > > > > This confuses me too. > >> > > > > It seems Scarlett wants to use a “deploy” stage [1] and a > job > >> > > > > >> > > > rule [2] > >> > > > > >> > > > > to run snap build&release jobs automatically when the > release is > >> > > > > >> > > > done. > >> > > > > >> > > > > If you mean that Gitlab CI should not be used to automate > >> > > > > >> > > > release jobs, > >> > > > > >> > > > > you should elaborate more how binary-factory is meant to be > >> > > > > >> > > > replaced. > >> > > > > >> > > > > Otherwise, do you just note that Gitlab CI is suboptimal, > >> > > > > or do you recommend to use something else? > >> > > > > Like: “Release build: automatic is fine. Release publish: > please > >> > > > > >> > > > only manual”? > >> > > > > >> > > > > Cheers, David > >> > > > > > >> > > > > > >> > > > > [1] https://docs.gitlab.com/ee/ci/yaml/#stages > >> > > > > [2] like this: > >> > > > > > >> > > > > snap-release-job: > >> > > > > rules: > >> > > > > -if: $CI_COMMIT_TAG =~ > /^v[0-9][0-9]\.[0-9][0-9]\.[0-9][0-9]$/ > >> > > > > > >> > > > > [...] > >> > > > > > >> > > > > see also: > >> > > > > https://docs.gitlab.com/ee/ci/jobs/job_control.html#use-predefined-c > >> > > > icd-variables-to-run-jobs-only-in-specific-pipeline-types > >> > >> > >> > >> >
