On Mon, Dec 6, 2021 at 1:44 PM Sandro Bonazzola <sbona...@redhat.com> wrote:
> > > Il giorno lun 6 dic 2021 alle ore 12:18 Martin Perina <mper...@redhat.com> > ha scritto: > >> Hi, >> >> I've tried to move ovirt-engine-extensions-api >> <https://github.com/ovirt/ovirt-engine-extensions-api> to github and >> setup GH action to build to have some sort of a basic template for all our >> Java based projects: >> >> >> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml >> >> During the work I discovered some parts which should be further discussed: >> >> 1. Most of projects requires packages from ovirt-master-release RPM >> repositories to build itself >> - In past we have been using >> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm to >> fetch the latest version of this RPM >> - Do we plan to keep it and have this URL as the main source for >> oVirt master repositories? >> > > No. > Please follow > https://ovirt.org/develop/dev-process/install-nightly-snapshot.html > Fixed, thanks! https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L32 > >> - ovirt-engine-extensions-api projects requires only Virt SIG repo so >> I've applied below hack to provide it: >> >> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/prepare-env-cs8.sh#L10 >> > - I don't like the solution, it would be much nicer to have a constant >> URL for the latest ovirt-release-master RPM >> > > Instrucitons in > https://ovirt.org/develop/dev-process/install-nightly-snapshot.html > whould be stable. > For CentOS Virt SIG specifically I haven't yet built the 4.5 release rpms > for CentOS . Once they'll be ready you should be able to just dnf install > the centos release ovirt-4.5 rpm to enable them. I'll try to handle it > before Christmas. > No worries, we are still far from the stable oVirt 4.5 release :-( And build jobs should depend on ovirt-release-master anyway ... > > > >> >> 2. Providing RPM repositories for different OS inside single build >> artifact >> - I wanted to build the project for both CentOS Stream 8 and 9, but I >> figured out that upload-artifact action merges results for all different OS >> builds into a single directory, so created repositories are mixed together. >> > > This may have a solution, try using the wildcard pattern > https://github.com/actions/upload-artifact#upload-using-a-wildcard-pattern > For now the solution works fine, but we just need to agree on standards. For example: Archive: artifacts.zip centos-stream-8/*.rpm centos-stream-8/repodata/* centos-stream-9/*.rpm centos-stream-9/repodata/ > > >> - I've use GH strategy option and used centos-stream-8 for CS8 and >> centos-stream-9 for CS9 builds: >> >> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L14 >> - Do we already have that documented so an OST plugin to use those >> artifacts per OS can be created? If not, where to put it? >> > > Didn't see one yet. > > > >> >> 3. Using maven cache for Java project builds >> - When build maven projects in jenkins.ovirt.org we were using >> artifactory.ovirt.org to cache maven artifacts (and also save download >> bandwidth to PHX datacenter) >> - It doesn't make to use artifactory.ovirt.org inside GH actions, so >> I've tried to define a GH action specific cache: >> >> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L37 >> > > make a lot of sense. > > > >> >> 4. Caching required dependencies >> - Above maven dependencies helps for maven invocations before RPM >> build but it doesn't help to fetch all required RPM dependencies for pro >> RPM maven based build >> - Right now it's almost 500 GB of packages to download and install on >> top of the default container and it for ovirt-engine-extensions-api it >> takes 50 % of the whole build >> time just to download and install those packages (I know it's just >> 1 minute, but anyway :-) >> - I've noticed that VDSM is using its own customized container >> <https://github.com/oVirt/vdsm/blob/master/.github/workflows/ci.yml#L8> >> - Wouldn't it be worthwhile to also create customized containers for >> Java based projects? >> > > I got suggestion from @Yaniv Kaul <yk...@redhat.com> to do something as > in Gluster: https://github.com/gluster/Gluster-Builds - they are > building Gluster RPMs by running a container (the 'build container'), which > has the required dependencies and they updates once a month, or when > there's a new dependency. > > If someone volunteer to maintain those containers, it may save build time > and bandwidth. > It's not a must for the switch, but it can be really useful. If there is no volunteer till, I will try to take a look, but not earlier than January > >> >> 5. Differences between CS8 and CS9 >> - There are again some disturbing changes between CS8 and CS9: >> - PowerTools repo from CS8 is now called CRB in CS9 >> - pki-deps javapackages-tools modules from CS8 don't exist in CS9 >> - Core DNF plugins are not installed by default in CS9 container >> - To bypass those differences I've create 2 different shell scripts >> and plug them into a single workflow job: >> >> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L33 >> >> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/prepare-env-cs8.sh >> >> https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/prepare-env-cs9.sh >> >> 6. git commands cannot be easily used inside GH action >> - GH doesn't checkout the project completely during checkout action, >> so .git directory is not present >> - In the past I've been using 'git archive' >> <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l45> >> to create .tar.gz source archive, but on GH it needed to be switched to a >> classic >> tar >> <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automation/build-artifacts.sh#L17> >> - In the past I've been using 'git rev-list' >> <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=automation/build-artifacts.sh;h=965012c1195eaf288e149bde71e64b2e9647b2c4;hb=refs/heads/master#l39> >> to fetch short git hash for snapshot builds, but on GH it needed to be >> switched to below hack >> <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L47> >> > > You can dnf install git before running checkout action and then configure > checkout action to download full git content. > ``` > - uses: actions/checkout@v2 > with: > fetch-depth: 0 > ``` > Ah, I found the issue, git command needs to be installed before performing a checkout. If the git is not installed, then GH provide checkout without .git directory If git is installed, then 'git rev-parse' and 'git archive' work fine even with the default fetch-depth (which means only latest commit): https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/workflows/build.yml#L72 > > >> >> >> Regards, >> Martin >> >> -- >> Martin Perina >> Manager, Software Engineering >> Red Hat Czech s.r.o. >> _______________________________________________ >> Devel mailing list -- devel@ovirt.org >> To unsubscribe send an email to devel-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/PBUC7OKVHJJPDLI6C2YAMZNE7LGCF73A/ >> > > > -- > > Sandro Bonazzola > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA <https://www.redhat.com/> > > sbona...@redhat.com > <https://www.redhat.com/> > > *Red Hat respects your work life balance. Therefore there is no need to > answer this email out of your office hours.* > > > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/IXIMHDSJLAVU3OVJUIPT63DYERK4EPA4/
