[ovirt-devel] Re: CI not working again

2021-12-14 Thread Ehud Yonasi 
Hi,
After digging to the issues we see that el7-worker-02 had dhcp configuration 
issues.

There were changes in the dhcp configuration last week thus making the dns not 
resolvable.

For further information about it see:
https://issues.redhat.com/browse/CPDEVOPS-407 


After rebooting the vm, it is now working.

Regards,
Ehud.

> On 14 Dec 2021, at 13:45, Nir Soffer  wrote:
> 
> On Tue, Dec 14, 2021 at 1:09 PM Nir Soffer  > wrote:
>> 
>> On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer  wrote:
>> ...
> Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> build artifacts
> so we can at least get working OST?
 
 I tried this here:
 https://gerrit.ovirt.org/c/vdsm/+/118022
>>> 
>>> Patch should be ready, I rebased by current thin-scratch-disks branch on
>>> top of it, and some OST jobs started.
>> 
>> Current state:
>> https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open
>> 
>> 6 patches ready for merge.
>> 3 OST runs started successfully
>> 3 build artifacts failed (50% success rate) - I retriggered the failed runs
>> 
>> Al the failures happened on el7-worker-02:
>> - 
>> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31470/pipeline
>> - 
>> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31485/pipeline
>> - 
>> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31488/pipeline
> 
> Retriggered runs fail again on el7-worker-02:
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31493/pipeline
>  
> 
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31490/pipeline
>  
> 
> 
>> 
>> Build history for this host - all builds fail in the last 4 days:
>> https://jenkins.ovirt.org/computer/el7-worker-02/builds 
>> 
>> 
>> Build history for other hosts:
>> https://jenkins.ovirt.org/computer/el7-worker-03/builds 
>> 
>> 
>> Nir

___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GOQYTNMO7CNFC3LKECVXWDKUUTAD4N3K/

[ovirt-devel] Re: CI not working again

2021-12-14 Thread Nir Soffer 
On Tue, Dec 14, 2021 at 1:09 PM Nir Soffer  wrote:
>
> On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer  wrote:
> ...
> > > > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > > > build artifacts
> > > > so we can at least get working OST?
> > >
> > > I tried this here:
> > > https://gerrit.ovirt.org/c/vdsm/+/118022
> >
> > Patch should be ready, I rebased by current thin-scratch-disks branch on
> > top of it, and some OST jobs started.
>
> Current state:
> https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open
>
> 6 patches ready for merge.
> 3 OST runs started successfully
> 3 build artifacts failed (50% success rate) - I retriggered the failed runs
>
> Al the failures happened on el7-worker-02:
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31470/pipeline
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31485/pipeline
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31488/pipeline

Retriggered runs fail again on el7-worker-02:
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31493/pipeline
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31490/pipeline

>
> Build history for this host - all builds fail in the last 4 days:
> https://jenkins.ovirt.org/computer/el7-worker-02/builds
>
> Build history for other hosts:
> https://jenkins.ovirt.org/computer/el7-worker-03/builds
>
> Nir
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AYKWKTJXGTBZK65OTVYKCRIBAFFLXLOX/

[ovirt-devel] Re: CI not working again

2021-12-14 Thread Nir Soffer 
On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer  wrote:
...
> > > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > > build artifacts
> > > so we can at least get working OST?
> >
> > I tried this here:
> > https://gerrit.ovirt.org/c/vdsm/+/118022
>
> Patch should be ready, I rebased by current thin-scratch-disks branch on
> top of it, and some OST jobs started.

Current state:
https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open

6 patches ready for merge.
3 OST runs started successfully
3 build artifacts failed (50% success rate) - I retriggered the failed runs

Al the failures happened on el7-worker-02:
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31470/pipeline
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31485/pipeline
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31488/pipeline

Build history for this host - all builds fail in the last 4 days:
https://jenkins.ovirt.org/computer/el7-worker-02/builds

Build history for other hosts:
https://jenkins.ovirt.org/computer/el7-worker-03/builds

Nir
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UNZ34TJIQHQOVI6Q7T4BKCULYMDRMMP5/

[ovirt-devel] Re: CI not working again

2021-12-14 Thread Nir Soffer 
On Tue, Dec 14, 2021 at 12:39 AM Nir Soffer  wrote:
>
> On Tue, Dec 14, 2021 at 12:32 AM Nir Soffer  wrote:
> >
> > On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal  wrote:
> > >
> > > Hi,
> > >
> > > it looks like Jenkins CI jobs don't finish in any reasonable time.  In
> > > case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
> > >
> > >   Cannot download 
> > > x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: 
> > > All mirrors were tried
> > >
> > > Is there a way to get fixed at least one of the runners?
> >
> > I'm using github CI now, no issue seen in the last week.
> >
> > But we have now a wrose issue, build-artifacts job fail randomly in one of
> > the targets. This means there is no way to use the automatic OST run.
> >
> > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > build artifacts
> > so we can at least get working OST?
>
> I tried this here:
> https://gerrit.ovirt.org/c/vdsm/+/118022

Patch should be ready, I rebased by current thin-scratch-disks branch on
top of it, and some OST jobs started.
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/2SVJSSHZHKXDGLYDGSTADJ5XVGI3HQHJ/

[ovirt-devel] Re: CI not working again

2021-12-14 Thread Nir Soffer 
.On Tue, Dec 14, 2021 at 11:15 AM Vojtech Juranek  wrote:
...
> > > we should move to GitHub completely ASAP.  The only thing
> > > missing is OST, right?
> >
> >
> > Yes. I think what we need is a way to start OST build with a github
> > pull request url:
> >
> > CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7}
> >
> > Or if it is easier, a way to upload zip files to jenkins for running OST.
>
> If Jenkins wipes the workspace before/after each build, that won't be probably
> the most easy way to go. Github actions deliver the rpm in zip file, so we
> basically need
> * download this zip file before build and unzip

This requires Personal Access Token:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

So we need to add a github users with PAT that can download the built artifacts.

We need to find the artifacts in the build, likey using github API.

For example:
https://github.com/oVirt/ovirt-imageio/actions/runs/1573283335

The artifact we need is the centos-8.zip:
https://github.com/oVirt/ovirt-imageio/suites/4639547952/artifacts/125747744

The URL does not tell us that this is the centos stream 8 artifact, so we need
a way to locate it programatically.

> * create local repo from it (vdsm.repo with baseurl=file://$PATH_TO_UNZIP_DIR)
> * use this local repo instead of previous Jenkins build artifact job URL

This is for a single host, all hosts in OST need to use the new repo.

Another option is to have a web server serving built rpms, e.g.

https://repo.ost.example.com/

And unzip each built rpm before the test to:

/serverroot/93d9ee4f-4cda-403a-a29d-9a3669ee49af/

So we have this layout:

/serverroot/
   93d9ee4f-4cda-403a-a29d-9a3669ee49af/
   repodata/
   *.rpm
   ...

And start the OST job with:


CUSTOM_REPOS=https://repo.ost.example.com/93d9ee4f-4cda-403a-a29d-9a3669ee49af

When the OST job finish, delete
/serverroot/93d9ee4f-4cda-403a-a29d-9a3669ee49af/

This can also work with multiple pull requests, e.g. testing both vdsm
and engine pull request
at the same time.

Nir
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/LQNP4UNRCNQF7I7Q7K2BIOZY5SBPK2PZ/

[ovirt-devel] Re: CI not working again

2021-12-14 Thread Vojtech Juranek 
On Tuesday, 14 December 2021 09:44:44 CET Nir Soffer wrote:
> On Tue, Dec 14, 2021 at 10:14 AM Milan Zamazal  wrote:
> >
> >
> > Nir Soffer  writes:
> >
> >
> >
> > > I'm using github CI now, no issue seen in the last week.
> > >
> > >
> > >
> > > But we have now a wrose issue, build-artifacts job fail randomly in one
> > > of
> > > the targets. This means there is no way to use the automatic OST run.
> > >
> > >
> > >
> > > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > > build artifacts so we can at least get working OST?
> >
> >
> >
> > Since tests on gerrit are unusable, we should do it this way, good idea.
> >
> >
> >
> > Nevertheless manually testing patches by pushing them to GitHub is
> > tiresome,
> 
> 
> This is not that bad, I'm using this flow for a few years:
> 
> while work needed:
> hack...
> push -f github # push to my github fork
> check the build in github
> 
> git review  # push to gerrit for review

which is basically the same workflow which we used when we run test also in 
Travis IIRC, so actually no change here ...

> 
> > we should move to GitHub completely ASAP.  The only thing
> > missing is OST, right?
> 
> 
> Yes. I think what we need is a way to start OST build with a github
> pull request url:
> 
> CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7}
> 
> Or if it is easier, a way to upload zip files to jenkins for running OST.

If Jenkins wipes the workspace before/after each build, that won't be probably 
the most easy way to go. Github actions deliver the rpm in zip file, so we 
basically need
* download this zip file before build and unzip
* create local repo from it (vdsm.repo with baseurl=file://$PATH_TO_UNZIP_DIR)
* use this local repo instead of previous Jenkins build artifact job URL

So we just need to implement this or am I missing something and there are some 
issues with this approach and it's more complicated?

> Once we have something like this, we can move vdsm to github.
> 
> Triggering OST builds automatically ("gating") is nice to have and
> can be done later.
> 
> Nir
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/ List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Y6CO6UJG6SD4P
> IUTPWTMBDT7CC6DGDOH/



signature.asc
Description: This is a digitally signed message part.
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/L4ZFAC4I7AOKHDGESONS5K5R5CLQCZAW/

[ovirt-devel] Re: CI not working again

2021-12-14 Thread Nir Soffer 
On Tue, Dec 14, 2021 at 10:14 AM Milan Zamazal  wrote:
>
> Nir Soffer  writes:
>
> > I'm using github CI now, no issue seen in the last week.
> >
> > But we have now a wrose issue, build-artifacts job fail randomly in one of
> > the targets. This means there is no way to use the automatic OST run.
> >
> > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > build artifacts so we can at least get working OST?
>
> Since tests on gerrit are unusable, we should do it this way, good idea.
>
> Nevertheless manually testing patches by pushing them to GitHub is
> tiresome,

This is not that bad, I'm using this flow for a few years:

while work needed:
hack...
push -f github # push to my github fork
check the build in github

git review  # push to gerrit for review

> we should move to GitHub completely ASAP.  The only thing
> missing is OST, right?

Yes. I think what we need is a way to start OST build with a github
pull request url:

CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7}

Or if it is easier, a way to upload zip files to jenkins for running OST.

Once we have something like this, we can move vdsm to github.

Triggering OST builds automatically ("gating") is nice to have and
can be done later.

Nir
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Y6CO6UJG6SD4PIUTPWTMBDT7CC6DGDOH/

[ovirt-devel] Re: CI not working again

2021-12-14 Thread Milan Zamazal 
Nir Soffer  writes:

> I'm using github CI now, no issue seen in the last week.
>
> But we have now a wrose issue, build-artifacts job fail randomly in one of
> the targets. This means there is no way to use the automatic OST run.
>
> Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> build artifacts so we can at least get working OST?

Since tests on gerrit are unusable, we should do it this way, good idea.

Nevertheless manually testing patches by pushing them to GitHub is
tiresome, we should move to GitHub completely ASAP.  The only thing
missing is OST, right?
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/THVPG25CPQZBWMAKG7ITLZSXDBGGGEMQ/

[ovirt-devel] Re: CI not working again

2021-12-13 Thread Nir Soffer 
On Tue, Dec 14, 2021 at 12:32 AM Nir Soffer  wrote:
>
> On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal  wrote:
> >
> > Hi,
> >
> > it looks like Jenkins CI jobs don't finish in any reasonable time.  In
> > case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
> >
> >   Cannot download 
> > x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: 
> > All mirrors were tried
> >
> > Is there a way to get fixed at least one of the runners?
>
> I'm using github CI now, no issue seen in the last week.
>
> But we have now a wrose issue, build-artifacts job fail randomly in one of
> the targets. This means there is no way to use the automatic OST run.
>
> Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> build artifacts
> so we can at least get working OST?

I tried this here:
https://gerrit.ovirt.org/c/vdsm/+/118022
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/AOGRM5EBCJEIH6SDEKHTFIOUKU6SETNO/

[ovirt-devel] Re: CI not working again

2021-12-13 Thread Nir Soffer 
On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal  wrote:
>
> Hi,
>
> it looks like Jenkins CI jobs don't finish in any reasonable time.  In
> case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
>
>   Cannot download 
> x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: All 
> mirrors were tried
>
> Is there a way to get fixed at least one of the runners?

I'm using github CI now, no issue seen in the last week.

But we have now a wrose issue, build-artifacts job fail randomly in one of
the targets. This means there is no way to use the automatic OST run.

Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
build artifacts
so we can at least get working OST?
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RZVXGUKEB5B5RH33L6QV33LOFRPUU5S5/

[ovirt-devel] Re: CI not working again

2021-12-12 Thread Saif Abu Saleh 
I am getting other failed CI's in other repositories:



*Api-model:* fatal: unable to access 'https://gerrit.ovirt.org/jenkins/':
Could not resolve host: gerrit.ovirt.org; Unknown error

Link:
https://jenkins.ovirt.org/job/ovirt-engine-api-model_standard-check-patch/353/

*OST:*

failing on: Error: Unable to find a match: python3-ansible-runner

Link:
https://ovirt-devops-jenkins.upshift.rdu2.redhat.com/job/ovirt-system-tests_standard-check-patch/8733/

Thanks,
Saif


On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal  wrote:

> Hi,
>
> it looks like Jenkins CI jobs don't finish in any reasonable time.  In
> case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
>
>   Cannot download
> x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm:
> All mirrors were tried
>
> Is there a way to get fixed at least one of the runners?
>
> Thanks,
> Milan
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SQ2DTNP63SQQYCFT4SUP54QWVO2YCV6Y/
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/HDRW37OZZ2HCETB4PAYW5HNNIKDI34RP/