Re: CI not working again

[Nir Soffer]

On Tue, Dec 14, 2021 at 1:09 PM Nir Soffer  wrote:
>
> On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer  wrote:
> ...
> > > > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > > > build artifacts
> > > > so we can at least get working OST?
> > >
> > > I tried this here:
> > > https://gerrit.ovirt.org/c/vdsm/+/118022
> >
> > Patch should be ready, I rebased by current thin-scratch-disks branch on
> > top of it, and some OST jobs started.
>
> Current state:
> https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open
>
> 6 patches ready for merge.
> 3 OST runs started successfully
> 3 build artifacts failed (50% success rate) - I retriggered the failed runs
>
> Al the failures happened on el7-worker-02:
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31470/pipeline
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31485/pipeline
> - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31488/pipeline

Retriggered runs fail again on el7-worker-02:
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31493/pipeline
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31490/pipeline

>
> Build history for this host - all builds fail in the last 4 days:
> https://jenkins.ovirt.org/computer/el7-worker-02/builds
>
> Build history for other hosts:
> https://jenkins.ovirt.org/computer/el7-worker-03/builds
>
> Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/AYKWKTJXGTBZK65OTVYKCRIBAFFLXLOX/

Re: CI not working again

[Nir Soffer]

On Tue, Dec 14, 2021 at 12:13 PM Nir Soffer  wrote:
...
> > > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > > build artifacts
> > > so we can at least get working OST?
> >
> > I tried this here:
> > https://gerrit.ovirt.org/c/vdsm/+/118022
>
> Patch should be ready, I rebased by current thin-scratch-disks branch on
> top of it, and some OST jobs started.

Current state:
https://gerrit.ovirt.org/q/topic:thin-scratch-disks+is:open

6 patches ready for merge.
3 OST runs started successfully
3 build artifacts failed (50% success rate) - I retriggered the failed runs

Al the failures happened on el7-worker-02:
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31470/pipeline
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31485/pipeline
- 
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/31488/pipeline

Build history for this host - all builds fail in the last 4 days:
https://jenkins.ovirt.org/computer/el7-worker-02/builds

Build history for other hosts:
https://jenkins.ovirt.org/computer/el7-worker-03/builds

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/UNZ34TJIQHQOVI6Q7T4BKCULYMDRMMP5/

Re: CI not working again

[Nir Soffer]

On Tue, Dec 14, 2021 at 12:39 AM Nir Soffer  wrote:
>
> On Tue, Dec 14, 2021 at 12:32 AM Nir Soffer  wrote:
> >
> > On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal  wrote:
> > >
> > > Hi,
> > >
> > > it looks like Jenkins CI jobs don't finish in any reasonable time.  In
> > > case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
> > >
> > >   Cannot download 
> > > x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: 
> > > All mirrors were tried
> > >
> > > Is there a way to get fixed at least one of the runners?
> >
> > I'm using github CI now, no issue seen in the last week.
> >
> > But we have now a wrose issue, build-artifacts job fail randomly in one of
> > the targets. This means there is no way to use the automatic OST run.
> >
> > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > build artifacts
> > so we can at least get working OST?
>
> I tried this here:
> https://gerrit.ovirt.org/c/vdsm/+/118022

Patch should be ready, I rebased by current thin-scratch-disks branch on
top of it, and some OST jobs started.
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/2SVJSSHZHKXDGLYDGSTADJ5XVGI3HQHJ/

Re: [ovirt-devel] Re: CI not working again

[Nir Soffer]

.On Tue, Dec 14, 2021 at 11:15 AM Vojtech Juranek  wrote:
...
> > > we should move to GitHub completely ASAP.  The only thing
> > > missing is OST, right?
> >
> >
> > Yes. I think what we need is a way to start OST build with a github
> > pull request url:
> >
> > CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7}
> >
> > Or if it is easier, a way to upload zip files to jenkins for running OST.
>
> If Jenkins wipes the workspace before/after each build, that won't be probably
> the most easy way to go. Github actions deliver the rpm in zip file, so we
> basically need
> * download this zip file before build and unzip

This requires Personal Access Token:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

So we need to add a github users with PAT that can download the built artifacts.

We need to find the artifacts in the build, likey using github API.

For example:
https://github.com/oVirt/ovirt-imageio/actions/runs/1573283335

The artifact we need is the centos-8.zip:
https://github.com/oVirt/ovirt-imageio/suites/4639547952/artifacts/125747744

The URL does not tell us that this is the centos stream 8 artifact, so we need
a way to locate it programatically.

> * create local repo from it (vdsm.repo with baseurl=file://$PATH_TO_UNZIP_DIR)
> * use this local repo instead of previous Jenkins build artifact job URL

This is for a single host, all hosts in OST need to use the new repo.

Another option is to have a web server serving built rpms, e.g.

https://repo.ost.example.com/

And unzip each built rpm before the test to:

/serverroot/93d9ee4f-4cda-403a-a29d-9a3669ee49af/

So we have this layout:

/serverroot/
   93d9ee4f-4cda-403a-a29d-9a3669ee49af/
   repodata/
   *.rpm
   ...

And start the OST job with:


CUSTOM_REPOS=https://repo.ost.example.com/93d9ee4f-4cda-403a-a29d-9a3669ee49af

When the OST job finish, delete
/serverroot/93d9ee4f-4cda-403a-a29d-9a3669ee49af/

This can also work with multiple pull requests, e.g. testing both vdsm
and engine pull request
at the same time.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/LQNP4UNRCNQF7I7Q7K2BIOZY5SBPK2PZ/

Re: CI not working again

[Nir Soffer]

On Tue, Dec 14, 2021 at 10:14 AM Milan Zamazal  wrote:
>
> Nir Soffer  writes:
>
> > I'm using github CI now, no issue seen in the last week.
> >
> > But we have now a wrose issue, build-artifacts job fail randomly in one of
> > the targets. This means there is no way to use the automatic OST run.
> >
> > Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> > build artifacts so we can at least get working OST?
>
> Since tests on gerrit are unusable, we should do it this way, good idea.
>
> Nevertheless manually testing patches by pushing them to GitHub is
> tiresome,

This is not that bad, I'm using this flow for a few years:

while work needed:
hack...
push -f github # push to my github fork
check the build in github

git review  # push to gerrit for review

> we should move to GitHub completely ASAP.  The only thing
> missing is OST, right?

Yes. I think what we need is a way to start OST build with a github
pull request url:

CUSTOM_REPOS=https://github.com/oVirt/{project}/pull/{7}

Or if it is easier, a way to upload zip files to jenkins for running OST.

Once we have something like this, we can move vdsm to github.

Triggering OST builds automatically ("gating") is nice to have and
can be done later.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/Y6CO6UJG6SD4PIUTPWTMBDT7CC6DGDOH/

Re: CI not working again

[Nir Soffer]

On Tue, Dec 14, 2021 at 12:32 AM Nir Soffer  wrote:
>
> On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal  wrote:
> >
> > Hi,
> >
> > it looks like Jenkins CI jobs don't finish in any reasonable time.  In
> > case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
> >
> >   Cannot download 
> > x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: 
> > All mirrors were tried
> >
> > Is there a way to get fixed at least one of the runners?
>
> I'm using github CI now, no issue seen in the last week.
>
> But we have now a wrose issue, build-artifacts job fail randomly in one of
> the targets. This means there is no way to use the automatic OST run.
>
> Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
> build artifacts
> so we can at least get working OST?

I tried this here:
https://gerrit.ovirt.org/c/vdsm/+/118022
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/AOGRM5EBCJEIH6SDEKHTFIOUKU6SETNO/

Re: CI not working again

[Nir Soffer]

On Thu, Dec 9, 2021 at 3:04 PM Milan Zamazal  wrote:
>
> Hi,
>
> it looks like Jenkins CI jobs don't finish in any reasonable time.  In
> case of Vdsm, at least jenkins-psi2 jobs run but they fail early with
>
>   Cannot download 
> x86_64/python3-ioprocess-1.4.2-1.202111071801.git53786ff.el8.x86_64.rpm: All 
> mirrors were tried
>
> Is there a way to get fixed at least one of the runners?

I'm using github CI now, no issue seen in the last week.

But we have now a wrose issue, build-artifacts job fail randomly in one of
the targets. This means there is no way to use the automatic OST run.

Should we disable all vdsm tests on gerrit, and enable only x64_64 el8
build artifacts
so we can at least get working OST?
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/RZVXGUKEB5B5RH33L6QV33LOFRPUU5S5/

[JIRA] (OVIRT-3126) Build artifacts jobs always fail - OST cannot run

[Nir Soffer (oVirt JIRA)]

 [ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nir Soffer updated OVIRT-3126:
--
Priority: Highest  (was: Medium)

> Build artifacts jobs always fail - OST cannot run
> -
>
> Key: OVIRT-3126
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3126
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>    Reporter: Nir Soffer
>Assignee: infra
>Priority: Highest
>
> Since last week, all build artifacts jobs fail with "Testing system error":
> - 
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31062/artifact/ci_build_summary.html
> - 
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31064/artifact/ci_build_summary.html
> - 
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31063/artifact/ci_build_summary.html
> This breaks OST, waiting for the builds.
> The error seems to be in global_setup.sh:
> [2021-11-26T16:02:21.443Z] + sudo -n systemctl start libvirtd haveged 
> firewalld
> [2021-11-26T16:02:21.443Z] Failed to start libvirtd.service: Unit not found.
> [2021-11-26T16:02:21.443Z] Failed to start haveged.service: Unit not found.
> Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100183)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/RATA65JV2WAWTMVG6LA66D3SINR2QYE7/

[JIRA] (OVIRT-3126) Build artifacts jobs always fail - OST cannot run

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3126:
-

 Summary: Build artifacts jobs always fail - OST cannot run
 Key: OVIRT-3126
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3126
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Since last week, all build artifacts jobs fail with "Testing system error":
- 
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31062/artifact/ci_build_summary.html
- 
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31064/artifact/ci_build_summary.html
- 
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31063/artifact/ci_build_summary.html

This breaks OST, waiting for the builds.

The error seems to be in global_setup.sh:

[2021-11-26T16:02:21.443Z] + sudo -n systemctl start libvirtd haveged firewalld
[2021-11-26T16:02:21.443Z] Failed to start libvirtd.service: Unit not found.
[2021-11-26T16:02:21.443Z] Failed to start haveged.service: Unit not found.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100183)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/YLA7RJ3RNEYTO6QFYH4NU7ZDVYCCXASP/

[JIRA] (OVIRT-3125) Gerrit https links broken (since server was moved?)

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3125:
-

 Summary: Gerrit https links broken (since server was moved?)
 Key: OVIRT-3125
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3125
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Trying to "git pull" from gerrit get stuck now.

Checking https://gerrit.ovirt.org/admin/repos/vdsm
we see:

git clone "https://gerrit.ovirt.org/vdsm;

But visiting this url shows:

$ curl -i https://gerrit.ovirt.org/vdsm
HTTP/1.1 404 Not Found
Date: Thu, 25 Nov 2021 12:12:17 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Content-Type: text/plain;charset=iso-8859-1
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Content-Length: 9

Not Found

Same for ovirt-engine:

$ curl -i https://gerrit.ovirt.org/ovirt-engine
HTTP/1.1 404 Not Found
Date: Thu, 25 Nov 2021 12:13:13 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Content-Type: text/plain;charset=iso-8859-1
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Content-Length: 9

Not Found

I guess this is related to moving to new hardware yesterday?

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100183)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7RWYRPYS5ZVD2KHREX2OG3ORMNVX3YZE/

Re: [ovirt-devel] OST gating oddities

[Nir Soffer]

On Tue, Nov 16, 2021 at 9:43 PM Milan Zamazal  wrote:
>
> Hi,
>
> putting recently experienced problems with OST runs failures aside, I
> wonder about what currently happens in gerrit on gating.  For instance in
> https://gerrit.ovirt.org/c/vdsm/+/117523/6:
>
> - OST OST -1
>   Patch Set 6: OST-1
>   https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger1/4579 : Looking 
> for build artifacts for OST.
>   ci build
>
>   * Why setting OST-1 when looking for a build?
>
> - Jenkins CI
>   Patch Set 6:
>
>   No Builds Executed
>
>   https://jenkins.ovirt.org/job/vdsm_standard-check-patch/30672/ : To avoid 
> overloading the infrastructure, a whitelist for
>   running gerrit triggered jobs has been set in place, if
>   you feel like you should be in it, please contact infra at
>   ovirt dot org.
>
>   * OST not allowed to trigger builds?
>
> - OST OST +1
>   Patch Set 6: OST+1
>
>   https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger1/4583 : start OST 
> for https://jenkins.ovirt.org/job/vdsm_standard-check-patch/30681/
>
>   * Shouldn't OST set OST+1 after it runs rather than when it starts?
>
> Is there any explanation of these phenomenons?

I have seen the same issues in the last week.

I think we should start moving the project to github or gitlab instead
of wasting time on fixing the legacy CI.

This is a good time to do this.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CF624PR42O5B5FBTWK6SWNZMWOQ2AAEM/

[JIRA] (OVIRT-3116) Fail fast on unsupported configuration

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3116:
-

 Summary: Fail fast on unsupported configuration
 Key: OVIRT-3116
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3116
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


When using:

host-distro: el8

The CI ignores the unknown configuration "el8" runs the job anywhere.
Then the job fails after many minutes, wasting resources and developers
time running on el7, where the tests cannot pass.

Lets fix to fail fast on the first unknown configuration, don't run anything
if the configuration is invalid.

Example builds:
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/29292/
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/29293/
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/29295/



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100172)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/X2CYTJN3XEOOOBACAKFY7PCOYBWKLJD3/

[JIRA] (OVIRT-3114) Unexplained timeouts in CI, need to investigate on a host

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=41053#comment-41053
 ] 

Nir Soffer commented on OVIRT-3114:
---

On Thu, Aug 12, 2021 at 11:30 PM Nir Soffer  wrote:
>
> We see unexplained timeouts in vdsm tests in the last 2 days.
>
> This code that should finish in no time times out after 30 seconds:
>
> op = qemuimg.rebase(
> vol.volumePath,
> "wrong",
> format=qemuimg.FORMAT.QCOW2,
> backingFormat=qemuimg.FORMAT.RAW,
> unsafe=True)
> op.run()
>
> See
> https://gerrit.ovirt.org/c/vdsm/+/116185
>
> I'm disabling this test for now, but I need to understand what's going
> on in the CI. I think the only way to understand this issue is to ssh to
> a slave and debug this code manually in the same environment used
> by the tests.

We skip 10 tests now:
https://gerrit.ovirt.org/c/vdsm/+/116189

This should be reverted when the root cause is fixed.

> Unexplained timeouts in CI, need to investigate on a host
> -
>
> Key: OVIRT-3114
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3114
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> We see unexplained timeouts in vdsm tests in the last 2 days.
> This code that should finish in no time times out after 30 seconds:
> op = qemuimg.rebase(
> vol.volumePath,
> "wrong",
> format=qemuimg.FORMAT.QCOW2,
> backingFormat=qemuimg.FORMAT.RAW,
> unsafe=True)
> op.run()
> See
> https://gerrit.ovirt.org/c/vdsm/+/116185
> I'm disabling this test for now, but I need to understand what's going
> on in the CI. I think the only way to understand this issue is to ssh to
> a slave and debug this code manually in the same environment used
> by the tests.
> Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100172)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/SKVT725JMNNV6VB46B3UK4QRTJVAEUNX/

[JIRA] (OVIRT-3115) Inconsistent Jenkins node time zones - use UTC?

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3115:
-

 Summary: Inconsistent Jenkins node time zones - use UTC?
 Key: OVIRT-3115
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3115
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


We found a broken test that fail when running on EDT timezone:
https://gerrit.ovirt.org/c/vdsm/+/116187

However this test fails randomly. Looking at the run logs, we see that
some nodes
are using UTC:

[2021-08-13T15:04:26.855Z] ## Fri Aug 13 15:04:26 UTC 2021 Running
env: el8stream:centos-stream-8-x86_64

But others use EDT:

[2021-08-13T16:32:26.866Z] ## Fri Aug 13 12:32:26 PM EDT 2021
Running env: el8stream:centos-stream-8-x86_64

Run logs:
- 
https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_standard-check-patch/runs/29231/nodes/152/steps/240/log/?start=0
- 
https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_standard-check-patch/runs/29229/nodes/152/steps/264/log/?start=0

Having different time zones on the same system wastes time for everyone.

Please use the same timezone on all systems. I think using UTC will
make everyone
life easier since developers and admins are located on multiple
timezones and everyone
know what is UTC.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100172)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/I5J5GDS2TWT4HXRCW23SBNRM6FZRAQBC/

[JIRA] (OVIRT-3114) Unexplained timeouts in CI, need to investigate on a host

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3114:
-

 Summary: Unexplained timeouts in CI, need to investigate on a host
 Key: OVIRT-3114
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3114
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


We see unexplained timeouts in vdsm tests in the last 2 days.

This code that should finish in no time times out after 30 seconds:

op = qemuimg.rebase(
vol.volumePath,
"wrong",
format=qemuimg.FORMAT.QCOW2,
backingFormat=qemuimg.FORMAT.RAW,
unsafe=True)
op.run()

See
https://gerrit.ovirt.org/c/vdsm/+/116185

I'm disabling this test for now, but I need to understand what's going
on in the CI. I think the only way to understand this issue is to ssh to
a slave and debug this code manually in the same environment used
by the tests.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100172)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/RAYQZXVFDIJNO6K56O32HTVUWA47MWVA/

Re: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 2046 - Failure!

[Nir Soffer]

On Wed, Jul 7, 2021 at 4:42 PM Eitan Raviv  wrote:
>
> Adding Ales as well.
> AFAIK vdsm does not actively poll engine for liveness, nor does any retries. 
> But retries might be at a deeper infra level where Marcin is the person to 
> ask IIUC.

Right, no retries in vdsm, we send replies or events, and we don't have any
way to tell if engine got the message.

> On Wed, Jul 7, 2021 at 4:40 PM Yedidyah Bar David  wrote:
>>
>> On Wed, Jun 23, 2021 at 12:30 PM Yedidyah Bar David  wrote:
>> >
>> > On Wed, Jun 23, 2021 at 12:02 PM Sandro Bonazzola  
>> > wrote:
>> >>
>> >>
>> >>
>> >> Il giorno mer 23 giu 2021 alle ore 07:48 Yedidyah Bar David 
>> >>  ha scritto:
>> >>>
>> >>> On Wed, Jun 9, 2021 at 12:13 PM Yedidyah Bar David  
>> >>> wrote:
>> >>> >
>> >>> > On Tue, Jun 8, 2021 at 9:01 AM Yedidyah Bar David  
>> >>> > wrote:
>> >>> > >
>> >>> > > On Tue, Jun 8, 2021 at 6:08 AM  wrote:
>> >>> > > >
>> >>> > > > Project: 
>> >>> > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/
>> >>> > > > Build: 
>> >>> > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/2046/
>> >>> > > > Build Number: 2046
>> >>> > > > Build Status:  Failure
>> >>> > > > Triggered By: Started by timer
>> >>> > > >
>> >>> > > > -
>> >>> > > > Changes Since Last Success:
>> >>> > > > -
>> >>> > > > Changes for Build #2046
>> >>> > > > [Eitan Raviv] network: force select spm - wait for dc status
>> >>> > > >
>> >>> > > >
>> >>> > > >
>> >>> > > >
>> >>> > > > -
>> >>> > > > Failed Tests:
>> >>> > > > -
>> >>> > > > 1 tests failed.
>> >>> > > > FAILED:  
>> >>> > > > he-basic-suite-master.test-scenarios.test_004_basic_sanity.test_template_export
>> >>> > > >
>> >>> > > > Error Message:
>> >>> > > > ovirtsdk4.Error: Failed to read response: [(> >>> > > > 0x5624fe64d108>, 7, 'Failed to connect to 192.168.200.99 port 443: 
>> >>> > > > Connection refused')]
>> >>> > >
>> >>> > > - The engine VM went down:
>> >>> > >
>> >>> > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/2046/artifact/exported-artifacts/test_logs/he-basic-suite-master/lago-he-basic-suite-master-host-0/_var_log/ovirt-hosted-engine-ha/agent.log
>> >>> > >
>> >>> > > MainThread::INFO::2021-06-08
>> >>> > > 05:07:34,414::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop)
>> >>> > > Current state EngineUp (score: 3400)
>> >>> > > MainThread::INFO::2021-06-08
>> >>> > > 05:07:44,575::states::135::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)
>> >>> > > Penalizing score by 960 due to network status
>> >>> > >
>> >>> > > - Because HA monitoring failed to get a reply from the dns server:
>> >>> > >
>> >>> > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/2046/artifact/exported-artifacts/test_logs/he-basic-suite-master/lago-he-basic-suite-master-host-0/_var_log/ovirt-hosted-engine-ha/broker.log
>> >>> > >
>> >>> > > Thread-1::WARNING::2021-06-08
>> >>> > > 05:07:25,486::network::120::network.Network::(_dns) DNS query failed:
>> >>> > > ; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> +tries=1 +time=5
>> >>> > > ;; global options: +cmd
>> >>> > > ;; connection timed out; no servers could be reached
>> >>> > >
>> >>> > > Thread-3::INFO::2021-06-08
>> >>> > > 05:07:28,543::mem_free::51::mem_free.MemFree::(action) memFree: 1801
>> >>> > > Thread-5::INFO::2021-06-08
>> >>> > > 05:07:28,972::engine_health::246::engine_health.EngineHealth::(_result_from_stats)
>> >>> > > VM is up on this host with healthy engine
>> >>> > > Thread-2::INFO::2021-06-08
>> >>> > > 05:07:31,532::mgmt_bridge::65::mgmt_bridge.MgmtBridge::(action) Found
>> >>> > > bridge ovirtmgmt in up state
>> >>> > > Thread-1::WARNING::2021-06-08
>> >>> > > 05:07:33,011::network::120::network.Network::(_dns) DNS query failed:
>> >>> > > ; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> +tries=1 +time=5
>> >>> > > ;; global options: +cmd
>> >>> > > ;; connection timed out; no servers could be reached
>> >>> > >
>> >>> > > Thread-4::INFO::2021-06-08
>> >>> > > 05:07:37,433::cpu_load_no_engine::126::cpu_load_no_engine.CpuLoadNoEngine::(calculate_load)
>> >>> > > System load total=0.3196, engine=0.1724, non-engine=0.1472
>> >>> > > Thread-3::INFO::2021-06-08
>> >>> > > 05:07:37,839::mem_free::51::mem_free.MemFree::(action) memFree: 1735
>> >>> > > Thread-5::INFO::2021-06-08
>> >>> > > 05:07:39,146::engine_health::246::engine_health.EngineHealth::(_result_from_stats)
>> >>> > > VM is up on this host with healthy engine
>> >>> > > Thread-1::WARNING::2021-06-08
>> >>> > > 05:07:40,535::network::120::network.Network::(_dns) DNS query failed:
>> >>> > > ; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> +tries=1 +time=5
>> >>> > > ;; global options: +cmd
>> >>> > > ;; connection timed out; no servers could be reached
>> >>> > >
>> >>> > > Thread-1::WARNING::2021-06-08
>> >>> > > 

[JIRA] (OVIRT-3110) OST failure, bad comment

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=41047#comment-41047
 ] 

Nir Soffer commented on OVIRT-3110:
---

On Sat, Jul 10, 2021 at 12:59 AM Nir Soffer  wrote:
>
> OST failed, and posted unprocessed comment:
>
> OST Verified -1
> Patch Set 1: Verified-1
> https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313 :
> OST https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/$OST_NUMBER
> failed with: $OST_MESSAGE
>
> https://gerrit.ovirt.org/c/vdsm/+/115642#message-b2993761_adbc1b6a
>
> Looking at:
> https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313
>
> It looks like environment issue:
>
> ERROR: Build step failed with exception
>
> java.lang.NullPointerException: no workspace from node
> hudson.slaves.DumbSlave[hpe-bl280g6-12...] which is computer
> hudson.slaves.SlaveComputer@2e2cb321 and has channel null
>
> So we have 2 issues:
> 1. If build fails, error is not handled properly producing bad comment
> 2. the actual build failure

I started another build manually (8543) and it started normally, so this was
probably a temporary issue.

>
> The first issue should be handled by posting a better comment:
>
> OST build error:
> https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313
>
> And OST should not mark the patch as verified -1, since it did not run any 
> test.
>
> Marking a patch as -1 means some tests have failed,  and the author
> needs to check
> why. A build error means OST or CI maintainers need to check why the
> error happened.

> OST failure, bad comment
> 
>
> Key: OVIRT-3110
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3110
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> OST failed, and posted unprocessed comment:
> OST Verified -1
> Patch Set 1: Verified-1
> https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313 :
> OST https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/$OST_NUMBER
> failed with: $OST_MESSAGE
> https://gerrit.ovirt.org/c/vdsm/+/115642#message-b2993761_adbc1b6a
> Looking at:
> https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313
> It looks like environment issue:
> ERROR: Build step failed with exception
> java.lang.NullPointerException: no workspace from node
> hudson.slaves.DumbSlave[hpe-bl280g6-12...] which is computer
> hudson.slaves.SlaveComputer@2e2cb321 and has channel null
> So we have 2 issues:
> 1. If build fails, error is not handled properly producing bad comment
> 2. the actual build failure
> The first issue should be handled by posting a better comment:
> OST build error:
> https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313
> And OST should not mark the patch as verified -1, since it did not run any 
> test.
> Marking a patch as -1 means some tests have failed,  and the author
> needs to check
> why. A build error means OST or CI maintainers need to check why the
> error happened.



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100168)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/RO6DSRNC572BCZC3ATLWNQQHTETKCSZ5/

[JIRA] (OVIRT-3110) OST failure, bad comment

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3110:
-

 Summary: OST failure, bad comment
 Key: OVIRT-3110
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3110
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


OST failed, and posted unprocessed comment:

OST Verified -1
Patch Set 1: Verified-1
https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313 :
OST https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/$OST_NUMBER
failed with: $OST_MESSAGE

https://gerrit.ovirt.org/c/vdsm/+/115642#message-b2993761_adbc1b6a

Looking at:
https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313

It looks like environment issue:

ERROR: Build step failed with exception

java.lang.NullPointerException: no workspace from node
hudson.slaves.DumbSlave[hpe-bl280g6-12...] which is computer
hudson.slaves.SlaveComputer@2e2cb321 and has channel null

So we have 2 issues:
1. If build fails, error is not handled properly producing bad comment
2. the actual build failure

The first issue should be handled by posting a better comment:

OST build error:
https://redir.apps.ovirt.org/dj/job/ds-ost-gating-trigger2/313

And OST should not mark the patch as verified -1, since it did not run any test.

Marking a patch as -1 means some tests have failed,  and the author
needs to check
why. A build error means OST or CI maintainers need to check why the
error happened.



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100168)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/J3HQ2VGHGVYV6L5MW4ZEFICTGFVP6S2N/

[JIRA] (OVIRT-3108) Please resync centos-stream-el8stream/base mirror

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=41042#comment-41042
 ] 

Nir Soffer commented on OVIRT-3108:
---

On Wed, Jul 7, 2021 at 11:12 AM Ehud Yonasi  wrote:
>
> Thanks for the report Sandro,
>
> Resyncing again.

Ehud, do we have now centos-stream mirror?

What is the way to specify the mirror in the check-patch.repos file?

Currently we use:
https://github.com/oVirt/vdsm/blob/master/automation/check-patch.repos
https://github.com/oVirt/ovirt-imageio/blob/master/automation/check-patch.repos

> On 7 Jul 2021, at 9:33, Sandro Bonazzola  wrote:
>
> Looks like the mirror needs a re-sync:
>
> [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status 
> code: 404 for 
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
>  (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status 
> code: 404 for 
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
>  (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status 
> code: 404 for 
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
>  (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] [FAILED] libpmem-1.6.1-1.el8.x86_64.rpm: Status 
> code: 404 for 
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
>  (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] Error: Error downloading packages:
> [2021-07-07T06:24:14.859Z]   Status code: 404 for 
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
>  (IP: 66.187.230.49)
>
> Thanks
> --
> Sandro Bonazzola
> MANAGER, SOFTWARE ENGINEERING, EMEA R RHV
>
> Red Hat EMEA
>
> sbona...@redhat.com
> Red Hat respects your work life balance. Therefore there is no need to answer 
> this email out of your office hours.
>
>
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/J3X5FHOKXTG2BA7XNBAZ3XWIRVC7PHPN/
>
>
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/QAEMTSN5WTOVMART6N44BUEJEDZZYTPV/

> Please resync centos-stream-el8stream/base mirror
> -
>
> Key: OVIRT-3108
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3108
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Sandro Bonazzola
>Assignee: infra
>
> Looks like the mirror needs a re-sync:
> [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status
> code: 404 for
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
> (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status
> code: 404 for
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
> (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] [MIRROR] libpmem-1.6.1-1.el8.x86_64.rpm: Status
> code: 404 for
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
> (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] [FAILED] libpmem-1.6.1-1.el8.x86_64.rpm: Status
> code: 404 for
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
> (IP: 66.187.230.49)
> [2021-07-07T06:24:14.859Z] Error: Error downloading packages:
> [2021-07-07T06:24:14.859Z]   Status code: 404 for
> http://mirrors.phx.ovirt.org/repos/yum/centos-stream-el8stream/base/Packages/libpmem-1.6.1-1.el8.x86_64.rpm
> (IP: 66.187.230.49)
> Thanks
> -- 
> Sandro Bonazzola
> MANAGER, SOFTWARE ENGINEERING, EMEA R RHV
> Red Hat EMEA <https://www.redhat.com/>
> sbona...@redhat.com
> <https://www.redhat.com/>
> *

[JIRA] (OVIRT-3102) Broken commit-msg hook in gerrit

[Nir Soffer (oVirt JIRA)]

 [ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nir Soffer updated OVIRT-3102:
--
Attachment: commit-msg

On Fri, Jun 25, 2021 at 10:38 PM Nir Soffer  wrote:
>
> In ovirt-imageio developer docs, we ask to install the commit-msg hook
> from gerrit:
>
> wget -P .git/hooks https://gerrit.ovirt.org/tools/hooks/commit-msg
>
> This downloads now a broken script that does not add a ChangeId header.

I found the issue, our instructions were missing:

chmod +x .git/hooks/commit-msg

After adding it, the new script works fine.

> In the past I think it returned something like the attached file,
> without the code for
> adding signed-off-by.
>
> I suggest to replace the current file at:
> https://gerrit.ovirt.org/tools/hooks/commit-msg
>
> Or if we want to keep the default file for some reason, add:
> https://gerrit.ovirt.org/tools/ovirt/hooks/commit-msg
>
> This file contains the code for adding the signed-off-by header, so so
> developers
> do not need to add this manually:
>
> # Add Signed-off-by trailer.
> sob=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: 
> \1/p')
> git interpret-trailers --in-place --trailer "$sob" "$1"

But developers still need to add the code above.

Attaching a version with this code to replace the current file.

> Broken commit-msg hook in gerrit
> 
>
> Key: OVIRT-3102
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3102
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
> Attachments: commit-msg, commit-msg
>
>
> In ovirt-imageio developer docs, we ask to install the commit-msg hook
> from gerrit:
> wget -P .git/hooks https://gerrit.ovirt.org/tools/hooks/commit-msg
> This downloads now a broken script that does not add a ChangeId header.
> In the past I think it returned something like the attached file,
> without the code for
> adding signed-off-by.
> I suggest to replace the current file at:
> https://gerrit.ovirt.org/tools/hooks/commit-msg
> Or if we want to keep the default file for some reason, add:
> https://gerrit.ovirt.org/tools/ovirt/hooks/commit-msg
> This file contains the code for adding the signed-off-by header, so so
> developers
> do not need to add this manually:
> # Add Signed-off-by trailer.
> sob=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: 
> \1/p')
> git interpret-trailers --in-place --trailer "$sob" "$1"
> Later we need to replace this ugly shell script with proper code.
> Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/EMFMZUPBK34J7RMZWPBR4AYAUBWGNMWY/

[JIRA] (OVIRT-3102) Broken commit-msg hook in gerrit

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3102:
-

 Summary: Broken commit-msg hook in gerrit
 Key: OVIRT-3102
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3102
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


In ovirt-imageio developer docs, we ask to install the commit-msg hook
from gerrit:

wget -P .git/hooks https://gerrit.ovirt.org/tools/hooks/commit-msg

This downloads now a broken script that does not add a ChangeId header.

In the past I think it returned something like the attached file,
without the code for
adding signed-off-by.

I suggest to replace the current file at:
https://gerrit.ovirt.org/tools/hooks/commit-msg

Or if we want to keep the default file for some reason, add:
https://gerrit.ovirt.org/tools/ovirt/hooks/commit-msg

This file contains the code for adding the signed-off-by header, so so
developers
do not need to add this manually:

# Add Signed-off-by trailer.
sob=$(git var GIT_AUTHOR_IDENT | sed -n 's/^\(.*>\).*$/Signed-off-by: \1/p')
git interpret-trailers --in-place --trailer "$sob" "$1"

Later we need to replace this ugly shell script with proper code.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/YNBBU74P44X2TVOOFEI7XALFH4WZZQJN/

[JIRA] (OVIRT-3101) Add "ci system-test" command

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=41033#comment-41033
 ] 

Nir Soffer commented on OVIRT-3101:
---

On Thu, Jun 24, 2021 at 1:49 PM Marcin Sobczyk  wrote:
>
>
>
> On 6/23/21 5:44 PM, Nir Soffer wrote:
> > Similar to "ci build", "ci test", "ci merge" add a new command that
> > triggers OST run.
> >
> > Running OST is tied now in vdsm (and engine?) to Code-Review: +2.
> > This causes trouble and does not allow non-maintainers to use the 
> > convenient OST
> > infrastructure.
> >
> > Expected flow:
> >
> > 1. User add a comment with "ci system-test"
> "ci system-test" is sooo long, I vote for "ci ost".

Works for me, but "system-test" is more consistent with other ci commands:
https://ovirt-infra-docs.readthedocs.io/en/latest/CI/Using_STDCI_with_Gerrit/index.html#manual-functionality-of-the-ovirt-ci-system

> Add "ci system-test" command
> 
>
> Key: OVIRT-3101
>     URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3101
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> Similar to "ci build", "ci test", "ci merge" add a new command that
> triggers OST run.
> Running OST is tied now in vdsm (and engine?) to Code-Review: +2.
> This causes trouble and does not allow non-maintainers to use the convenient 
> OST
> infrastructure.
> Expected flow:
> 1. User add a comment with "ci system-test"
> 2. OST flow building and running OST triggered



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7YWOEXG6SHMOWZZHGJ7IDKIV7B422AQU/

[JIRA] (OVIRT-3101) Add "ci system-test" command

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=41032#comment-41032
 ] 

Nir Soffer commented on OVIRT-3101:
---

On Thu, Jun 24, 2021 at 2:11 PM Yedidyah Bar David  wrote:
>
> On Thu, Jun 24, 2021 at 1:51 PM Marcin Sobczyk  wrote:
> >
> >
> >
> > On 6/23/21 5:44 PM, Nir Soffer wrote:
> > > Similar to "ci build", "ci test", "ci merge" add a new command that
> > > triggers OST run.
> > >
> > > Running OST is tied now in vdsm (and engine?) to Code-Review: +2.
> > > This causes trouble and does not allow non-maintainers to use the 
> > > convenient OST
> > > infrastructure.
> > >
> > > Expected flow:
> > >
> > > 1. User add a comment with "ci system-test"
> > "ci system-test" is sooo long, I vote for "ci ost".
>
> +1.
>
> Perhaps we can add an optional suite name? E.g. 'ci ost ansible-suite-master'

+1

> Add "ci system-test" command
> ----
>
> Key: OVIRT-3101
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3101
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> Similar to "ci build", "ci test", "ci merge" add a new command that
> triggers OST run.
> Running OST is tied now in vdsm (and engine?) to Code-Review: +2.
> This causes trouble and does not allow non-maintainers to use the convenient 
> OST
> infrastructure.
> Expected flow:
> 1. User add a comment with "ci system-test"
> 2. OST flow building and running OST triggered



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/G37YB3WYQGHNMZ5QF53JW6TDI5SQRMLW/

[JIRA] (OVIRT-3101) Add "ci system-test" command

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3101:
-

 Summary: Add "ci system-test" command
 Key: OVIRT-3101
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3101
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Similar to "ci build", "ci test", "ci merge" add a new command that
triggers OST run.

Running OST is tied now in vdsm (and engine?) to Code-Review: +2.
This causes trouble and does not allow non-maintainers to use the convenient OST
infrastructure.

Expected flow:

1. User add a comment with "ci system-test"
2. OST flow building and running OST triggered



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/A5A2SHSJTW7S4EUD3XWAFX27G3AQXVXZ/

[JIRA] (OVIRT-3100) Add OST flag to all oVirt projects

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3100:
-

 Summary: Add OST flag to all oVirt projects
 Key: OVIRT-3100
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3100
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Similar to Continuous-Integration flag, OST needs its own flag.

OST will set this flag when OST build started from gerrit was started.

We want to make this field mandatory for merging patches, like
Continuous-Integration.

Like Continuous-Integration maintainers should be able to override the
value after
OST completed, or set the value before OST started.



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/JLPBDC3KRD3X4AF4C6JT4HESBBI42YZD/

[JIRA] (OVIRT-3099) Re: oVirt CI mirror for centos stream advancedvirt-common repo

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3099:
-

 Summary: Re: oVirt CI mirror for centos stream advancedvirt-common 
repo
 Key: OVIRT-3099
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3099
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


On Fri, Jun 11, 2021 at 12:41 PM Sandro Bonazzola 
wrote:

>
>
> Il giorno ven 4 giu 2021 alle ore 19:44 Nir Soffer 
> ha scritto:
>
>> I updated vdsm libvirt requirement to 7.0.0-14. The package exists in
>> centos
>> stream and rhel so the change is correct, but the install check fails in
>> the CI.
>>
>> I found that adding the repo to check-patch.repos works:
>> https://gerrit.ovirt.org/c/vdsm/+/115039/4/automation/check-patch.repos
>>
>> But depending on mirror.centos.org does not feel like the right way. I
>> think
>> keeping a local mirror is the right way.
>>
>
> Please note that despite we are pointing to centos mirrors, CI is running
> under proxy, so we are caching on our datacenter the rpms we consume anyway.
> That said, we can mirror advanced virtualization repos as well and the
> local mirror will be automatically picked up.
>
> I see we are already mirroring the test repo for CentOS Linux:
> ./data/mirrors-reposync.conf:
>70 : [ovirt-master-centos-advanced-virtualization-el8]
>72 : baseurl=
> https://buildlogs.centos.org/centos/8/virt/x86_64/advanced-virtualization/
>
> For CentOS Stream we'll need an addition there. Please open a ticket on
> infra-supp...@ovirt.org about it.
>

Trying


>
>
>
>> It looks like the stdci docs[1] are not maintained for a while,
>> listing mirrors for
>> fedora 30 and centos 7.
>>
>
> I would suggest to open a ticket for this as well
>
>
>
>>
>> Looking in the mirrors jobs[2] we have advanced-virtualization for
>> centos[3] but
>> it holds old versions (6.x).
>>
>> Can we add a local mirror for this repo?
>>
>> [1]
>> https://ovirt-infra-docs.readthedocs.io/en/latest/CI/List_of_mirrors/index.html
>> [2] https://jenkins.ovirt.org/search/?q=system-sync_mirrors
>> [3]
>> https://buildlogs.centos.org/centos/8/virt/x86_64/advanced-virtualization/Packages/l/
>>
>> Nir
>>
>>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbona...@redhat.com
> <https://www.redhat.com/>
>
> *Red Hat respects your work life balance. Therefore there is no need to
> answer this email out of your office hours.
> <https://mojo.redhat.com/docs/DOC-1199578>*
>
>
>



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PCW6VLBOOOJEN5ZJMHJODW3CRQMNSOLG/

[JIRA] (OVIRT-3098) Gerrit hook adds unrelated patches to bugs

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3098:
-

 Summary: Gerrit hook adds unrelated patches to bugs
 Key: OVIRT-3098
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3098
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Gerrit hook is wrongly looking for https://bugzilla.redhat.com/ URLs
in the commit message, and adding the patch to the bug.

Example patch:
https://gerrit.ovirt.org/c/vdsm/+/115339

I had to clean up the bug after the broken hook (see screenshot).

The hook should really look only in the single URL in (one or more)
Bug-Url headers:

Bug-Url: https://bugzilla.redhat.com/

I reported this years ago (I think for Related-To:), and I remember we had
a patch fixing this issue, but for some reason it was lost.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100166)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7RXO2L6I6FFMLBJFKXBYWHUXOXJFGEXJ/

Re: Running basic-suite-master more often

[Nir Soffer]

On Tue, Jan 26, 2021 at 1:29 PM Marcin Sobczyk  wrote:
>
> Hi,
>
> On 1/20/21 10:44 AM, Yedidyah Bar David wrote:
> > Hi all,
> >
> > We currently run [1], despite its name, 3 times at night - at 01:00,
> > 03:00, 05:00 (UTC+00, I think). What do you think about running it
> > also during the day? This should help identify breakages caused by
> > merging stuff during the day (most developers, I think, are in EMEA).
> I don't think it will help with breakages caused by merging stuff - if you
> don't build RPMs out of the merged stuff it won't get into OST.
>
> >
> > I know we worked on gating in the past and gave up on this. I do not
> > suggest to start that discussion again, nor to block anything any
> > other way. Just run a bit more. Even once an hour will not add too
> > much load IMO and can be useful.

Maybe run every 6 hours instead of every 2 hours during night?

> > I also do not suggest to run per-patch (say, for the engine git repo)
> > - this sometimes causes too much load, and sometimes is not enough -
> > if a patch is not always failing a run, running more than once does
> > add value.
> This is actually something I've been thinking to get back to.
> Not in the same form we had it before. We'd need PSI-based runs
> that are deployed faster than current CI first. The other thing is I was
> thinking about making a "gating suite". Think of a trimmed-down version
> of basic suite that can run in 15-20 mins.

Yes please!

> But for this, we need
> the CI first and then some work on modularization of suites.

I'm not sure about making it gating, but I think we need to make the
basic suite thinner and move most tests to vertical based suites.

Having a thin basic suite, maybe we could run 2 suites in parallel,
basicSuite and suite needed for particular patch (e.g storage).
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/UZ3WOFKV54BYRPWDLQJOBHDV4WG6DAHO/

[JIRA] (OVIRT-3070) global_setup.sh fail: "docker.service: Failed at step EXEC spawning /usr/bin/dockerd-current: Permission denied"

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3070:
-

 Summary: global_setup.sh fail: "docker.service: Failed at step 
EXEC spawning /usr/bin/dockerd-current: Permission denied"
 Key: OVIRT-3070
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3070
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Seems that all builds fail now because the global setup script fails
to setup the slave.

Some examples:
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25522/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25520/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25519/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25516/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25515/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25514/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25513/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25511/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25510/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25509/
- https://jenkins.ovirt.org/job/vdsm_standard-check-patch/25508/

Log from the failed step:
https://jenkins.ovirt.org/blue/rest/organizations/jenkins/pipelines/vdsm_standard-check-patch/runs/25522/nodes/150/steps/204/log/?start=0

[2020-12-07T18:09:18.284Z] Dec 07 15:11:08
vm0148.workers-phx.ovirt.org dockerd-current[1342]:
time="2020-12-07T15:11:08.426348650Z" level=warning
msg="d10f349413a9ce2b5f6cb33d7d721d133b323aba1bf00021ce4d714059b00813
cleanup: failed to unmount secrets: invalid argument"
[2020-12-07T18:09:18.284Z] Dec 07 15:11:39
vm0148.workers-phx.ovirt.org dockerd-current[1342]:
time="2020-12-07T15:11:39.112783512Z" level=warning
msg="55ff0f77d37781237a199a8ba86bc544932f10b1018a3e614bd967b8b5956fd5
cleanup: failed to unmount secrets: invalid argument"
[2020-12-07T18:09:18.284Z] Dec 07 15:11:40
vm0148.workers-phx.ovirt.org dockerd-current[1342]:
time="2020-12-07T15:11:40.654819711Z" level=info msg="Layer
sha256:9d23de0c6d4682039041ca8bb2825278a71c411a22799aeb7b47739514a731d0
cleaned up"
[2020-12-07T18:09:18.284Z] Dec 07 15:11:41
vm0148.workers-phx.ovirt.org dockerd-current[1342]:
time="2020-12-07T15:11:41.290807003Z" level=info msg="Layer
sha256:9d23de0c6d4682039041ca8bb2825278a71c411a22799aeb7b47739514a731d0
cleaned up"
[2020-12-07T18:09:18.284Z] Dec 07 15:11:45
vm0148.workers-phx.ovirt.org systemd[1]: docker.service: Main process
exited, code=exited, status=2/INVALIDARGUMENT
[2020-12-07T18:09:18.284Z] -- Subject: Unit process exited
[2020-12-07T18:09:18.284Z] -- Defined-By: systemd
[2020-12-07T18:09:18.284Z] -- Support:
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[2020-12-07T18:09:18.284Z] --
[2020-12-07T18:09:18.284Z] -- An ExecStart= process belonging to unit
docker.service has exited.
[2020-12-07T18:09:18.284Z] --
[2020-12-07T18:09:18.284Z] -- The process' exit code is 'exited' and
its exit status is 2.
[2020-12-07T18:09:18.284Z] Dec 07 15:11:45
vm0148.workers-phx.ovirt.org systemd[1]: docker.service: Failed with
result 'exit-code'.
[2020-12-07T18:09:18.284Z] -- Subject: Unit failed
[2020-12-07T18:09:18.284Z] -- Defined-By: systemd
[2020-12-07T18:09:18.284Z] -- Support:
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[2020-12-07T18:09:18.284Z] --
[2020-12-07T18:09:18.284Z] -- The unit docker.service has entered the
'failed' state with result 'exit-code'.
[2020-12-07T18:09:18.284Z] Dec 07 15:12:30
vm0148.workers-phx.ovirt.org systemd[1]: Starting Docker Application
Container Engine...
[2020-12-07T18:09:18.284Z] -- Subject: A start job for unit
docker.service has begun execution
[2020-12-07T18:09:18.284Z] -- Defined-By: systemd
[2020-12-07T18:09:18.284Z] -- Support:
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[2020-12-07T18:09:18.284Z] --
[2020-12-07T18:09:18.284Z] -- A start job for unit docker.service has
begun execution.
[2020-12-07T18:09:18.284Z] --
[2020-12-07T18:09:18.284Z] -- The job identifier is 109048.
[2020-12-07T18:09:18.284Z] Dec 07 15:12:30
vm0148.workers-phx.ovirt.org systemd[4524]: docker.service: Failed to
execute command: Permission denied
[2020-12-07T18:09:18.284Z] Dec 07 15:12:30
vm0148.workers-phx.ovirt.org systemd[4524]: docker.service: Failed at
step EXEC spawning /usr/bin/dockerd-current: Permission denied
[2020-12-07T18:09:18.284Z] -- Subject: Process
/usr/bin/dockerd-current could not be executed
[2020-12-07T18:09:18.284Z] -- Defined-By: systemd
[2020-12-07T18:09:18.284Z] -- Support:
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[2020-12-07T18:09:18.284Z] --
[2020-12-07T18:09:18.284Z] -- The process /usr/bin/dockerd-current
could not be executed and failed.
[2020-12-07T18:09:18.284Z] --
[2020-12-07T18:09:18.284Z] -- The error n

Re: [ovirt-devel] Re: Planned gerrit.ovirt.org outage 24.11.2020 from 21:00 UTC

[Nir Soffer]

On Mon, Nov 30, 2020 at 3:18 PM Evgheni Dereveanchin
 wrote:
>
> Hi Nir,
>
> Indeed, the "Projects" menu is broken for some reason in the new UI. We 
> already have a ticket reported for this glitch. The old UI is still available 
> if needed ("switch to old UI" link in the bottom right corner of each page) 
> but it will be gone once we upgrade to Gerrit 3.x in the future. I will 
> review available fixes to make the menu work but as noted, the "repos" page 
> seems to display projects just fine.
>
> As for broken git pull - I just ran some tests, both http:// and https:// 
> protocols worked for me with or without the .git suffix in the repo name. 
> Could you please provide some more info on how to reproduce this so that we 
> can see if this can be fixed?

Gerrit works for me after changing the url from
http://gerrit.ovirt.org/p/vdsm.git to
https://gerrit.ovirt.org/vdsm.

I think we need to fix the site, still using the old url:
https://ovirt.org/develop/developer-guide/vdsm/developers.html#getting-the-source

And vdsm REAME:
https://github.com/oVirt/vdsm#development-environment-setup

> Regards,
> Evgheni
>
> On Sun, Nov 29, 2020 at 6:38 PM Nir Soffer  wrote:
>>
>> On Wed, Nov 25, 2020 at 12:21 AM Evgheni Dereveanchin
>>  wrote:
>> >
>> > The upgrade is complete, Gerrit is now running at version 2.16 with the 
>> > following notable features:
>> > * PolyGerrit UI now used by default (old UI still available)
>>
>> Nice change, but "Projects" link is still broken and the old UI
>> is needed.
>>
>> In the past we got an error message that old ui is needed, now the link
>> does not do anything except fancy animation.
>>
>> I found that it is possible to use:
>> https://gerrit.ovirt.org/#/admin/projects/
>>
>> It redirects to:
>> https://gerrit.ovirt.org/admin/repos/
>>
>> And this is available via Browse > Repositories
>>
>> So we can just remove "Projects" from the menu.
>>
>> > * Drafts replaced by WIP/private changes (all existing drafts migrated to 
>> > WIP changes)
>> >
>> > Full release notes are available here (we upgraded from 2.14):
>> > https://www.gerritcodereview.com/2.15.html#release-highlights
>> > https://www.gerritcodereview.com/2.16.html#release-highlights
>> >
>> > If you spot any problems, please log a ticket or reach out to me directly.
>>
>> git pull fails now:
>>
>> $ git pull
>> fatal: http://gerrit.ovirt.org/p/vdsm.git/info/refs not valid: is this
>> a git repository?
>>
>> I found that the project ur was changed:
>>
>> -   url = http://gerrit.ovirt.org/p/vdsm.git
>> +   url = https://gerrit.ovirt.org/vdsm
>>
>> Changing this in .git/config fixes the issue, but *all* developers with
>> the old config need to change this, so it can be nice to keep the server
>> configuration that works with the old config if possible.
>>
>> Thanks!
>>
>
>
> --
> Regards,
> Evgheni Dereveanchin
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/U6OI3WXBM6O3W537IEM3KG7W6DOJ5A3Y/

Re: [ovirt-devel] Re: Planned gerrit.ovirt.org outage 24.11.2020 from 21:00 UTC

[Nir Soffer]

On Wed, Nov 25, 2020 at 12:21 AM Evgheni Dereveanchin
 wrote:
>
> The upgrade is complete, Gerrit is now running at version 2.16 with the 
> following notable features:
> * PolyGerrit UI now used by default (old UI still available)

Nice change, but "Projects" link is still broken and the old UI
is needed.

In the past we got an error message that old ui is needed, now the link
does not do anything except fancy animation.

I found that it is possible to use:
https://gerrit.ovirt.org/#/admin/projects/

It redirects to:
https://gerrit.ovirt.org/admin/repos/

And this is available via Browse > Repositories

So we can just remove "Projects" from the menu.

> * Drafts replaced by WIP/private changes (all existing drafts migrated to WIP 
> changes)
>
> Full release notes are available here (we upgraded from 2.14):
> https://www.gerritcodereview.com/2.15.html#release-highlights
> https://www.gerritcodereview.com/2.16.html#release-highlights
>
> If you spot any problems, please log a ticket or reach out to me directly.

git pull fails now:

$ git pull
fatal: http://gerrit.ovirt.org/p/vdsm.git/info/refs not valid: is this
a git repository?

I found that the project ur was changed:

-   url = http://gerrit.ovirt.org/p/vdsm.git
+   url = https://gerrit.ovirt.org/vdsm

Changing this in .git/config fixes the issue, but *all* developers with
the old config need to change this, so it can be nice to keep the server
configuration that works with the old config if possible.

Thanks!
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B3CHNVXJ5DGENHFRNYU4DQHTHTG63WOK/

[JIRA] (OVIRT-3021) Very slow push/pull to/from gerrit

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=40818#comment-40818
 ] 

Nir Soffer commented on OVIRT-3021:
---

On Wed, Sep 23, 2020 at 1:38 PM Nir Soffer  wrote:
>
> Recently pushing changes to gerrit is very slow. It seems that the issue
> is pulling changes during rebase in "git review".
>
> Sometimes "git review" is very slow, but "git review -R" is very quick. This
> show that the issue was getting changes from gerrit during when git-review
> try to rebase the current branch on master.
>
> I noticed that origin/master and gerrit/master do not show the same commit:
>
> $ git log origin/master | head -1
> commit 6e6e93cb1d237f9b6fdd533d5afdc1b9ba1c197a
>
> $ git log gerrit/master | head -1
> commit 353e7b1e322aa02d4767b6617ed094be0643b094
>
> $ git remote -v
> gerrit http://gerrit.ovirt.org/p/vdsm.git (fetch)
> gerrit ssh://nsof...@gerrit.ovirt.org:29418/vdsm.git (push)
> origin http://gerrit.ovirt.org/p/vdsm.git (fetch)
> origin http://gerrit.ovirt.org/p/vdsm.git (push)
>
> I've been using this configuration for several years, and the slowness
> started only recently, so I guess the issue is not on my side.
>
> Same issues also on ovirt-engine, using similar configuration:
>
> $ git remote -v
> gerrit http://gerrit.ovirt.org/ovirt-engine (fetch)
> gerrit ssh://nsof...@gerrit.ovirt.org:29418/ovirt-engine.git (push)
> origin git://gerrit.ovirt.org/ovirt-engine (fetch)
> origin git://gerrit.ovirt.org/ovirt-engine (push)
>
> Anyone experienced this?
>
> I guess that restarting the gerrit server will "fix" this issue.

More data:

Cloning ovirt-engine-api-model:

$ git clone git://gerrit.ovirt.org/ovirt-engine-api-model
10.0 KiB/s

$ git clone https://gerrit.ovirt.org/ovirt-engine-api-model
22.0 KiB/s

$ git clone https://github.com/oVirt/ovirt-engine-api-model.git
136.00 KiB/s

Looks like a gerrit issue.

> Very slow push/pull to/from gerrit
> --
>
> Key: OVIRT-3021
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3021
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> Recently pushing changes to gerrit is very slow. It seems that the issue
> is pulling changes during rebase in "git review".
> Sometimes "git review" is very slow, but "git review -R" is very quick. This
> show that the issue was getting changes from gerrit during when git-review
> try to rebase the current branch on master.
> I noticed that origin/master and gerrit/master do not show the same commit:
> $ git log origin/master | head -1
> commit 6e6e93cb1d237f9b6fdd533d5afdc1b9ba1c197a
> $ git log gerrit/master | head -1
> commit 353e7b1e322aa02d4767b6617ed094be0643b094
> $ git remote -v
> gerrit http://gerrit.ovirt.org/p/vdsm.git (fetch)
> gerrit ssh://nsof...@gerrit.ovirt.org:29418/vdsm.git (push)
> origin http://gerrit.ovirt.org/p/vdsm.git (fetch)
> origin http://gerrit.ovirt.org/p/vdsm.git (push)
> I've been using this configuration for several years, and the slowness
> started only recently, so I guess the issue is not on my side.
> Same issues also on ovirt-engine, using similar configuration:
> $ git remote -v
> gerrit http://gerrit.ovirt.org/ovirt-engine (fetch)
> gerrit ssh://nsof...@gerrit.ovirt.org:29418/ovirt-engine.git (push)
> origin git://gerrit.ovirt.org/ovirt-engine (fetch)
> origin git://gerrit.ovirt.org/ovirt-engine (push)
> Anyone experienced this?
> I guess that restarting the gerrit server will "fix" this issue.
> Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100146)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/JK2LGFGLCHM4ZBJYVNM7E3UUPH4PCTOK/

[JIRA] (OVIRT-3021) Very slow push/pull to/from gerrit

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3021:
-

 Summary: Very slow push/pull to/from gerrit
 Key: OVIRT-3021
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3021
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Recently pushing changes to gerrit is very slow. It seems that the issue
is pulling changes during rebase in "git review".

Sometimes "git review" is very slow, but "git review -R" is very quick. This
show that the issue was getting changes from gerrit during when git-review
try to rebase the current branch on master.

I noticed that origin/master and gerrit/master do not show the same commit:

$ git log origin/master | head -1
commit 6e6e93cb1d237f9b6fdd533d5afdc1b9ba1c197a

$ git log gerrit/master | head -1
commit 353e7b1e322aa02d4767b6617ed094be0643b094

$ git remote -v
gerrit http://gerrit.ovirt.org/p/vdsm.git (fetch)
gerrit ssh://nsof...@gerrit.ovirt.org:29418/vdsm.git (push)
origin http://gerrit.ovirt.org/p/vdsm.git (fetch)
origin http://gerrit.ovirt.org/p/vdsm.git (push)

I've been using this configuration for several years, and the slowness
started only recently, so I guess the issue is not on my side.

Same issues also on ovirt-engine, using similar configuration:

$ git remote -v
gerrit http://gerrit.ovirt.org/ovirt-engine (fetch)
gerrit ssh://nsof...@gerrit.ovirt.org:29418/ovirt-engine.git (push)
origin git://gerrit.ovirt.org/ovirt-engine (fetch)
origin git://gerrit.ovirt.org/ovirt-engine (push)

Anyone experienced this?

I guess that restarting the gerrit server will "fix" this issue.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100146)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/HLD64NVSYAE4EMMVPVR76FJH23VAOUPI/

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=40733#comment-40733
 ] 

Nir Soffer commented on OVIRT-3010:
---

On Tue, Sep 8, 2020 at 1:14 PM Ehud Yonasi  wrote:

> I found out the root cause was because of space issues in the nightly
> publisher run.
>
> The 3 ovirt publishers ran in together and the jenkins dir did not have
> that capacity.
>
> Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2,
>

Thanks!


> plus sent a patch [1] to ensure they will not run in parallel.
>
> [1]: https://gerrit.ovirt.org/60
>

I probably don't understand the change, but it seems that all jobs
will run now at 00:00?

I think we need to work on a better mechanism for updating the repository.
For example, we can have a symlink the to current repo:

current -> 200908143754

200908143754/
repo files...

200908143830/
repo files...

When creating a new repo, create a new directory. Update the "current"
symlink only
if creating the new repo succeeded. Then delete the previous respo.

With this the cases when the entire repo or some files are missing are not
possible.

On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi  wrote:
>
>> Hey,
>>
>> I am looking into this.
>> Will update with the results I will find.
>>
>> On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde  wrote:
>>
>>> Hi Nir,
>>>
>>> It looks like the CI job broke it again, breaking 4.2, 4.3 and master
>>> snapshot release RPM.
>>>
>>> I guess it's the same issue we had last time.
>>>
>>> Thanks in advance,
>>>
>>> On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer  wrote:
>>>
>>>> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer  wrote:
>>>> >
>>>> > This is the second time, last time there was an issue during a
>>>> nightly job that
>>>> > left the master snapshot repo broken.
>>>>
>>>> This is still broken today.
>>>>
>>>> In the past sending mail to infra-support created a bug, and someone
>>>> was handling
>>>> issue quickly. This seems to be broken now.
>>>>
>>>> We depend on
>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> for development and CI. When the rpm is missing, CI jobs fail, and
>>>> developers cannot
>>>> update their setup.
>>>>
>>>> Please fix it as soon as possible.
>>>>
>>>> > I have 3 failure builds:
>>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
>>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
>>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>>>> >
>>>> > All seems to fail in:
>>>> >
>>>> > [2020-09-07T10:00:56.163Z] + dnf install -y
>>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> > [2020-09-07T10:01:08.524Z] Status code: 403 for
>>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
>>>> > 66.187.230.40)
>>>> >
>>>> > Checking manually show:
>>>> >
>>>> > $ curl
>>>> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> > 
>>>> > 
>>>> > 403 Forbidden
>>>> > 
>>>> > Forbidden
>>>> > You don't have permission to access
>>>> /pub/yum-repo/ovirt-release-master.rpm
>>>> > on this server.
>>>> > 
>>>> >
>>>> > Looking at:
>>>> > https://resources.ovirt.org/pub/yum-repo/
>>>> >
>>>> > ovirt-elease-master.rpm does not exist.
>>>> >
>>>> > So we have 2 issues:
>>>> > - The file does not exist
>>>> > - We return "403 Forbidden" instead of "404 Not Found"
>>>>
>>>>
>>>
>>> --
>>>
>>> Lev Veyde
>>>
>>> Senior Software Engineer, RHCE | RHCVA | MCITP
>>>
>>> Red Hat Israel
>>>
>>> <https://www.redhat.com>
>>>
>>> l...@redhat.com | lve...@redhat.com
>>> <https://red.ht/sig>
>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>>> ___
>>> Devel mailing list -- de...@o

[JIRA] (OVIRT-3011) CI cannot find ioprocess package

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=40737#comment-40737
 ] 

Nir Soffer commented on OVIRT-3011:
---

On Wed, Sep 9, 2020 at 10:41 AM Vojtech Juranek  wrote:
>
> Hi,
> CI fails quite often with
>
> Error: Unable to find a match: python3-ioprocess-1.4.1

It was fixed in master and 4.3 today. The issue was an old requirement
in automation for ioprocess-1.4.1. Eyal published 1.4.2 2 days ago, and for
some reason 1.4.1 was removed from the repos, breaking the requirement.

Since we have now copr repo, the builds are available few minutes after merging,
there should not be any reason to add such requirements in automation.

>
> see e.g. [1, 2].
> Can someone please take a look?
> Thanks
> Vojta
>
> [1] https://jenkins.ovirt.org/job/vdsm_standard-check-patch/23502
> [2] 
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/23503___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/SYN66JCV4GPYS3JK6XQB6X4ZAJH3QU7E/

> CI cannot find ioprocess package
> 
>
> Key: OVIRT-3011
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3011
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Vojtech Juranek
>Assignee: infra
> Attachments: signature.asc
>
>
> Hi,
> CI fails quite often with
> Error: Unable to find a match: python3-ioprocess-1.4.1
> see e.g. [1, 2].
> Can someone please take a look?
> Thanks
> Vojta
> [1] https://jenkins.ovirt.org/job/vdsm_standard-check-patch/23502
> [2] https://jenkins.ovirt.org/job/vdsm_standard-check-patch/23503



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100145)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/DPNBMDHANN5GPMEG2ETRYC5CZQ6Z3ZUJ/

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=40723#comment-40723
 ] 

Nir Soffer commented on OVIRT-3010:
---

On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer  wrote:
>
> This is the second time, last time there was an issue during a nightly job 
> that
> left the master snapshot repo broken.

This is still broken today.

In the past sending mail to infra-support created a bug, and someone
was handling
issue quickly. This seems to be broken now.

We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
for development and CI. When the rpm is missing, CI jobs fail, and
developers cannot
update their setup.

Please fix it as soon as possible.

> I have 3 failure builds:
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>
> All seems to fail in:
>
> [2020-09-07T10:00:56.163Z] + dnf install -y
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> [2020-09-07T10:01:08.524Z] Status code: 403 for
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
> 66.187.230.40)
>
> Checking manually show:
>
> $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> 
> 
> 403 Forbidden
> 
> Forbidden
> You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm
> on this server.
> 
>
> Looking at:
> https://resources.ovirt.org/pub/yum-repo/
>
> ovirt-elease-master.rpm does not exist.
>
> So we have 2 issues:
> - The file does not exist
> - We return "403 Forbidden" instead of "404 Not Found"

> [CI] Builds broken by "Status code: 403 for 
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm;
> -
>
> Key: OVIRT-3010
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> This is the second time, last time there was an issue during a nightly job 
> that
> left the master snapshot repo broken.
> I have 3 failure builds:
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
> All seems to fail in:
> [2020-09-07T10:00:56.163Z] + dnf install -y
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> [2020-09-07T10:01:08.524Z] Status code: 403 for
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
> 66.187.230.40)
> Checking manually show:
> $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> 
> 
> 403 Forbidden
> 
> Forbidden
> You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm
> on this server.
> 
> Looking at:
> https://resources.ovirt.org/pub/yum-repo/
> ovirt-elease-master.rpm does not exist.
> So we have 2 issues:
> - The file does not exist
> - We return "403 Forbidden" instead of "404 Not Found"



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100145)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/YVH7ZGALA3QXRNAZJGGS34EQZMAU2ZIT/

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-3010:
-

 Summary: [CI] Builds broken by "Status code: 403 for 
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm;
 Key: OVIRT-3010
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


This is the second time, last time there was an issue during a nightly job that
left the master snapshot repo broken.

I have 3 failure builds:
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/

All seems to fail in:

[2020-09-07T10:00:56.163Z] + dnf install -y
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
[2020-09-07T10:01:08.524Z] Status code: 403 for
http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
66.187.230.40)

Checking manually show:

$ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm


403 Forbidden

Forbidden
You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm
on this server.


Looking at:
https://resources.ovirt.org/pub/yum-repo/

ovirt-elease-master.rpm does not exist.

So we have 2 issues:
- The file does not exist
- We return "403 Forbidden" instead of "404 Not Found"



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100145)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7XYHLBWJJHTU65ABN3TSI5CFOZVBPS4H/

Re: [ovirt-devel] Re: [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"

[Nir Soffer]

On Tue, Sep 8, 2020 at 1:14 PM Ehud Yonasi  wrote:

> I found out the root cause was because of space issues in the nightly
> publisher run.
>
> The 3 ovirt publishers ran in together and the jenkins dir did not have
> that capacity.
>
> Currently, I am restoring the ovirt-master snapshot and then 4.3 + 4.2,
>

Thanks!


> plus sent a patch [1] to ensure they will not run in parallel.
>
> [1]: https://gerrit.ovirt.org/60
>

I probably don't understand the change, but it seems that all jobs
will run now at 00:00?

I think we need to work on a better mechanism for updating the repository.
For example, we can have a symlink the to current repo:

current -> 200908143754

200908143754/
repo files...

200908143830/
repo files...

When creating a new repo, create a new directory. Update the "current"
symlink only
if creating the new repo succeeded. Then delete the previous respo.

With this the cases when the entire repo or some files are missing are not
possible.

On Tue, Sep 8, 2020 at 12:13 PM Ehud Yonasi  wrote:
>
>> Hey,
>>
>> I am looking into this.
>> Will update with the results I will find.
>>
>> On Tue, Sep 8, 2020 at 12:00 PM Lev Veyde  wrote:
>>
>>> Hi Nir,
>>>
>>> It looks like the CI job broke it again, breaking 4.2, 4.3 and master
>>> snapshot release RPM.
>>>
>>> I guess it's the same issue we had last time.
>>>
>>> Thanks in advance,
>>>
>>> On Tue, Sep 8, 2020 at 11:45 AM Nir Soffer  wrote:
>>>
>>>> On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer  wrote:
>>>> >
>>>> > This is the second time, last time there was an issue during a
>>>> nightly job that
>>>> > left the master snapshot repo broken.
>>>>
>>>> This is still broken today.
>>>>
>>>> In the past sending mail to infra-support created a bug, and someone
>>>> was handling
>>>> issue quickly. This seems to be broken now.
>>>>
>>>> We depend on
>>>> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> for development and CI. When the rpm is missing, CI jobs fail, and
>>>> developers cannot
>>>> update their setup.
>>>>
>>>> Please fix it as soon as possible.
>>>>
>>>> > I have 3 failure builds:
>>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
>>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
>>>> > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>>>> >
>>>> > All seems to fail in:
>>>> >
>>>> > [2020-09-07T10:00:56.163Z] + dnf install -y
>>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> > [2020-09-07T10:01:08.524Z] Status code: 403 for
>>>> > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
>>>> > 66.187.230.40)
>>>> >
>>>> > Checking manually show:
>>>> >
>>>> > $ curl
>>>> https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
>>>> > 
>>>> > 
>>>> > 403 Forbidden
>>>> > 
>>>> > Forbidden
>>>> > You don't have permission to access
>>>> /pub/yum-repo/ovirt-release-master.rpm
>>>> > on this server.
>>>> > 
>>>> >
>>>> > Looking at:
>>>> > https://resources.ovirt.org/pub/yum-repo/
>>>> >
>>>> > ovirt-elease-master.rpm does not exist.
>>>> >
>>>> > So we have 2 issues:
>>>> > - The file does not exist
>>>> > - We return "403 Forbidden" instead of "404 Not Found"
>>>>
>>>>
>>>
>>> --
>>>
>>> Lev Veyde
>>>
>>> Senior Software Engineer, RHCE | RHCVA | MCITP
>>>
>>> Red Hat Israel
>>>
>>> <https://www.redhat.com>
>>>
>>> l...@redhat.com | lve...@redhat.com
>>> <https://red.ht/sig>
>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>>> ___
>>> Devel mailing list -- de...@ovirt.org
>>> To unsubscribe send an email to devel-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/B7VMLC7JKYSU2LI6TRMEAJCXXM3NLSSG/
>>>
>>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MIHJ3W2VJ4QPWNKPKPO265Q7LTKNPWBB/

Re: [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"

[Nir Soffer]

On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer  wrote:
>
> This is the second time, last time there was an issue during a nightly job 
> that
> left the master snapshot repo broken.

This is still broken today.

In the past sending mail to infra-support created a bug, and someone
was handling
issue quickly. This seems to be broken now.

We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
for development and CI. When the rpm is missing, CI jobs fail, and
developers cannot
update their setup.

Please fix it as soon as possible.

> I have 3 failure builds:
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/
>
> All seems to fail in:
>
> [2020-09-07T10:00:56.163Z] + dnf install -y
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> [2020-09-07T10:01:08.524Z] Status code: 403 for
> http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP:
> 66.187.230.40)
>
> Checking manually show:
>
> $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm
> 
> 
> 403 Forbidden
> 
> Forbidden
> You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm
> on this server.
> 
>
> Looking at:
> https://resources.ovirt.org/pub/yum-repo/
>
> ovirt-elease-master.rpm does not exist.
>
> So we have 2 issues:
> - The file does not exist
> - We return "403 Forbidden" instead of "404 Not Found"
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/LPAN56ZMNKMUBLHZKVKGN7VYSOF3VNRW/

Re: Engine build-artifacts is broken - "ERROR: Error fetching remote repo 'origin'"

[Nir Soffer]

On Sun, Aug 23, 2020 at 1:24 AM Nir Soffer  wrote:
>
> Since Friday, engine build-artifacts job is broken. Here are latest
> failing builds:
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7425/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7426/
> https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7427/

Still fails with same error
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7431/

>
> Error from last failing build:
>
> [2020-08-22T18:16:16.995Z] No credentials specified
>
> [2020-08-22T18:16:17.019Z] Fetching changes from the remote Git repository
>
> [2020-08-22T18:16:17.035Z] Cleaning workspace
>
> [2020-08-22T18:16:17.006Z]  > git rev-parse --is-inside-work-tree # timeout=10
>
> [2020-08-22T18:16:17.028Z]  > git config remote.origin.url
> https://gerrit.ovirt.org/ovirt-engine # timeout=10
>
> [2020-08-22T18:16:17.046Z]  > git rev-parse --verify HEAD # timeout=10
>
> [2020-08-22T18:16:17.136Z] Resetting working tree
>
> [2020-08-22T18:16:17.136Z]  > git reset --hard # timeout=10
>
> [2020-08-22T18:16:17.797Z]  > git clean -fdx # timeout=10
>
> [2020-08-22T18:16:18.179Z] ERROR: Error fetching remote repo 'origin'
>
> [2020-08-22T18:16:18.179Z] hudson.plugins.git.GitException: Failed to
> fetch from https://gerrit.ovirt.org/ovirt-engine
>
> [2020-08-22T18:16:18.179Z] at
> hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909)
>
> [2020-08-22T18:16:18.179Z] at
> hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131)
>
> [2020-08-22T18:16:18.179Z] at
> hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
>
> [2020-08-22T18:16:18.179Z] at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
>
> [2020-08-22T18:16:18.179Z] at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
>
> [2020-08-22T18:16:18.179Z] at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
>
> [2020-08-22T18:16:18.179Z] at
> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
>
> [2020-08-22T18:16:18.179Z] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> [2020-08-22T18:16:18.179Z] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> [2020-08-22T18:16:18.180Z] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> [2020-08-22T18:16:18.180Z] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> [2020-08-22T18:16:18.180Z] at java.lang.Thread.run(Thread.java:748)
>
> [2020-08-22T18:16:18.180Z] Caused by: hudson.plugins.git.GitException:
> Command "git clean -fdx" returned status code 1:
>
> [2020-08-22T18:16:18.180Z] stdout:
>
> [2020-08-22T18:16:18.180Z] stderr: warning: failed to remove
> rpmbuild/BUILD/ovirt-engine-4.4.2.3/frontend/webadmin/modules/webadmin/target/tmp/gwtc15338967752245717048.tmp/org.ovirt.engine.ui.webadmin.WebAdmin/compiler/permutation-0.js
>
> [2020-08-22T18:16:18.180Z]
>
> [2020-08-22T18:16:18.180Z] at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
>
> [2020-08-22T18:16:18.180Z] at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2360)
>
> [2020-08-22T18:16:18.180Z] at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2356)
>
> [2020-08-22T18:16:18.180Z] at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1916)
>
> [2020-08-22T18:16:18.180Z] at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1928)
>
> [2020-08-22T18:16:18.180Z] at
> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:1010)
>
> [2020-08-22T18:16:18.180Z] at
> sun.reflect.GeneratedMethodAccessor1271.invoke(Unknown Source)
>
> [2020-08-22T18:16:18.180Z] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> [2020-08-22T18:16:18.180Z] at java.lang.reflect.Method.invoke(Method.java:498)
>
> [2020-08-22T18:16:18.180Z] at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:931)
>
> [2020-08-22T18:16:18.180Z] at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:905)
>
> [2020-08-22T18:16:18.180Z] at
> hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:857)
>
> [2020-08-22T18:16:18.180Z] at
> hudson.remoting.UserRequest.perform(UserRequest.java:211)
>
> [2020-08-22T18:16:18.180Z] at

Re: [ovirt-devel] Re: [VDSM] Trouble merging patch, related to jenkins whitelist or new RHEL CI?

[Nir Soffer]

On Mon, Aug 3, 2020, 09:55 Ales Musil  wrote:

>
>
> On Mon, Aug 3, 2020 at 8:51 AM Ehud Yonasi  wrote:
>
>> Try again please, I think it will now work.
>>
>
> It does. Thank you.
>
>
>>
>> On Mon, Aug 3, 2020 at 9:47 AM Ales Musil  wrote:
>>
>>>
>>>
>>> On Mon, Aug 3, 2020 at 8:30 AM Ehud Yonasi  wrote:
>>>
>>>> Hey,
>>>> That's my fault there, I've added the label to prepare for the rhel
>>>> tests on ovirt and forgot to add the maintainers group.
>>>>
>>>
Ehud, please discuss here changes to vdsm CI before applying them.


>>>> Could you verify you can add it now?
>>>>
>>>
>>> Unfortunately not. It still has the same message/error.
>>>
>>>
>>>>
>>>> Thanks,
>>>> Ehud.
>>>>
>>>> On Mon, Aug 3, 2020 at 8:43 AM Ales Musil  wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 3, 2020 at 2:24 AM Germano Veit Michel 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 1, 2020 at 7:11 AM Nir Soffer  wrote:
>>>>>>
>>>>>>> I could not merge:
>>>>>>> https://gerrit.ovirt.org/c/109402/
>>>>>>>
>>>>>>> Although it was approved and verified, and got +1 from Continuous
>>>>>>> Integration.
>>>>>>>
>>>>>>> There is a new "RHEL-Continuous-Integration", which does not run with
>>>>>>> this change,
>>>>>>> even when I trigger the tests manually with "ci test".
>>>>>>>
>>>>>>> I tried to add +1 for "RHEL-Continuous-Integration" but this is not
>>>>>>> possible, I see:
>>>>>>>
>>>>>>> RHEL-Continuous-Integration You don't have permission to edit
>>>>>>> this label.
>>>>>>>
>>>>>>
>>>>> I cannot merge anything as well. I wonder why maintainers don't get
>>>>> the permission
>>>>> for the new flag? Apparently the automation for this is not working
>>>>> yet.
>>>>>
>>>>>
>>>>>>
>>>>>>> So finally I downloaded the patch and pushed it manually.
>>>>>>>
>>>>>>> I think this patch will fix the problem:
>>>>>>> https://gerrit.ovirt.org/#/c/110576/
>>>>>>>
>>>>>>> But I need CI experts to review this. Since we have 50 projects that
>>>>>>> need this,
>>>>>>> this probably should be fixed elsewhere, and inherited by all
>>>>>>> projects.
>>>>>>>
>>>>>>> I know that Germano was not able to trigger tests because he was
>>>>>>> missing in the
>>>>>>> jenkins whitelist, but this was fixed last week.
>>>>>>>
>>>>>>> Germano, maybe just to check that everything works for you, you can
>>>>>>> post some
>>>>>>> trivial patch?
>>>>>>>
>>>>>> Build and test are working for me now after contacting infra.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Nir
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Ales Musil
>>>>>
>>>>> Software Engineer - RHV Network
>>>>>
>>>>> Red Hat EMEA <https://www.redhat.com>
>>>>>
>>>>> amu...@redhat.comIM: amusil
>>>>> <https://red.ht/sig>
>>>>> ___
>>>>> Devel mailing list -- de...@ovirt.org
>>>>> To unsubscribe send an email to devel-le...@ovirt.org
>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>>>>> oVirt Code of Conduct:
>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>> List Archives:
>>>>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/GFIT7NPOYQUINUYLQR75LNRAWSXYH65L/
>>>>>
>>>>
>>>
>>> --
>>>
>>> Ales Musil
>>>
>>> Software Engineer - RHV Network
>>>
>>> Red Hat EMEA <https://www.redhat.com>
>>>
>>> amu...@redhat.comIM: amusil
>>> <https://red.ht/sig>
>>>
>>
>
> --
>
> Ales Musil
>
> Software Engineer - RHV Network
>
> Red Hat EMEA <https://www.redhat.com>
>
> amu...@redhat.comIM: amusil
> <https://red.ht/sig>
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/H7FLG4HKUDNGXXLIPZTBAVSL7DVKMVOK/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ICYLYEPROEHTZQ6DZQQ2DDZ2WJ4K7OEC/

Re: [VDSM] Trouble merging patch, related to jenkins whitelist or new RHEL CI?

[Nir Soffer]

On Sat, Aug 1, 2020 at 12:10 AM Nir Soffer  wrote:
>
> I could not merge:
> https://gerrit.ovirt.org/c/109402/

Another patch I cannot merge:
https://gerrit.ovirt.org/c/108393/

> Although it was approved and verified, and got +1 from Continuous Integration.
>
> There is a new "RHEL-Continuous-Integration", which does not run with
> this change,
> even when I trigger the tests manually with "ci test".
>
> I tried to add +1 for "RHEL-Continuous-Integration" but this is not
> possible, I see:
>
> RHEL-Continuous-Integration You don't have permission to edit this label.
>
> So finally I downloaded the patch and pushed it manually.
>
> I think this patch will fix the problem:
> https://gerrit.ovirt.org/#/c/110576/
>
> But I need CI experts to review this. Since we have 50 projects that need 
> this,
> this probably should be fixed elsewhere, and inherited by all projects.
>
> I know that Germano was not able to trigger tests because he was missing in 
> the
> jenkins whitelist, but this was fixed last week.
>
> Germano, maybe just to check that everything works for you, you can post some
> trivial patch?
>
> Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/GTAXL2RAYBJJTELHCMEAQBVLBLKY6H66/

[VDSM] Trouble merging patch, related to jenkins whitelist or new RHEL CI?

[Nir Soffer]

I could not merge:
https://gerrit.ovirt.org/c/109402/

Although it was approved and verified, and got +1 from Continuous Integration.

There is a new "RHEL-Continuous-Integration", which does not run with
this change,
even when I trigger the tests manually with "ci test".

I tried to add +1 for "RHEL-Continuous-Integration" but this is not
possible, I see:

RHEL-Continuous-Integration You don't have permission to edit this label.

So finally I downloaded the patch and pushed it manually.

I think this patch will fix the problem:
https://gerrit.ovirt.org/#/c/110576/

But I need CI experts to review this. Since we have 50 projects that need this,
this probably should be fixed elsewhere, and inherited by all projects.

I know that Germano was not able to trigger tests because he was missing in the
jenkins whitelist, but this was fixed last week.

Germano, maybe just to check that everything works for you, you can post some
trivial patch?

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/R52SKXKHWIEJU5ZPIM5ZDC5JCIOXQTWX/

[JIRA] (OVIRT-2965) Vdsm CI for el8 is broken

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=40595#comment-40595
 ] 

Nir Soffer commented on OVIRT-2965:
---

On Sun, Jun 21, 2020 at 1:18 PM Amit Bawer  wrote:
>
> Hi
>
> Seems that in the last few days the el8 CI for vdsm cannot run, were there 
> any el8 repo changes?
>
> [2020-06-21T09:20:03.003Z] Mock Version: 2.2
> [2020-06-21T09:20:03.003Z] INFO: Mock Version: 2.2
> [2020-06-21T09:20:03.987Z] Start: dnf install
> [2020-06-21T09:20:12.326Z] ERROR: Command failed:
> [2020-06-21T09:20:12.326Z]  # /usr/bin/dnf --installroot 
> /var/lib/mock/epel-8-x86_64-8e9eeb575ab4da7bf0fbfdc80a25b9c0-30232/root/ 
> --releasever 8 --setopt=deltarpm=False --allowerasing --disableplugin=local 
> --disableplugin=spacewalk --disableplugin=local --disableplugin=spacewalk 
> install dnf tar gcc-c++ redhat-rpm-config which xz sed make bzip2 gzip gcc 
> coreutils unzip shadow-utils diffutils cpio bash gawk rpm-build info patch 
> util-linux findutils grep python36 autoconf automake createrepo dnf dnf-utils 
> e2fsprogs gcc gdb git iproute-tc iscsi-initiator-utils libguestfs-tools-c 
> lshw make openvswitch ovirt-imageio-common python3-augeas python3-blivet 
> python3-coverage python3-dateutil python3-dbus python3-decorator 
> python3-devel python3-dmidecode python3-inotify python3-ioprocess-1.4.1 
> python3-libselinux python3-libvirt python3-magic python3-netaddr python3-nose 
> python3-pip python3-policycoreutils python3-pyyaml python3-requests 
> python3-sanlock python3-six python3-yaml rpm-build rpmlint sanlock sudo 
> systemd-udev xfsprogs --setopt=tsflags=nocontexts
> [2020-06-21T09:20:12.326Z] No matches found for the following disable plugin 
> patterns: local, spacewalk
> [2020-06-21T09:20:12.326Z] Last metadata expiration check: 0:00:02 ago on Sun 
> Jun 21 09:20:07 2020.
> [2020-06-21T09:20:12.326Z] No match for argument: openvswitch
> [2020-06-21T09:20:12.326Z] Error: Unable to find a match: openvswitch
>
> Taken from 
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/21966/consoleText

I think this is a result of trying to move to CentOS 8.2 before all
the packages are
ready.

Please use Travis until this issue is resolved. You should use it
anyway for storage
patches.

Nir

> Vdsm CI for el8 is broken
> -
>
> Key: OVIRT-2965
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2965
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Amit Bawer
>Assignee: infra
>
> Hi
> Seems that in the last few days the el8 CI for vdsm cannot run, were there
> any el8 repo changes?
> [2020-06-21T09:20:03.003Z] Mock Version: 2.2
> [2020-06-21T09:20:03.003Z] INFO: Mock Version: 2.2
> [2020-06-21T09:20:03.987Z] Start: dnf install
> [2020-06-21T09:20:12.326Z] ERROR: Command failed:
> [2020-06-21T09:20:12.326Z]  # /usr/bin/dnf --installroot
> /var/lib/mock/epel-8-x86_64-8e9eeb575ab4da7bf0fbfdc80a25b9c0-30232/root/
> --releasever 8 --setopt=deltarpm=False --allowerasing --disableplugin=local
> --disableplugin=spacewalk --disableplugin=local --disableplugin=spacewalk
> install dnf tar gcc-c++ redhat-rpm-config which xz sed make bzip2 gzip gcc
> coreutils unzip shadow-utils diffutils cpio bash gawk rpm-build info patch
> util-linux findutils grep python36 autoconf automake createrepo dnf
> dnf-utils e2fsprogs gcc gdb git iproute-tc iscsi-initiator-utils
> libguestfs-tools-c lshw make openvswitch ovirt-imageio-common
> python3-augeas python3-blivet python3-coverage python3-dateutil
> python3-dbus python3-decorator python3-devel python3-dmidecode
> python3-inotify python3-ioprocess-1.4.1 python3-libselinux python3-libvirt
> python3-magic python3-netaddr python3-nose python3-pip
> python3-policycoreutils python3-pyyaml python3-requests python3-sanlock
> python3-six python3-yaml rpm-build rpmlint sanlock sudo systemd-udev
> xfsprogs --setopt=tsflags=nocontexts
> [2020-06-21T09:20:12.326Z] No matches found for the following disable
> plugin patterns: local, spacewalk
> [2020-06-21T09:20:12.326Z] Last metadata expiration check: 0:00:02 ago on
> Sun Jun 21 09:20:07 2020.
> [2020-06-21T09:20:12.326Z] No match for argument: openvswitch
> [2020-06-21T09:20:12.326Z] Error: Unable to find a match: openvswitch
> Taken from
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/21966/consoleText
> Thanks



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100130)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ANKY3OQO3UDFM6DLY6EWOIBWPVMMY4QT/

Add Evgheni Dereveanchin to ovirt quay owners group

[Nir Soffer]

I suggest to add Evgheni to the owners group:
https://quay.io/organization/ovirt/teams/owners

So he managed groups and repos.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/76CWLH5P4D55D447BRESJYJ37PPTYXGZ/

[JIRA] (OVIRT-2940) Stuck job - running 8 days

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2940:
-

 Summary: Stuck job - running 8 days
 Key: OVIRT-2940
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2940
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


I found this stuck job in jenkins:
https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/5660/

Don't we have a timeout for killing run away jobs?



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100126)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/SNPYOWBWMMMYNTQ4DASYHSJ45AUQ7D2I/

[JIRA] (OVIRT-2939) Assign admin role for nsoffer for project ioprocess on github

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2939:
-

 Summary: Assign admin role for nsoffer for project ioprocess on 
github
 Key: OVIRT-2939
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2939
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


I need admin role for:
https://github.com/oVirt/ioprocess

So I can se tup webhooks and other stuff.

I have correct permissions in
https://github.com/oVirt/vdsm
https://github.com/oVirt/ovirt-imageio

Thanks,
Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100126)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MNCLAE7QKQPPUZHLR52HSQA2WSVPMDWB/

Re: [ovirt-devel] Re: oVirt and Fedora

[Nir Soffer]

On Mon, May 11, 2020 at 2:24 PM Neal Gompa  wrote:
>
> On Mon, May 11, 2020 at 2:16 AM Sandro Bonazzola  wrote:
>>
>> If you have followed the oVirt project for a few releases you already know 
>> oVirt has struggled to keep the pace with the fast innovation cycles Fedora 
>> Project is following.
>>
>> Back in September 2019 CentOS project launched CentOS Stream as a rolling 
>> preview of future RHEL kernels and features, providing an upstream 
>> development platform for ecosystem developers that sits between Fedora and 
>> RHEL.
>>
>> Since then the oVirt project tried to keep the software working on Fedora, 
>> CenOS Stream, and RHEL/CentOS but it became quickly evident the project 
>> lacked resources to keep the project running on three platforms. Further, 
>> our user surveys show that oVirt users strongly prefer using oVirt on CentOS 
>> and RHEL.
>>
>> With the upcoming end of life of Fedora 30 the oVirt project has decided to 
>> stop trying to keep the pace with this amazing platform, focusing on 
>> stabilizing the software codebase on RHEL / CentOS Linux. By focusing our 
>> resources and community efforts on RHEL/CentOS Linux and Centos Stream, we 
>> can provide better support for those platforms and use more time for moving 
>> oVirt forward.
>>
>
> This is a humongous mistake. Almost everything with virtualization and 
> storage starts in Fedora. And there are some configurations that will not be 
> possible in CentOS Stream because of the nature of it.
>
> As far as the oVirt software keeping up with Fedora, the main problem here 
> has always been that people aren't integrating their software into the 
> distribution itself. That's how everything can get tested together. And this 
> comes back to the old bug about fixing vdsm so that it doesn't use /rhev, but 
> instead something FHS-compliant (RHBZ#1369102). Once that is resolved, pretty 
> much the entire stack can go into Fedora. And then you benefit from the 
> Fedora community being able to use, test, and contribute to the oVirt 
> project. As it stands, why would anyone do this for you when you don't even 
> run on the cutting edge platform that feeds into Red Hat Enterprise Linux?

This was actually fixed a long time ago. With this commit:
https://github.com/oVirt/vdsm/commit/67ba9c4bc860840d6e103fe604b16f494f60a09d

You can configure a compatible vdsm that does not use /rhev.

Of course it is not backward compatible, for this we need much more
work to support live migration
between old and new vdsm using different data-center configurations.

> It also seems like the oVirt folks are not learning from the mistakes of the 
> RDO project. They gave up on Fedora several years ago, and wound up spending 
> close to two years playing catchup on Python 3, DNF, modularity, 
> virtualization packaging changes, storage APIs, and everything else all at 
> once. They ground to a halt. They paid a price for not keeping up. And their 
> excuse of unaligned lifecycles stopped being true more than two years ago, 
> when OpenStack's release cycles aligned on Fedora's again. They also proved 
> that Fedora's "churn" wasn't the problem because when push comes to shove, 
> they were able to do something based on Fedora 28 (knowing it was the base 
> for RHEL 8).
>
> CentOS Stream is worthless in most respects because you aren't really testing 
> or integrating anything new most of the time, you're just making new releases 
> of your software on a stale platform. Again, the purpose of CentOS Stream is 
> to provide a window into the RHEL stream development, which by the nature of 
> things isn't very useful for future-proofing.
>
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/JM5UNXG6YI7R23NUR5LLW3TV3KDMPU2A/
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/M47LI42YTFRCMNW5YTYVAYBXO277JXCN/

Re: [ovirt-devel] oVirt and Fedora

[Nir Soffer]

On Mon, May 11, 2020 at 10:01 AM Sandro Bonazzola 
wrote:

>
>
> Il giorno lun 11 mag 2020 alle ore 08:50 Nir Soffer 
> ha scritto:
>
>> On Mon, May 11, 2020 at 9:16 AM Sandro Bonazzola 
>> wrote:
>>
>>> If you have followed the oVirt project for a few releases you already
>>> know oVirt has struggled to keep the pace with the fast innovation cycles
>>> Fedora Project is following.
>>>
>>> Back in September 2019 CentOS project launched CentOS Stream as a
>>> rolling preview of future RHEL kernels and features, providing an upstream
>>> development platform for ecosystem developers that sits between Fedora and
>>> RHEL.
>>>
>>> Since then the oVirt project tried to keep the software working on
>>> Fedora, CenOS Stream, and RHEL/CentOS but it became quickly evident the
>>> project lacked resources to keep the project running on three platforms.
>>> Further, our user surveys show that oVirt users strongly prefer using oVirt
>>> on CentOS and RHEL.
>>>
>>> With the upcoming end of life of Fedora 30 the oVirt project has decided
>>> to stop trying to keep the pace with this amazing platform, focusing on
>>> stabilizing the software codebase on RHEL / CentOS Linux. By focusing our
>>> resources and community efforts on RHEL/CentOS Linux and Centos Stream, we
>>> can provide better support for those platforms and use more time for moving
>>> oVirt forward.
>>>
>>
>> Where was this discussed?
>>
>> There is nothing about this in de...@ovirt.org or any other public
>> mailing list.
>>
>> I think this is a big mistake. It will mainly harm development since
>> Fedora is the only platform where
>> we can test early upstream changes, many months (and sometimes years)
>> before the packages reach
>> RHEL/CentOS.
>>
>> Nir
>>
>
> It has been discussed within team leads meeting and made clear by replies
> we keep giving when someone ask about Fedora.
> See "[ovirt-users] install oVirt on Fedora31", "[ovirt-users] oVirt orb
> python3/fedora 31 support".
>
> This doesn't mean that individual developers can try to get their packages
> working on Fedora or test their code on Fedora.
> This means that we are not committed to keep trying to support Fedora as a
> project.
>

Again, there was no discussion about this in public, but we can start the
discussion here.

The result of such a change is that soon there will be no way to install
vdsm on Fedora,
because we need only one maintainer that does not care about Fedora.

Then we cannot test vdsm with upstream libvirt and qemu, which will harm
our ability to develop
features like incremental backup, which was developed in the last year on
Fedora, using upstream
libvirt and qemu or upstream patches.

Another example, if you want to test sanlock fixes that will be available
in RHEL 8.3, the only way
to do this now is on Fedora - the fixes are available in updates-testing.
So we can make sure our
code is compatible with new sanlock now, or wait until 8.3 is out and find
about regressions
very late before the release.

This is a common trend  in the last 6.5 years - oVirt is not tested on
Fedora (e.g. no ovirt
systems on Fedora), leading to late detection of issues, and regressions in
new versions
in RHEL/CentOS.

What we should do instead is focus on Fedora support and testing early
upstream packages.

Currently oVirt is tested on CentOS 8.1, which is irrelevant to oVirt 4.4.
What we really need
is to test on RHEL 8.2 nightly builds, but this is not available to the
public and cannot be
used by an upstream project. But Fedora 32 is available and includes all
the packages
needed for valuable testing, so this should be our main platform for
testing.

CentOS stream could be a good solution for testing, if we could get the
advanced virt module
*before* the module is released in RHEL. Otherwise we test on old version
and we don't detect
issues in time.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7VNRYA4773D7APLDO74DL22YBT4H5G65/

Re: [ovirt-devel] oVirt and Fedora

[Nir Soffer]

On Mon, May 11, 2020 at 9:16 AM Sandro Bonazzola 
wrote:

> If you have followed the oVirt project for a few releases you already know
> oVirt has struggled to keep the pace with the fast innovation cycles Fedora
> Project is following.
>
> Back in September 2019 CentOS project launched CentOS Stream as a rolling
> preview of future RHEL kernels and features, providing an upstream
> development platform for ecosystem developers that sits between Fedora and
> RHEL.
>
> Since then the oVirt project tried to keep the software working on Fedora,
> CenOS Stream, and RHEL/CentOS but it became quickly evident the project
> lacked resources to keep the project running on three platforms. Further,
> our user surveys show that oVirt users strongly prefer using oVirt on
> CentOS and RHEL.
>
> With the upcoming end of life of Fedora 30 the oVirt project has decided
> to stop trying to keep the pace with this amazing platform, focusing on
> stabilizing the software codebase on RHEL / CentOS Linux. By focusing our
> resources and community efforts on RHEL/CentOS Linux and Centos Stream, we
> can provide better support for those platforms and use more time for moving
> oVirt forward.
>

Where was this discussed?

There is nothing about this in de...@ovirt.org or any other public mailing
list.

I think this is a big mistake. It will mainly harm development since Fedora
is the only platform where
we can test early upstream changes, many months (and sometimes years)
before the packages reach
RHEL/CentOS.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/6CP7NPGHRFBULBWLD67HNHQWY4WEMZER/

[JIRA] (OVIRT-2935) Jenkins succeeded, Code-Review +1

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2935:
-

 Summary: Jenkins succeeded, Code-Review +1
 Key: OVIRT-2935
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2935
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


It is nice to get good review from Jeknins, but I don't think it is smart enough
to do code review, and it should really use Continuous-Integration+1.

Jenkins CI
Patch Set 2: Code-Review+1
Build Successful
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/2772/ : SUCCESS

See https://gerrit.ovirt.org/c/108772/2#message-6225787b_94ed3eac



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100126)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/YI3LP2ANAFPII3IX22KC4RT3SQU4YXCS/

[JIRA] (OVIRT-2903) Enable ovirt-imageio builds in travis

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2903:
-

 Summary: Enable ovirt-imageio builds in travis
 Key: OVIRT-2903
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2903
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


We have lot of tests that can work only on travis, and we use travis to run
with private forks like:
https://travis-ci.org/github/nirs/ovirt-imageio/builds

I tried to add a webhook to enable builds on:
https://travis-ci.org/github/oVirt/ovirt-imageio

But it seems that I don't have enough permissions on travis
to enable the build.

Please enable travis build for this project.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100124)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/5U3GEJIBG6Y2ESRATKANF5DY3BI532ZP/

[JIRA] (OVIRT-2897) ppc64le build-artifacts/check-merged job failing in "mock init"

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2897:
-

 Summary: ppc64le build-artifacts/check-merged job failing in "mock 
init"
 Key: OVIRT-2897
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2897
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


The ppc64le build artifacts jobs fail now in "mock init". Looks like
an environmental issue.

Here are few failing builds:
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/2574/
https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/2573/
https://jenkins.ovirt.org/job/ovirt-imageio_standard-on-merge/573/

This seems to be the last successful build, 4 days ago:
https://jenkins.ovirt.org/job/ovirt-imageio_standard-on-merge/563/



[2020-04-04T19:28:16.332Z] + ../jenkins/mock_configs/mock_runner.sh
--execute-script automation/build-artifacts.py3.sh --mock-confs-dir
../jenkins/mock_configs --secrets-file
/home/jenkins/workspace/ovirt-imageio_standard-check-patch/std_ci_secrets.yaml
--try-proxy --timeout-duration 10800 --try-mirrors
http://mirrors.phx.ovirt.org/repos/yum/all_latest.json 'el8.*ppc64le'

[2020-04-04T19:28:16.332Z]
##

[2020-04-04T19:28:16.332Z]
##

[2020-04-04T19:28:16.332Z] ## Sat Apr  4 19:28:16 UTC 2020 Running
env: el8:epel-8-ppc64le

[2020-04-04T19:28:16.332Z]
##

[2020-04-04T19:28:16.332Z]
@@

[2020-04-04T19:28:16.332Z] @@ Sat Apr  4 19:28:16 UTC 2020 Running
chroot for script: automation/build-artifacts.py3.sh

[2020-04-04T19:28:16.332Z]
@@

[2020-04-04T19:28:16.599Z] Using base mock conf
../jenkins/mock_configs/epel-8-ppc64le.cfg

[2020-04-04T19:28:16.599Z] WARN: Unable to find req file
automation/build-artifacts.py3.req or
automation/build-artifacts.py3.req.el8, skipping req

[2020-04-04T19:28:16.599Z] Using proxified config
../jenkins/mock_configs/epel-8-ppc64le_proxied.cfg

[2020-04-04T19:28:16.599Z] Generating temporary mock conf
/home/jenkins/workspace/ovirt-imageio_standard-check-patch/ovirt-imageio/mocker-epel-8-ppc64le.el8

[2020-04-04T19:28:16.599Z] Skipping mount points

[2020-04-04T19:28:16.599Z] WARN: Unable to find repos file
automation/build-artifacts.py3.repos or
automation/build-artifacts.py3.repos.el8, skipping repos

[2020-04-04T19:28:16.599Z] Using chroot cache =
/var/cache/mock/epel-8-ppc64le-ef003ab0662b9b04a2143d179b949705

[2020-04-04T19:28:16.599Z] Using chroot dir =
/var/lib/mock/epel-8-ppc64le-ef003ab0662b9b04a2143d179b949705-15351

[2020-04-04T19:28:16.599Z] Skipping environment variables

[2020-04-04T19:28:16.599Z] == Initializing chroot

[2020-04-04T19:28:16.599Z] mock \

[2020-04-04T19:28:16.599Z] --old-chroot \

[2020-04-04T19:28:16.599Z]
--configdir="/home/jenkins/workspace/ovirt-imageio_standard-check-patch/ovirt-imageio"
\

[2020-04-04T19:28:16.599Z] --root="mocker-epel-8-ppc64le.el8" \

[2020-04-04T19:28:16.599Z] --resultdir="/tmp/mock_logs.FrjZTOEI/init" \

[2020-04-04T19:28:16.599Z] --init

[2020-04-04T19:28:17.186Z] WARNING: Could not find required logging
config file: 
/home/jenkins/workspace/ovirt-imageio_standard-check-patch/ovirt-imageio/logging.ini.
Using default...

[2020-04-04T19:28:17.186Z] INFO: mock.py version 1.4.21 starting
(python version = 3.6.8)...

[2020-04-04T19:28:17.186Z] Start(bootstrap): init plugins

[2020-04-04T19:28:17.186Z] INFO: selinux enabled

[2020-04-04T19:28:17.845Z] Finish(bootstrap): init plugins

[2020-04-04T19:28:17.845Z] Start: init plugins

[2020-04-04T19:28:17.845Z] INFO: selinux enabled

[2020-04-04T19:28:17.845Z] Finish: init plugins

[2020-04-04T19:28:17.845Z] INFO: Signal handler active

[2020-04-04T19:28:17.845Z] Start: run

[2020-04-04T19:28:17.845Z] Start: clean chroot

[2020-04-04T19:28:17.845Z] Finish: clean chroot

[2020-04-04T19:28:17.845Z] Start(bootstrap): chroot init

[2020-04-04T19:28:17.845Z] INFO: calling preinit hooks

[2020-04-04T19:28:17.845Z] INFO: enabled root cache

[2020-04-04T19:28:17.846Z] INFO: enabled dnf cache

[2020-04-04T19:28:17.846Z] Start(bootstrap): cleaning dnf metadata

[2020-04-04T19:28:17.846Z] Finish(bootstrap): cleaning dnf metadata

[2020-04-04T19:28:17.846Z] INFO: enabled HW Info plugin

[2020-04-04T19:28:17.846Z] Mock Version: 1.4.21

[2020-04-04T19:28:17.846Z] INFO: Mock Version: 1.4.21

[2020-04-04T19:28:18.119Z] Start(bootstrap): yum install

[2020-04-04T19:28:23.477Z] ERROR: Command failed:

[2020-04-04T19:28:23.477Z]  # /usr/bin/yum --installroot
/var/lib/mock/epel-8-ppc64le-ef003ab0662b9b04a2143d179b949705-bootstrap-15351/root/
--releasever 8 install dnf dnf-p

Re: [CentOS-devel] CBS mash repositories going away next wednesday

[Nir Soffer]

On Mon, Mar 23, 2020 at 4:45 PM Sandro Bonazzola 
wrote:

> Heads up that we may have disruptions due to repositories being sunsetted.
> We'll need to update all the places pointing to repos hosted on
> cbs.centos.org to use https://buildlogs.centos.org/ instead.
> Note that repository structure is different so manual work is needed to
> adapt.
>

This seems to break ovirt-imageio builds, since we still have proxy builds
for el7:
https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-imageio_standard-check-patch/detail/ovirt-imageio_standard-check-patch/2476/pipeline

[2020-03-26T16:00:42.110Z]
https://cbs.centos.org/repos/virt7-ovirt-43-testing/ppc64le/os/repodata/repomd.xml:
[Errno 14] HTTPS Error 404 - Not Found

We are using:

$ cat ../automation/check-patch.repos.el7
centos-ovirt43-testing,
https://cbs.centos.org/repos/virt7-ovirt-43-testing/$basearch/os/
kvm-common,
https://cbs.centos.org/repos/virt7-kvm-common-testing/$basearch/os/

Since on master we are going to drop the proxy very soon, I think we can
disable the el7 bulids now.

Nir

-- Forwarded message -
> Da: Fabian Arrotin 
> Date: lun 23 mar 2020 alle ore 13:44
> Subject: [CentOS-devel] CBS mash repositories going away next wednesday
> To: The CentOS developers mailing list. 
>
>
> Tied to the previous announce, the new signing system will not depend on
> cbs-content-control, which itself was using repositories prepared by mash.
>
> Those repositories, while normally not meant to be consumed, will so go
> away.
> So all https://cbs.centos.org/repos/* will disappear next wednesday (FWIW)
>
> --
> Fabian Arrotin
> The CentOS Project | https://www.centos.org
> gpg key: 17F3B7A1 | twitter: @arrfab
>
> ___
> CentOS-devel mailing list
> centos-de...@centos.org
> https://lists.centos.org/mailman/listinfo/centos-devel
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R RHV
>
> Red Hat EMEA 
>
> sbona...@redhat.com
> *
> *
> *Red Hat respects your work life balance. Therefore there is no need to
> answer this email out of your office hours.
> *
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/3SDZHYPFHYWQ6I6J5A5FPHZLZVO6TJNP/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/BOBIZX5O6FHZZM5H6NGDABLZTXYVMLMU/

Re: OST fails in 002_bootstrap_pytest.py - setup_storage.sh

[Nir Soffer]

On Mon, Mar 23, 2020 at 3:26 PM Marcin Sobczyk  wrote:
>
>
>
> On 3/23/20 2:17 PM, Nir Soffer wrote:
> > On Mon, Mar 23, 2020 at 1:25 PM Marcin Sobczyk  wrote:
> >>
> >>
> >> On 3/21/20 1:18 AM, Nir Soffer wrote:
> >>
> >> On Fri, Mar 20, 2020 at 9:35 PM Nir Soffer  wrote:
> >>> Looks like infrastructure issue setting up storage on engine host.
> >>>
> >>> Here are 2 failing builds with unrelated changes:
> >>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6677/
> >>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6678/
> >>
> >> Rebuilding still fails in setup_storage:
> >>
> >> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6679/testReport/
> >> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6680/testReport/
> >>
> >>>
> >>> Is this a known issue?
> >>>
> >>> Error Message
> >>>
> >>> AssertionError: setup_storage.sh failed. Exit code is 1 assert 1 == 0   
> >>> -1   +0
> >>>
> >>> Stacktrace
> >>>
> >>> prefix = 
> >>>
> >>>  @pytest.mark.run(order=14)
> >>>  def test_configure_storage(prefix):
> >>>  engine = prefix.virt_env.engine_vm()
> >>>  result = engine.ssh(
> >>>  [
> >>>  '/tmp/setup_storage.sh',
> >>>  ],
> >>>  )
> >>>>assert result.code == 0, 'setup_storage.sh failed. Exit code is 
> >>>> %s' % result.code
> >>> E   AssertionError: setup_storage.sh failed. Exit code is 1
> >>> E   assert 1 == 0
> >>> E -1
> >>> E +0
> >>>
> >>>
> >>> The pytest traceback is nice, but in this case it is does not show any 
> >>> useful info.
> >>>
> >>> Since we run a script using ssh, the error message should include the 
> >>> process stdout and stderr
> >>> which probably can explain the failure.
> >> I posted https://gerrit.ovirt.org/#/c/107830/ to improve logging during 
> >> storage setup.
> >> Unfortunately AFAICS it didn't fail, so I guess we'll have to merge it and 
> >> wait for a failed job to get some helpful logs.
> > Thanks.
> >
> > It still fails for me with current code:
> > https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6689/testReport/
> >
> > Same when using current vdsm master.
> Updated the patch according to your suggestions and currently trying out
> OST for the 4th time -
> all previous runs succeeded. I guess I'm out of luck :)

It succeeds on your local OST setup but fail on Jenkins?

> >>> Also I wonder why this code is called as a test (test_configure_storage). 
> >>> This looks like setup
> >>> step so it should run as a fixture.
> >> That's true, but the pytest porting effort was about providing a bare 
> >> minimum to move away from nose.
> >> Organizing the tests into proper setup/fixtures is a huge task and will be 
> >> probably implemented
> >> incrementally in the nearest future.
> > Understood
> >
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ENQHFTYQJLBWXAI2MMY7OB6GQNURLBIA/

Re: OST fails in 002_bootstrap_pytest.py - setup_storage.sh

[Nir Soffer]

On Mon, Mar 23, 2020 at 1:25 PM Marcin Sobczyk  wrote:
>
>
>
> On 3/21/20 1:18 AM, Nir Soffer wrote:
>
> On Fri, Mar 20, 2020 at 9:35 PM Nir Soffer  wrote:
>>
>> Looks like infrastructure issue setting up storage on engine host.
>>
>> Here are 2 failing builds with unrelated changes:
>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6677/
>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6678/
>
>
> Rebuilding still fails in setup_storage:
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6679/testReport/
> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6680/testReport/
>
>>
>>
>> Is this a known issue?
>>
>> Error Message
>>
>> AssertionError: setup_storage.sh failed. Exit code is 1 assert 1 == 0   -1   
>> +0
>>
>> Stacktrace
>>
>> prefix = 
>>
>> @pytest.mark.run(order=14)
>> def test_configure_storage(prefix):
>> engine = prefix.virt_env.engine_vm()
>> result = engine.ssh(
>> [
>> '/tmp/setup_storage.sh',
>> ],
>> )
>> >   assert result.code == 0, 'setup_storage.sh failed. Exit code is %s' 
>> > % result.code
>> E   AssertionError: setup_storage.sh failed. Exit code is 1
>> E   assert 1 == 0
>> E -1
>> E +0
>>
>>
>> The pytest traceback is nice, but in this case it is does not show any 
>> useful info.
>>
>> Since we run a script using ssh, the error message should include the 
>> process stdout and stderr
>> which probably can explain the failure.
>
> I posted https://gerrit.ovirt.org/#/c/107830/ to improve logging during 
> storage setup.
> Unfortunately AFAICS it didn't fail, so I guess we'll have to merge it and 
> wait for a failed job to get some helpful logs.

Thanks.

It still fails for me with current code:
https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6689/testReport/

Same when using current vdsm master.

>> Also I wonder why this code is called as a test (test_configure_storage). 
>> This looks like setup
>> step so it should run as a fixture.
>
> That's true, but the pytest porting effort was about providing a bare minimum 
> to move away from nose.
> Organizing the tests into proper setup/fixtures is a huge task and will be 
> probably implemented
> incrementally in the nearest future.

Understood
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/5AP27PDBW6GU4E6ADXUDV5COD63477IC/

Re: OST fails in 002_bootstrap_pytest.py - setup_storage.sh

[Nir Soffer]

On Fri, Mar 20, 2020 at 9:35 PM Nir Soffer  wrote:

> Looks like infrastructure issue setting up storage on engine host.
>
> Here are 2 failing builds with unrelated changes:
> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6677/
> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6678/
>

Rebuilding still fails in setup_storage:

https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6679/testReport/
https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6680/testReport/


>
> Is this a known issue?
>
> Error Message
>
> AssertionError: setup_storage.sh failed. Exit code is 1 assert 1 == 0   -1
>   +0
>
> Stacktrace
>
> prefix = 
>
> @pytest.mark.run(order=14)
> def test_configure_storage(prefix):
> engine = prefix.virt_env.engine_vm()
> result = engine.ssh(
> [
> '/tmp/setup_storage.sh',
> ],
> )
> >   assert result.code == 0, 'setup_storage.sh failed. Exit code is
> %s' % result.code
> E   AssertionError: setup_storage.sh failed. Exit code is 1
> E   assert 1 == 0
> E -1
> E +0
>
>
> The pytest traceback is nice, but in this case it is does not show any
> useful info.
>
> Since we run a script using ssh, the error message should include the
> process stdout and stderr
> which probably can explain the failure.
>
> Also I wonder why this code is called as a test (test_configure_storage).
> This looks like setup
> step so it should run as a fixture.
>
> Nir
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MET62OLXJPMC2K7C25QAGJ7PX6BQ4VGQ/

OST fails in 002_bootstrap_pytest.py - setup_storage.sh

[Nir Soffer]

Looks like infrastructure issue setting up storage on engine host.

Here are 2 failing builds with unrelated changes:
https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6677/
https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6678/

Is this a known issue?

Error Message

AssertionError: setup_storage.sh failed. Exit code is 1 assert 1 == 0   -1
  +0

Stacktrace

prefix = 

@pytest.mark.run(order=14)
def test_configure_storage(prefix):
engine = prefix.virt_env.engine_vm()
result = engine.ssh(
[
'/tmp/setup_storage.sh',
],
)
>   assert result.code == 0, 'setup_storage.sh failed. Exit code is %s'
% result.code
E   AssertionError: setup_storage.sh failed. Exit code is 1
E   assert 1 == 0
E -1
E +0


The pytest traceback is nice, but in this case it is does not show any
useful info.

Since we run a script using ssh, the error message should include the
process stdout and stderr
which probably can explain the failure.

Also I wonder why this code is called as a test (test_configure_storage).
This looks like setup
step so it should run as a fixture.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K3LVUTLUAZNB2CHFWK32WEFS2L2JTLXZ/

Re: Sanlock lockspace inquire failure (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.3 - Build # 366 - Failure!)

[Nir Soffer]

On Tue, Mar 3, 2020 at 9:07 AM Yedidyah Bar David  wrote:
>
> On Mon, Mar 2, 2020 at 12:52 AM Nir Soffer  wrote:
> >
> > On Sun, Mar 1, 2020 at 10:10 AM Yedidyah Bar David  wrote:
> > >
> > > Hi all,
> > >
> > > On Sun, Mar 1, 2020 at 6:06 AM  wrote:
> > > >
> > > > Project: 
> > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/
> > > > Build: 
> > > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/366/
> > >
> > > I think the root cause is:
> > >
> > > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/366/artifact/exported-artifacts/test_logs/he-basic-suite-4.3/post-008_restart_he_vm.py/lago-he-basic-suite-4-3-host-0/_var_log/ovirt-hosted-engine-ha/broker.log
> > >
> > > StatusStorageThread::ERROR::2020-02-29
> > > 23:03:04,671::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
> > > Failed to update state.
> > > Traceback (most recent call last):
> > >   File 
> > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
> > > line 82, in run
> > > if (self._status_broker._inquire_whiteboard_lock() or
> > >   File 
> > > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
> > > line 195, in _inquire_whiteboard_lock
> > > self.host_id, self._lease_file)
> > > SanlockException: (104, 'Sanlock lockspace inquire failure',
> > > 'Connection reset by peer')
> >
> > Can you point us to the source using the sanlock API?
>
> I think it's right where the above error message says it is:
>
> https://github.com/oVirt/ovirt-hosted-engine-ha/blob/master/ovirt_hosted_engine_ha/broker/status_broker.py#L193
>
> >
> > The messages looks like client error accessing sanlock server socket
> > (maybe someone restarted sanlock at that point?)
>
> Maybe, I failed to find evidence :-)
>
> > but it may also be some error code reused for sanlock internal error for
> > accessing the storage.
> >
> > Usually you can find more info about the error in /var/sanlock.log
>
> Couldn't find anything:
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/366/artifact/exported-artifacts/test_logs/he-basic-suite-4.3/post-008_restart_he_vm.py/lago-he-basic-suite-4-3-host-0/_var_log/sanlock.log

This is the end of the log:

2020-02-29 22:51:46 5313 [6267]: s8 host 2 1 5295
eb348ac8-979d-48c4-b0ce-eb9bac37efe8.lago-he-ba
2020-02-29 23:03:39 6027 [6273]: s9 lockspace
hosted-engine:1:/var/run/vdsm/storage/93778fac-6989-4c6d-9bc0-edb0f43b35b1/19705ecb-0854-4f7f-8df6-146586c97b81/c32dbeaa-c935-43c9-a252-a7db6174f1c3:0

The error happened on:
2020-02-29 23:03:04,671

There is nothing in sanlock log, so this does not look like a network
issue. We would see
error logs in sanlock log in such case.

sanlock inq_lockspace returns 0, -ENOENT, or -EINPROGRESS, so the
error did not come
from sanlock.

So this looks like client failed when sending the request to sanlock,  here:
https://github.com/nirs/sanlock/blob/53f7f0284c084ac2e4542fd1f71d0406075adb5d/src/client.c#L89
https://github.com/nirs/sanlock/blob/53f7f0284c084ac2e4542fd1f71d0406075adb5d/src/client.c#L99

In vdsm log we see that sanlock was fine 15 seconds before the error:

2020-02-29 23:03:04,787-0500 INFO  (jsonrpc/5) [vdsm.api] FINISH
repoStats return={u'3f942432-c244-4969-b4bf-2ac02dee2f7f': {'code': 0,
'actual': True, 'version': 5, 'acquir
ed': True, 'delay': '0.00257567', 'lastCheck': '1.7', 'valid': True},
u'6c82d788-cd54-440d-aef0-ba24b0d8d7c4': {'code': 0, 'actual': True,
'version': 5, 'acquired': True, 'd
elay': '0.00595748', 'lastCheck': '0.5', 'valid': True},
u'93778fac-6989-4c6d-9bc0-edb0f43b35b1': {'code': 0, 'actual': True,
'version': 5, 'acquired': True, 'delay': '0.003
07431', 'lastCheck': '0.6', 'valid': True},
u'f970efe3-74a5-47ea-8319-f4c96aee4008': {'code': 0, 'actual': True,
'version': 0, 'acquired': True, 'delay': '0.00354385', 'last
Check': '2.3', 'valid': True},
u'8fc30a42-6a05-4a6b-8f43-bb6438868a84': {'code': 0, 'actual': True,
'version': 0, 'acquired': True, 'delay': '0.00254181', 'lastCheck':
'1.7'
, 'valid': True}} from=::1,48818,
task_id=9b2892b5-bbf4-4516-bb7b-353de93bbd8a (api:54)

Then we see the hosted engine error:

2020-02-29 23:03:09,832-0500 ERROR (jsonrpc/5) [root] failed to
retrieve Hosted Engine HA score (api:200)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182,
in _getHaInfo
stats = instance.get_all_stats(timeout=5)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 97

Re: Sanlock lockspace inquire failure (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.3 - Build # 366 - Failure!)

[Nir Soffer]

On Sun, Mar 1, 2020 at 10:10 AM Yedidyah Bar David  wrote:
>
> Hi all,
>
> On Sun, Mar 1, 2020 at 6:06 AM  wrote:
> >
> > Project: 
> > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/
> > Build: 
> > https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/366/
>
> I think the root cause is:
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/366/artifact/exported-artifacts/test_logs/he-basic-suite-4.3/post-008_restart_he_vm.py/lago-he-basic-suite-4-3-host-0/_var_log/ovirt-hosted-engine-ha/broker.log
>
> StatusStorageThread::ERROR::2020-02-29
> 23:03:04,671::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
> Failed to update state.
> Traceback (most recent call last):
>   File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
> line 82, in run
> if (self._status_broker._inquire_whiteboard_lock() or
>   File 
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
> line 195, in _inquire_whiteboard_lock
> self.host_id, self._lease_file)
> SanlockException: (104, 'Sanlock lockspace inquire failure',
> 'Connection reset by peer')

Can you point us to the source using the sanlock API?

The messages looks like client error accessing sanlock server socket
(maybe someone restarted sanlock at that point?)
but it may also be some error code reused for sanlock internal error for
accessing the storage.

Usually you can find more info about the error in /var/sanlock.log

> This caused the broker to restart itself,

Restarting because sanlock failed does sound like useful error handling
for broker clients.

> and while it was doing that,
> OST did 'hosted-engine --vm-status --json', which failed, thus failing
> the build.

If the broker may restart itself on errors, clients need to use a retry
mechanism to deal with the restarts, so the test should probably have
a retry mechanism before it fails.

> This seems to me like another case of a communication problem in CI.
> Not sure what else could have caused it to fail to inquire the status
> of the lock. This (communication) issue was mentioned several times in
> the past already. Are we doing anything re this?
>
> Thanks and best regards,
>
> > Build Number: 366
> > Build Status:  Failure
> > Triggered By: Started by timer
> >
> > -
> > Changes Since Last Success:
> > -
> > Changes for Build #366
> > [Marcin Sobczyk] el8: Don't try to collect whole '/etc/httpd' dir
> >
> >
> >
> >
> > -
> > Failed Tests:
> > -
> > 1 tests failed.
> > FAILED:  008_restart_he_vm.clear_global_maintenance
> >
> > Error Message:
> > 1 != 0
> >  >> begin captured logging << 
> > root: INFO: Waiting For System Stability...
> > lago.ssh: DEBUG: start task:29a79ef5-e211-4672-ac5b-12bf0e5f8ee9:Get ssh 
> > client for lago-he-basic-suite-4-3-host-0:
> > lago.ssh: DEBUG: end task:29a79ef5-e211-4672-ac5b-12bf0e5f8ee9:Get ssh 
> > client for lago-he-basic-suite-4-3-host-0:
> > lago.ssh: DEBUG: Running 9a90ca60 on lago-he-basic-suite-4-3-host-0: 
> > hosted-engine --set-maintenance --mode=none
> > lago.ssh: DEBUG: Command 9a90ca60 on lago-he-basic-suite-4-3-host-0 
> > returned with 1
> > lago.ssh: DEBUG: Command 9a90ca60 on lago-he-basic-suite-4-3-host-0  errors:
> >  Cannot connect to the HA daemon, please check the logs.
> >
> > ovirtlago.testlib: ERROR: * Unhandled exception in  
> > at 0x7f52673872a8>
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, 
> > in assert_equals_within
> > res = func()
> >   File 
> > "/home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-4.3/ovirt-system-tests/he-basic-suite-4.3/test-scenarios/008_restart_he_vm.py",
> >  line 87, in 
> > lambda: _set_and_test_maintenance_mode(host, False)
> >   File 
> > "/home/jenkins/agent/workspace/ovirt-system-tests_he-basic-suite-4.3/ovirt-system-tests/he-basic-suite-4.3/test-scenarios/008_restart_he_vm.py",
> >  line 108, in _set_and_test_maintenance_mode
> > nt.assert_equals(ret.code, 0)
> >   File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual
> > assertion_func(first, second, msg=msg)
> >   File "/usr/lib64/python2.7/unittest/case.py", line 546, in 
> > _baseAssertEqual
> > raise self.failureException(msg)
> > AssertionError: 1 != 0
> > - >> end captured logging << -
> >
> > Stack Trace:
> >   File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
> > testMethod()
> >   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
> > self.test(*self.arg)
> >   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, 
> > in wrapped_test
> > test()
> >   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", 

Re: type object 'LinuxBridge' has no attribute 'STP' (was: [CQ]: 66276a7 (ovirt-ansible-hosted-engine-setup) failed "ovirt-master" system tests)

[Nir Soffer]

On Wed, Jan 22, 2020 at 3:38 PM Michal Skrivanek 
wrote:

>
>
> On 22 Jan 2020, at 14:31, Nir Soffer  wrote:
>
> On Wed, Jan 22, 2020 at 1:41 PM Michal Skrivanek 
> wrote:
>
>>
>>
>> On 22 Jan 2020, at 11:46, Yedidyah Bar David  wrote:
>>
>> On Wed, Jan 22, 2020 at 12:34 PM Nir Soffer  wrote:
>>
>>
>> On Wed, Jan 22, 2020 at 12:25 PM Yedidyah Bar David 
>> wrote:
>>
>>
>> On Wed, Jan 22, 2020 at 12:10 PM Nir Soffer  wrote:
>>
>>
>>
>>
>> On Wed, Jan 22, 2020, 11:59 Yedidyah Bar David  wrote:
>>
>>
>> OK, I think I understand. STP is probably only in nmstate-0.2, and a
>> vdsm that requires this still didn't pass CQ.
>>
>>
>> master “tested” has vdsm witht he correct nmstate requirements.
>> But OST had issue picking it up properly, that was fixed in basic suite
>> yesterday evening by martin perina
>> errors below are older than that
>>
>>
>>
>> Same error happen when adding Fedora 30 host with master.
>>
>>
>> Adding a host broken now on Fedora. Netwok team need to revert the change
>> causing this issue.
>>
>>
>> No. They fixed it, but the fix didn't pass CQ yet.
>>
>>
>>
>> Not related to change queue, engine/vdsm master are broken.
>>
>> Did you try to add Fedora host with current engine and vdsm?
>>
>>
>> No, but the fix seems to deliberately not support Fedora, see also the
>> review comments:
>>
>> https://gerrit.ovirt.org/106413
>>
>>
>> that’s correct. Fro F30 you have to disable nmstate support and use the
>> fallback to ifcfg - set net_nmstate_enabled to false
>>
>
> This should be done by vdsm network automatically.
>
> We use Fedora 30 for development and we vdsm must continue to work on this
> distro until we
> move to Fedora 31.
>
>
> It seems it would be better to move to F31.
>

This requires effort to move the tests to containers. We have new
infrastructure but someone have to
spend time on this.


> But anyway, for now do you still have a host deployment problem when you
> disable nmstate on F30?
>

Yes, yesterday it failed in the same way with nmstate enabled or disabled,
but I will not have time to
investigate this before fosdem.


>
>
>
>>
>> Last CQ run that finished [1], failed for a different reason. vdsm.log
>> has [2]:
>>
>> libvirt.libvirtError: the CPU is incompatible with host CPU: Host CPU
>> does not provide required features: hle, rtm
>>
>> Adding Michal and Martin. Not sure if that's a known issue/already
>> fixed somewhere.
>>
>> [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18225/
>> [2]
>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18225/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.log
>>
>>
>>
>>
>> On Wed, Jan 22, 2020 at 11:33 AM Yedidyah Bar David 
>> wrote:
>>
>>
>> Resending and adding devel.
>>
>> This now happened to me again. I suspect this affects other runs. Any
>> clue?
>>
>>
>> https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7692/
>>
>> On Tue, Jan 21, 2020 at 12:34 PM Yedidyah Bar David 
>> wrote:
>>
>>
>> On Tue, Jan 21, 2020 at 1:18 AM oVirt Jenkins  wrote:
>>
>>
>> Change 66276a7 (ovirt-ansible-hosted-engine-setup) is probably the reason
>> behind recent system test failures in the "ovirt-master" change queue and
>> needs
>> to be fixed.
>>
>> This change had been removed from the testing queue. Artifacts build from
>> this
>> change will not be released until it is fixed.
>>
>> For further details about the change see:
>>
>> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/66276a7b4427014af5ecfb00138740ec8fbbfa4b
>>
>>
>> Above change is unrelated to the failure below. How can I make CQ look
>> at it again?
>>
>>
>> For failed test results see:
>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/
>>
>>
>> This failed in basic suite, in 002_bootstrap.verify_add_hosts.
>>
>> engine.log [1] has (e.g.):
>>
>> 2020-01-20 17:56:33,198-05 ERROR
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (EE-ManagedThreadFactory-engine-Thread-1) [66755eba] EVENT_ID:
>> VDS_BROKER_COMMAND_FAILURE(10,802), VDSM
>> lago-basic-suite-master-host-0 command Hos

Re: type object 'LinuxBridge' has no attribute 'STP' (was: [CQ]: 66276a7 (ovirt-ansible-hosted-engine-setup) failed "ovirt-master" system tests)

[Nir Soffer]

On Wed, Jan 22, 2020 at 1:41 PM Michal Skrivanek 
wrote:

>
>
> On 22 Jan 2020, at 11:46, Yedidyah Bar David  wrote:
>
> On Wed, Jan 22, 2020 at 12:34 PM Nir Soffer  wrote:
>
>
> On Wed, Jan 22, 2020 at 12:25 PM Yedidyah Bar David 
> wrote:
>
>
> On Wed, Jan 22, 2020 at 12:10 PM Nir Soffer  wrote:
>
>
>
>
> On Wed, Jan 22, 2020, 11:59 Yedidyah Bar David  wrote:
>
>
> OK, I think I understand. STP is probably only in nmstate-0.2, and a
> vdsm that requires this still didn't pass CQ.
>
>
> master “tested” has vdsm witht he correct nmstate requirements.
> But OST had issue picking it up properly, that was fixed in basic suite
> yesterday evening by martin perina
> errors below are older than that
>
>
>
> Same error happen when adding Fedora 30 host with master.
>
>
> Adding a host broken now on Fedora. Netwok team need to revert the change
> causing this issue.
>
>
> No. They fixed it, but the fix didn't pass CQ yet.
>
>
>
> Not related to change queue, engine/vdsm master are broken.
>
> Did you try to add Fedora host with current engine and vdsm?
>
>
> No, but the fix seems to deliberately not support Fedora, see also the
> review comments:
>
> https://gerrit.ovirt.org/106413
>
>
> that’s correct. Fro F30 you have to disable nmstate support and use the
> fallback to ifcfg - set net_nmstate_enabled to false
>

This should be done by vdsm network automatically.

We use Fedora 30 for development and we vdsm must continue to work on this
distro until we
move to Fedora 31.


>
> Last CQ run that finished [1], failed for a different reason. vdsm.log has
> [2]:
>
> libvirt.libvirtError: the CPU is incompatible with host CPU: Host CPU
> does not provide required features: hle, rtm
>
> Adding Michal and Martin. Not sure if that's a known issue/already
> fixed somewhere.
>
> [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18225/
> [2]
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18225/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.log
>
>
>
>
> On Wed, Jan 22, 2020 at 11:33 AM Yedidyah Bar David 
> wrote:
>
>
> Resending and adding devel.
>
> This now happened to me again. I suspect this affects other runs. Any clue?
>
> https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7692/
>
> On Tue, Jan 21, 2020 at 12:34 PM Yedidyah Bar David 
> wrote:
>
>
> On Tue, Jan 21, 2020 at 1:18 AM oVirt Jenkins  wrote:
>
>
> Change 66276a7 (ovirt-ansible-hosted-engine-setup) is probably the reason
> behind recent system test failures in the "ovirt-master" change queue and
> needs
> to be fixed.
>
> This change had been removed from the testing queue. Artifacts build from
> this
> change will not be released until it is fixed.
>
> For further details about the change see:
>
> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/66276a7b4427014af5ecfb00138740ec8fbbfa4b
>
>
> Above change is unrelated to the failure below. How can I make CQ look
> at it again?
>
>
> For failed test results see:
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/
>
>
> This failed in basic suite, in 002_bootstrap.verify_add_hosts.
>
> engine.log [1] has (e.g.):
>
> 2020-01-20 17:56:33,198-05 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engine-Thread-1) [66755eba] EVENT_ID:
> VDS_BROKER_COMMAND_FAILURE(10,802), VDSM
> lago-basic-suite-master-host-0 command HostSetupNetworksVDS failed:
> Internal JSON-RPC error: {'reason': "type object 'LinuxBridge' has no
> attribute 'STP'"}
>
> supervdsm log [2] has:
>
> MainProcess|jsonrpc/4::INFO::2020-01-20
> 17:56:32,430::configurator::190::root::(_setup_nmstate) Processing
> setup through nmstate
> MainProcess|jsonrpc/4::ERROR::2020-01-20
> 17:56:32,695::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper)
> Error in setupNetworks
> Traceback (most recent call last):
>  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py",
> line 95, in wrapper
>res = func(*args, **kwargs)
>  File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line
> 240, in setupNetworks
>_setup_networks(networks, bondings, options, net_info)
>  File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line
> 265, in _setup_networks
>networks, bondings, options, net_info, in_rollback
>  File
> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
> line 15

Re: type object 'LinuxBridge' has no attribute 'STP' (was: [CQ]: 66276a7 (ovirt-ansible-hosted-engine-setup) failed "ovirt-master" system tests)

[Nir Soffer]

On Wed, Jan 22, 2020 at 12:25 PM Yedidyah Bar David  wrote:

> On Wed, Jan 22, 2020 at 12:10 PM Nir Soffer  wrote:
> >
> >
> >
> > On Wed, Jan 22, 2020, 11:59 Yedidyah Bar David  wrote:
> >>
> >> OK, I think I understand. STP is probably only in nmstate-0.2, and a
> >> vdsm that requires this still didn't pass CQ.
> >
> >
> > Same error happen when adding Fedora 30 host with master.
> >
> > Adding a host broken now on Fedora. Netwok team need to revert the
> change causing this issue.
>
> No. They fixed it, but the fix didn't pass CQ yet.
>

Not related to change queue, engine/vdsm master are broken.

Did you try to add Fedora host with current engine and vdsm?

Last CQ run that finished [1], failed for a different reason. vdsm.log has
> [2]:
>
> libvirt.libvirtError: the CPU is incompatible with host CPU: Host CPU
> does not provide required features: hle, rtm
>
> Adding Michal and Martin. Not sure if that's a known issue/already
> fixed somewhere.
>
> [1] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18225/
> [2]
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18225/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.log
>
> >
> >
> >>
> >> On Wed, Jan 22, 2020 at 11:33 AM Yedidyah Bar David 
> wrote:
> >> >
> >> > Resending and adding devel.
> >> >
> >> > This now happened to me again. I suspect this affects other runs. Any
> clue?
> >> >
> >> >
> https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7692/
> >> >
> >> > On Tue, Jan 21, 2020 at 12:34 PM Yedidyah Bar David 
> wrote:
> >> > >
> >> > > On Tue, Jan 21, 2020 at 1:18 AM oVirt Jenkins 
> wrote:
> >> > > >
> >> > > > Change 66276a7 (ovirt-ansible-hosted-engine-setup) is probably
> the reason
> >> > > > behind recent system test failures in the "ovirt-master" change
> queue and needs
> >> > > > to be fixed.
> >> > > >
> >> > > > This change had been removed from the testing queue. Artifacts
> build from this
> >> > > > change will not be released until it is fixed.
> >> > > >
> >> > > > For further details about the change see:
> >> > > >
> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/66276a7b4427014af5ecfb00138740ec8fbbfa4b
> >> > >
> >> > > Above change is unrelated to the failure below. How can I make CQ
> look
> >> > > at it again?
> >> > >
> >> > > >
> >> > > > For failed test results see:
> >> > > >
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/
> >> > >
> >> > > This failed in basic suite, in 002_bootstrap.verify_add_hosts.
> >> > >
> >> > > engine.log [1] has (e.g.):
> >> > >
> >> > > 2020-01-20 17:56:33,198-05 ERROR
> >> > >
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >> > > (EE-ManagedThreadFactory-engine-Thread-1) [66755eba] EVENT_ID:
> >> > > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM
> >> > > lago-basic-suite-master-host-0 command HostSetupNetworksVDS failed:
> >> > > Internal JSON-RPC error: {'reason': "type object 'LinuxBridge' has
> no
> >> > > attribute 'STP'"}
> >> > >
> >> > > supervdsm log [2] has:
> >> > >
> >> > > MainProcess|jsonrpc/4::INFO::2020-01-20
> >> > > 17:56:32,430::configurator::190::root::(_setup_nmstate) Processing
> >> > > setup through nmstate
> >> > > MainProcess|jsonrpc/4::ERROR::2020-01-20
> >> > >
> 17:56:32,695::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper)
> >> > > Error in setupNetworks
> >> > > Traceback (most recent call last):
> >> > >   File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py",
> >> > > line 95, in wrapper
> >> > > res = func(*args, **kwargs)
> >> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line
> >> > > 240, in setupNetworks
> >> > > _setup_networks(networks, bondings, options, net_info)
> >> > >   File "/usr/lib/python3.6/site-packag

Re: type object 'LinuxBridge' has no attribute 'STP' (was: [CQ]: 66276a7 (ovirt-ansible-hosted-engine-setup) failed "ovirt-master" system tests)

[Nir Soffer]

On Wed, Jan 22, 2020, 11:59 Yedidyah Bar David  wrote:

> OK, I think I understand. STP is probably only in nmstate-0.2, and a
> vdsm that requires this still didn't pass CQ.
>

Same error happen when adding Fedora 30 host with master.

Adding a host broken now on Fedora. Netwok team need to revert the change
causing this issue.



> On Wed, Jan 22, 2020 at 11:33 AM Yedidyah Bar David 
> wrote:
> >
> > Resending and adding devel.
> >
> > This now happened to me again. I suspect this affects other runs. Any
> clue?
> >
> >
> https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/7692/
> >
> > On Tue, Jan 21, 2020 at 12:34 PM Yedidyah Bar David 
> wrote:
> > >
> > > On Tue, Jan 21, 2020 at 1:18 AM oVirt Jenkins 
> wrote:
> > > >
> > > > Change 66276a7 (ovirt-ansible-hosted-engine-setup) is probably the
> reason
> > > > behind recent system test failures in the "ovirt-master" change
> queue and needs
> > > > to be fixed.
> > > >
> > > > This change had been removed from the testing queue. Artifacts build
> from this
> > > > change will not be released until it is fixed.
> > > >
> > > > For further details about the change see:
> > > >
> https://github.com/oVirt/ovirt-ansible-hosted-engine-setup/commit/66276a7b4427014af5ecfb00138740ec8fbbfa4b
> > >
> > > Above change is unrelated to the failure below. How can I make CQ look
> > > at it again?
> > >
> > > >
> > > > For failed test results see:
> > > >
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/
> > >
> > > This failed in basic suite, in 002_bootstrap.verify_add_hosts.
> > >
> > > engine.log [1] has (e.g.):
> > >
> > > 2020-01-20 17:56:33,198-05 ERROR
> > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> > > (EE-ManagedThreadFactory-engine-Thread-1) [66755eba] EVENT_ID:
> > > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM
> > > lago-basic-suite-master-host-0 command HostSetupNetworksVDS failed:
> > > Internal JSON-RPC error: {'reason': "type object 'LinuxBridge' has no
> > > attribute 'STP'"}
> > >
> > > supervdsm log [2] has:
> > >
> > > MainProcess|jsonrpc/4::INFO::2020-01-20
> > > 17:56:32,430::configurator::190::root::(_setup_nmstate) Processing
> > > setup through nmstate
> > > MainProcess|jsonrpc/4::ERROR::2020-01-20
> > > 17:56:32,695::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper)
> > > Error in setupNetworks
> > > Traceback (most recent call last):
> > >   File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py",
> > > line 95, in wrapper
> > > res = func(*args, **kwargs)
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line
> > > 240, in setupNetworks
> > > _setup_networks(networks, bondings, options, net_info)
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line
> > > 265, in _setup_networks
> > > networks, bondings, options, net_info, in_rollback
> > >   File
> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
> > > line 154, in setup
> > > _setup_nmstate(networks, bondings, options, in_rollback, net_info)
> > >   File
> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
> > > line 195, in _setup_nmstate
> > > desired_state = nmstate.generate_state(networks, bondings)
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py",
> > > line 73, in generate_state
> > > networks, rconfig.networks, current_ifaces_state
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py",
> > > line 603, in generate_state
> > > for netname, netattrs in six.viewitems(networks)
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py",
> > > line 603, in 
> > > for netname, netattrs in six.viewitems(networks)
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py",
> > > line 339, in __init__
> > > self._create_interfaces_state()
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py",
> > > line 430, in _create_interfaces_state
> > > sb_iface, vlan_iface, bridge_iface = self._create_ifaces()
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py",
> > > line 444, in _create_ifaces
> > > options=self._create_bridge_options(),
> > >   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py",
> > > line 492, in _create_bridge_options
> > > LinuxBridge.STP.ENABLED: self._netconf.stp
> > > AttributeError: type object 'LinuxBridge' has no attribute 'STP'
> > >
> > > Perhaps that's related to recent changes adding/updating nmstate?
> > >
> > > [1]
> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/18189/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/engine.log
> > > [2]
> 

[JIRA] (OVIRT-2832) Add Fedora 31 support in the CI

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39974#comment-39974
 ] 

Nir Soffer commented on OVIRT-2832:
---

On Thu, Nov 14, 2019 at 9:24 AM Barak Korren (oVirt JIRA)
 wrote:
>
>
> [ 
> https://ovirt-jira.atlassian.net/browse/OVIRT-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39971#comment-39971
>  ]
>
> Barak Korren commented on OVIRT-2832:
> -
>
> While we could make mock work, it'd be nicer if we could get help to move the 
> new container-based backend forward so the upstream container image could be 
> used directly instead.

Will be even better, thanks!

> > Add Fedora 31 support in the CI
> > ---
> >
> > Key: OVIRT-2832
> > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2832
> > Project: oVirt - virtualization made easy
> >  Issue Type: By-EMAIL
> >Reporter: Nir Soffer
> >Assignee: infra
> >
> > Vdsm want to support only Fedora 31 at this point.
> > Having Fedora 31 available, we can simplify the code since we have
> > same versions of lvm and other packages in Fedora 31 and RHEL 8.
> > We need:
> > - mirrors for fedora 31 repos
> > - mock config for fedora 31
> > Thanks,
> > Nir
>
>
>
> --
> This message was sent by Atlassian Jira
> (v1001.0.0-SNAPSHOT#100114)
>

> Add Fedora 31 support in the CI
> ---
>
> Key: OVIRT-2832
>         URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2832
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> Vdsm want to support only Fedora 31 at this point.
> Having Fedora 31 available, we can simplify the code since we have
> same versions of lvm and other packages in Fedora 31 and RHEL 8.
> We need:
> - mirrors for fedora 31 repos
> - mock config for fedora 31
> Thanks,
> Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100114)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/LERIJRDVL4IPUO7V7W4RAG5PC6B54YSO/

Re: [JIRA] (OVIRT-2832) Add Fedora 31 support in the CI

[Nir Soffer]

On Thu, Nov 14, 2019 at 9:24 AM Barak Korren (oVirt JIRA)
 wrote:
>
>
> [ 
> https://ovirt-jira.atlassian.net/browse/OVIRT-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39971#comment-39971
>  ]
>
> Barak Korren commented on OVIRT-2832:
> -
>
> While we could make mock work, it'd be nicer if we could get help to move the 
> new container-based backend forward so the upstream container image could be 
> used directly instead.

Will be even better, thanks!

> > Add Fedora 31 support in the CI
> > ---
> >
> > Key: OVIRT-2832
> > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2832
> > Project: oVirt - virtualization made easy
> >  Issue Type: By-EMAIL
> >Reporter: Nir Soffer
> >Assignee: infra
> >
> > Vdsm want to support only Fedora 31 at this point.
> > Having Fedora 31 available, we can simplify the code since we have
> > same versions of lvm and other packages in Fedora 31 and RHEL 8.
> > We need:
> > - mirrors for fedora 31 repos
> > - mock config for fedora 31
> > Thanks,
> > Nir
>
>
>
> --
> This message was sent by Atlassian Jira
> (v1001.0.0-SNAPSHOT#100114)
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/J3KQ7QF55B3INUA6KMPPH36STRXBEQUS/
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/5TVC4FJDK4U3NLCCX3JGHFPKRIKPMQ6E/

[JIRA] (OVIRT-2827) Building vdsm with standard-manual-runner fails to clone the repo

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39970#comment-39970
 ] 

Nir Soffer commented on OVIRT-2827:
---

On Tue, Nov 12, 2019 at 5:48 PM Nir Soffer (oVirt JIRA)
 wrote:
>
> Nir Soffer created OVIRT-2827:
> -
>
>  Summary: Building vdsm with standard-manual-runner fails to 
> clone the repo
>  Key: OVIRT-2827
>  URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2827
>  Project: oVirt - virtualization made easy
>   Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: infra
>
>
> Building vdsm using standard-manual-runner fails now with:
>
> ERROR: Error cloning remote repo 'origin'
>
> hudson.plugins.git.GitException: Command "git fetch --tags --progress
> https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*"
> returned status code 128:
> stdout:
> stderr: error: RPC failed; result=22, HTTP code = 503
> fatal: The remote end hung up unexpectedly
>
> https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/830/pipeline
> https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/829/pipeline
>
> Can someone look into this?

Few hours later running the same builds worked, so this must have been
a temporary
issue.

Would be nice to understand what was the issue if possible.



> Building vdsm with standard-manual-runner fails to clone the repo
> -
>
> Key: OVIRT-2827
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2827
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> Building vdsm using standard-manual-runner fails now with:
> ERROR: Error cloning remote repo 'origin'
> hudson.plugins.git.GitException: Command "git fetch --tags --progress
> https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*"
> returned status code 128:
> stdout:
> stderr: error: RPC failed; result=22, HTTP code = 503
> fatal: The remote end hung up unexpectedly
> https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/830/pipeline
> https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/829/pipeline
> Can someone look into this?



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100114)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/MZIZG4UYKXHNUFST3S2P3KQZGDMGMKZF/

Re: [JIRA] (OVIRT-2827) Building vdsm with standard-manual-runner fails to clone the repo

[Nir Soffer]

On Tue, Nov 12, 2019 at 5:48 PM Nir Soffer (oVirt JIRA)
 wrote:
>
> Nir Soffer created OVIRT-2827:
> -
>
>  Summary: Building vdsm with standard-manual-runner fails to 
> clone the repo
>  Key: OVIRT-2827
>  URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2827
>  Project: oVirt - virtualization made easy
>   Issue Type: By-EMAIL
> Reporter: Nir Soffer
> Assignee: infra
>
>
> Building vdsm using standard-manual-runner fails now with:
>
> ERROR: Error cloning remote repo 'origin'
>
> hudson.plugins.git.GitException: Command "git fetch --tags --progress
> https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*"
> returned status code 128:
> stdout:
> stderr: error: RPC failed; result=22, HTTP code = 503
> fatal: The remote end hung up unexpectedly
>
> https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/830/pipeline
> https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/829/pipeline
>
> Can someone look into this?

Few hours later running the same builds worked, so this must have been
a temporary
issue.

Would be nice to understand what was the issue if possible.
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/D7ULGX7LR3P5UGFXGSQREMGHTZ36RJIX/

[JIRA] (OVIRT-2832) Add Fedora 31 support in the CI

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2832:
-

 Summary: Add Fedora 31 support in the CI
 Key: OVIRT-2832
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2832
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Vdsm want to support only Fedora 31 at this point.

Having Fedora 31 available, we can simplify the code since we have
same versions of lvm and other packages in Fedora 31 and RHEL 8.

We need:
- mirrors for fedora 31 repos
- mock config for fedora 31

Thanks,
Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100114)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/EWBPX2XM5UJLVD7B2HWUZ5PTJG7G2RNG/

[JIRA] (OVIRT-2828) Re: [ovirt-devel] Re: Check patch failure in vdsm

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39955#comment-39955
 ] 

Nir Soffer commented on OVIRT-2828:
---

On Tue, Nov 12, 2019 at 7:30 PM Nir Soffer  wrote:
>
> On Tue, Nov 12, 2019 at 5:34 PM Miguel Duarte de Mora Barroso
>  wrote:
> >
> > On Mon, Nov 11, 2019 at 11:12 AM Eyal Shenitzky  wrote:
> > >
> > > Hi,
> > >
> > > I encounter the following error for py36 test in vdsm check patch:
> > >
> > > ...
> > > ...
> > > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", 
> > > line 187, in _multicall
> > > 14:03:37 res = hook_impl.function(*args)
> > > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", 
> > > line 86, in 
> > > 14:03:37 firstresult=hook.spec.opts.get("firstresult") if hook.spec else 
> > > False,
> > > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", 
> > > line 92, in _hookexec
> > > 14:03:37 return self._inner_hookexec(hook, methods, kwargs)
> > > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/hooks.py", 
> > > line 286, in __call__
> > > 14:03:37 return self._hookexec(self, self.get_hookimpls(), kwargs)
> > > 14:03:37 File 
> > > "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/storage-py37/lib/python3.7/site-packages/_pytest/config/__init__.py",
> > >  line 82, in main
> > > 14:03:37 return config.hook.pytest_cmdline_main(config=config)
> > > 14:03:37 File 
> > > "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/storage-py37/bin/pytest",
> > >  line 8, in 
> > > 14:03:37 sys.exit(main())
> > > 14:03:37 [Inferior 1 (process 22145) detached]
> > > 14:03:37 =
> > > 14:03:37 = Terminating watched process =
> > > 14:03:37 =
> > > 14:03:37 PROFILE {"command": ["python", "py-watch", "600", "pytest", 
> > > "-m", "not (slow or stress)", "--durations=10", "--cov=vdsm.storage", 
> > > "--cov-report=html:htmlcov-storage-py37", "--cov-fail-under=62", 
> > > "storage"], "cpu": 39.921942808919184, "elapsed": 604.4699757099152, 
> > > "idrss": 0, "inblock": 1693453, "isrss": 0, "ixrss": 0, "majflt": 2, 
> > > "maxrss": 331172, "minflt": 5606489, "msgrcv": 0, "msgsnd": 0, "name": 
> > > "storage-py37", "nivcsw": 139819, "nsignals": 0, "nswap": 0, "nvcsw": 
> > > 187576, "oublock": 2495645, "start": 1573386812.7961884, "status": 143, 
> > > "stime": 118.260961, "utime": 123.055197}
> > > 14:03:37 ERROR: InvocationError for command 
> > > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/storage-py37/bin/python
> > >  profile storage-py37 python py-watch 600 pytest -m 'not (slow or 
> > > stress)' --durations=10 --cov=vdsm.storage 
> > > --cov-report=html:htmlcov-storage-py37 --cov-fail-under=62 storage 
> > > (exited with code 143)
> > >
> > >
> > > Is there any known issue?
> >
> > Anyone able to pitch in ? I think something similar is happening in
> > [0], also on check-patch [1].
> >
> > [0] - 
> > https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/14234/
>
> Yes it looks the same issue.
>
> > [1] - https://gerrit.ovirt.org/#/c/104274/
>
> Jenkins slaves are very slow recently. I suspect we run too many jobs
> concurrently or using
> too many virtual cpus.

I hope this will avoid the random failures:
https://gerrit.ovirt.org/c/104629/

> Re: [ovirt-devel] Re: Check patch failure in vdsm
> -
>
> Key: OVIRT-2828
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2828
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> On Tue, Nov 12, 2019 at 5:34 PM Miguel Duarte de Mora Barroso
>  wrote:
> >
> > On Mon, Nov 11, 2019 at 11:12 AM E

[JIRA] (OVIRT-2828) Re: [ovirt-devel] Re: Check patch failure in vdsm

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2828:
-

 Summary: Re: [ovirt-devel] Re: Check patch failure in vdsm
 Key: OVIRT-2828
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2828
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


On Tue, Nov 12, 2019 at 5:34 PM Miguel Duarte de Mora Barroso
 wrote:
>
> On Mon, Nov 11, 2019 at 11:12 AM Eyal Shenitzky  wrote:
> >
> > Hi,
> >
> > I encounter the following error for py36 test in vdsm check patch:
> >
> > ...
> > ...
> > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", 
> > line 187, in _multicall
> > 14:03:37 res = hook_impl.function(*args)
> > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", 
> > line 86, in 
> > 14:03:37 firstresult=hook.spec.opts.get("firstresult") if hook.spec else 
> > False,
> > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", 
> > line 92, in _hookexec
> > 14:03:37 return self._inner_hookexec(hook, methods, kwargs)
> > 14:03:37 File "/usr/local/lib/python3.7/site-packages/pluggy/hooks.py", 
> > line 286, in __call__
> > 14:03:37 return self._hookexec(self, self.get_hookimpls(), kwargs)
> > 14:03:37 File 
> > "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/storage-py37/lib/python3.7/site-packages/_pytest/config/__init__.py",
> >  line 82, in main
> > 14:03:37 return config.hook.pytest_cmdline_main(config=config)
> > 14:03:37 File 
> > "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/storage-py37/bin/pytest",
> >  line 8, in 
> > 14:03:37 sys.exit(main())
> > 14:03:37 [Inferior 1 (process 22145) detached]
> > 14:03:37 =
> > 14:03:37 = Terminating watched process =
> > 14:03:37 =
> > 14:03:37 PROFILE {"command": ["python", "py-watch", "600", "pytest", "-m", 
> > "not (slow or stress)", "--durations=10", "--cov=vdsm.storage", 
> > "--cov-report=html:htmlcov-storage-py37", "--cov-fail-under=62", 
> > "storage"], "cpu": 39.921942808919184, "elapsed": 604.4699757099152, 
> > "idrss": 0, "inblock": 1693453, "isrss": 0, "ixrss": 0, "majflt": 2, 
> > "maxrss": 331172, "minflt": 5606489, "msgrcv": 0, "msgsnd": 0, "name": 
> > "storage-py37", "nivcsw": 139819, "nsignals": 0, "nswap": 0, "nvcsw": 
> > 187576, "oublock": 2495645, "start": 1573386812.7961884, "status": 143, 
> > "stime": 118.260961, "utime": 123.055197}
> > 14:03:37 ERROR: InvocationError for command 
> > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/storage-py37/bin/python
> >  profile storage-py37 python py-watch 600 pytest -m 'not (slow or stress)' 
> > --durations=10 --cov=vdsm.storage --cov-report=html:htmlcov-storage-py37 
> > --cov-fail-under=62 storage (exited with code 143)
> >
> >
> > Is there any known issue?
>
> Anyone able to pitch in ? I think something similar is happening in
> [0], also on check-patch [1].
>
> [0] - 
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/14234/

Yes it looks the same issue.

> [1] - https://gerrit.ovirt.org/#/c/104274/

Jenkins slaves are very slow recently. I suspect we run too many jobs
concurrently or using
too many virtual cpus.



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100114)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/PT52ZBK7VHVJPSFHJ7YSORYPPESI5DPS/

[JIRA] (OVIRT-2827) Building vdsm with standard-manual-runner fails to clone the repo

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2827:
-

 Summary: Building vdsm with standard-manual-runner fails to clone 
the repo
 Key: OVIRT-2827
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2827
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Building vdsm using standard-manual-runner fails now with:

ERROR: Error cloning remote repo 'origin'

hudson.plugins.git.GitException: Command "git fetch --tags --progress
https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*"
returned status code 128:
stdout:
stderr: error: RPC failed; result=22, HTTP code = 503
fatal: The remote end hung up unexpectedly

https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/830/pipeline
https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner/detail/standard-manual-runner/829/pipeline

Can someone look into this?



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100114)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/6QYQGT4WK6OFMAGZNO7KQ6MWNUA7RYFK/

[JIRA] (OVIRT-2813) EL8 builds fail to mount loop device - kernel too old?

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2813:
-

 Summary: EL8 builds fail to mount loop device - kernel too old?
 Key: OVIRT-2813
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2813
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


I'm trying to run imageio tests on el8 mock, and the tests fail early when
trying to create storage
for testing:

[userstorage] INFOCreating filesystem
/var/tmp/imageio-storage/file-512-ext4-mount
Suggestion: Use Linux kernel >= 3.18 for improved stability of the
metadata and journal checksum features.
[userstorage] INFOCreating file
/var/tmp/imageio-storage/file-512-ext4-mount/file
[userstorage] INFOCreating backing file
/var/tmp/imageio-storage/file-512-xfs-backing
[userstorage] INFOCreating loop device
/var/tmp/imageio-storage/file-512-xfs-loop
[userstorage] INFOCreating filesystem
/var/tmp/imageio-storage/file-512-xfs-mount
mount: /var/tmp/imageio-storage/file-512-xfs-mount: wrong fs type, bad
option, bad superblock on /dev/loop4, missing codepage or helper
program, or other error.
Traceback (most recent call last):
  File "/usr/local/bin/userstorage", line 10, in 
sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py",
line 42, in main
create(cfg)
  File "/usr/local/lib/python3.6/site-packages/userstorage/__main__.py",
line 52, in create
b.create()
  File "/usr/local/lib/python3.6/site-packages/userstorage/file.py",
line 47, in create
self._mount.create()
  File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py",
line 53, in create
self._mount_loop()
  File "/usr/local/lib/python3.6/site-packages/userstorage/mount.py",
line 94, in _mount_loop
["sudo", "mount", "-t", self.fstype, self._loop.path, self.path])
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', 'mount', '-t', 'xfs',
'/var/tmp/imageio-storage/file-512-xfs-loop',
'/var/tmp/imageio-storage/file-512-xfs-mount']' returned non-zero exit
status 32.

https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/1593//artifact/check-patch.el8.ppc64le/mock_logs/script/stdout_stderr.log


Same code runs fine in Travis:
https://travis-ci.org/nirs/ovirt-imageio/jobs/595794863


And also locally on Fedora 29:
$ ../jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el8
...
## Wed Oct  9 23:37:22 IDT 2019 Finished env: el8:epel-8-x86_64
##  took 85 seconds
##  rc = 0


My guess is that we run el8 jobs on el7 hosts with old kernels
(Suggestion: Use Linux kernel >= 3.18 for improved stability of the
metadata and journal checksum features.)

This issue will affects vdsm, using similar code to create storage for
testing.

For vdsm the issue is more tricky, since it requires same host/distro:

 49 - tests-py37:
 50 runtime-requirements:
 51   host-distro: same

So I think we need:
- make slaves with newer kernel (fc29? fc30?) with the el8 mock env
- add el8 slaves for running el8 mock with "host-distro: same"

If we don't have a solution we need to disable el8 tests, or continue
testing without
storage, which will skip about 200 tests.

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100111)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/H3AOXLB6CSOZEXO7SGBG6RUSHGI32LJ2/

[JIRA] (OVIRT-2810) CI build error with: hudson.plugins.git.GitException: Failed to fetch from https://gerrit.ovirt.org/jenkins

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2810:
-

 Summary: CI build error with: hudson.plugins.git.GitException: 
Failed to fetch from https://gerrit.ovirt.org/jenkins
 Key: OVIRT-2810
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2810
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Any imageio patch fail now with the error bellow.


Examples:


*https://gerrit.ovirt.org/q/topic:py3+is:open+project:ovirt-imageio
<https://gerrit.ovirt.org/q/topic:py3+is:open+project:ovirt-imageio>*


*00:07:26*  ERROR: Error fetching remote repo 'origin'

*00:07:26*  hudson.plugins.git.GitException: Failed to fetch from
https://gerrit.ovirt.org/jenkins*00:07:26*  at
hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)*00:07:26*  at
hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)*00:07:26*
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)*00:07:26*
at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)*00:07:26*
at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)*00:07:26*
at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)*00:07:26*
at 
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)*00:07:26*
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)*00:07:26*
at java.util.concurrent.FutureTask.run(FutureTask.java:266)*00:07:26*
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*00:07:26*
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*00:07:26*
at java.lang.Thread.run(Thread.java:748)*00:07:26*  Caused by:
hudson.plugins.git.GitException: Command "git fetch --tags --progress
https://gerrit.ovirt.org/jenkins
+133b1a1729ce5c6caf1745ad8034ceaa4dd187c5:myhead" returned status code
128:*00:07:26*  stdout: *00:07:26*  stderr: error: no such remote ref
133b1a1729ce5c6caf1745ad8034ceaa4dd187c5*00:07:26*  *00:07:26*  at
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)*00:07:26*
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)*00:07:26*
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)*00:07:26*
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)*00:07:26*
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)*00:07:26*
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)*00:07:26*
at hudson.remoting.UserRequest.perform(UserRequest.java:212)*00:07:26*
at hudson.remoting.UserRequest.perform(UserRequest.java:54)*00:07:26*
at hudson.remoting.Request$2.run(Request.java:369)*00:07:26*at
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)*00:07:26*
... 4 more*00:07:26*Suppressed:
hudson.remoting.Channel$CallSiteStackTrace: Remote call to
vm0008.workers-phx.ovirt.org*00:07:26*  at
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)*00:07:26*
at 
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)*00:07:26*
at hudson.remoting.Channel.call(Channel.java:957)*00:07:26* 
at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)*00:07:26*
at sun.reflect.GeneratedMethodAccessor715.invoke(Unknown
Source)*00:07:26*   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*00:07:26*
at java.lang.reflect.Method.invoke(Method.java:498)*00:07:26*   
at
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)*00:07:26*
at com.sun.proxy.$Proxy83.execute(Unknown Source)*00:07:26* 
at
hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)*00:07:26*  
at
hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)*00:07:26*
at 
hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)*00:07:26*
at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)*00:07:26*
at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)*00:07:26*
at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.j

Re: [oVirt] [CQ weekly status] [04-10-2019]

[Nir Soffer]

On Fri, Oct 4, 2019 at 7:01 PM Dusan Fodor  wrote:

> Hi,
>
> This mail is to provide the current status of CQ and allow people to
> review status before and after the weekend.
> Please refer to below colour map for further information on the meaning of
> the colours.
>
> *CQ-4.3*:GREEN (#1)
> Last failure was on 03-10 caused by pipeline configuration, it was fixed
> soon after.
>
> *CQ-Master:*  RED (#1)
> Last failure was on 04-10 caused when vdsm failed building using bad
> interpreter. Patch 103655  is pending to
> fix this issue.
>

Merged, build should work now.


>
>
>
> Current running jobs for 4.3 [1] and master [2] can be found here:
>
> [1]
> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change-queue-tester/
>
> [2]
> http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change-queue-tester/
>
> Have a nice week!
> Dusan
>
>
>
> ---
> COLOUR MAP
>
> Green = job has been passing successfully
>
> ** green for more than 3 days may suggest we need a review of our test
> coverage
>
>
>1.
>
>1-3 days   GREEN (#1)
>2.
>
>4-7 days   GREEN (#2)
>3.
>
>Over 7 days GREEN (#3)
>
>
> Yellow = intermittent failures for different projects but no lasting or
> current regressions
>
> ** intermittent would be a healthy project as we expect a number of
> failures during the week
>
> ** I will not report any of the solved failures or regressions.
>
>
>1.
>
>Solved job failuresYELLOW (#1)
>2.
>
>Solved regressions  YELLOW (#2)
>
>
> Red = job has been failing
>
> ** Active Failures. The colour will change based on the amount of time the
> project/s has been broken. Only active regressions would be reported.
>
>
>1.
>
>1-3 days  RED (#1)
>2.
>
>4-7 days  RED (#2)
>3.
>
>Over 7 days RED (#3)
>
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/YCNCKRK3G4EJXA3OCYAUS4VMKRDA67F4/
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/OPZEMQSUCPRHGLKZOF5JDMG4JBYORYMH/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/EJFKNJ3P5AEF5PCOSGABWSUT7LZIMTUD/

[JIRA] (OVIRT-2808) CI broken - builds fail early in "loading code" stage with "attempted duplicate class definition for name: "Project""

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2808:
-

 Summary: CI broken - builds fail early in "loading code" stage 
with "attempted duplicate class definition for name: "Project""
 Key: OVIRT-2808
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2808
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
    Reporter: Nir Soffer
Assignee: infra


Example builds:

- http://jenkins.ovirt.org/job/vdsm_standard-check-patch/12526/

- http://jenkins.ovirt.org/job/vdsm_standard-check-patch/12527/


java.lang.LinkageError: loader (instance of
org/jenkinsci/plugins/workflow/cps/CpsGroovyShell$CleanGroovyClassLoader):
attempted  duplicate class definition for name: "Project"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at groovy.lang.GroovyClassLoader.access$400(GroovyClassLoader.java:62)
at 
groovy.lang.GroovyClassLoader$ClassCollector.createClass(GroovyClassLoader.java:500)
at 
groovy.lang.GroovyClassLoader$ClassCollector.onClassNode(GroovyClassLoader.java:517)
at 
groovy.lang.GroovyClassLoader$ClassCollector.call(GroovyClassLoader.java:521)
at 
org.codehaus.groovy.control.CompilationUnit$17.call(CompilationUnit.java:834)
at 
org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
at 
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
at 
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
at 
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
at 
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
at 
org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
at 
org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.parse(CpsGroovyShell.java:113)
at groovy.lang.GroovyShell.parse(GroovyShell.java:736)
at groovy.lang.GroovyShell.parse(GroovyShell.java:727)
at 
org.jenkinsci.plugins.workflow.cps.steps.LoadStepExecution.start(LoadStepExecution.java:49)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:269)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:177)
at 
org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at 
org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at 
com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at WorkflowScript.load_code(WorkflowScript:45)
at Script4.on_load(Script4.groovy:22)
at WorkflowScript.load_code(WorkflowScript:47)
at Script1.on_load(Script1.groovy:13)
at WorkflowScript.load_code(WorkflowScript:47)
at WorkflowScript.run(WorkflowScript:22)
at ___cps.transform___(Native Method)
at 
com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:84)
at 
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at 
com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at sun.reflect.GeneratedMethodAccessor681.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at 
com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
at 
com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
at 
com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
at 
com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at 
org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCat

[JIRA] (OVIRT-2805) build-artifacts job stuck on s390x - cannot run OST

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39872#comment-39872
 ] 

Nir Soffer commented on OVIRT-2805:
---

On Sun, Sep 29, 2019 at 2:35 AM Nir Soffer  wrote:

> On Sun, Sep 29, 2019 at 1:13 AM Nir Soffer  wrote:
>
>> Since Friday build-artifacts on s390x get stuck again, so we cannot run
>> OST.
>> This is not a new issue, we have issues on s390x every few weeks.
>>
>> I posted this patch to disable this job:
>> https://gerrit.ovirt.org/c/97851
>>
>
> The patch does not work, it still runs the s390x job, and it run it
> incorrectly.
> Maybe the issue is not at the slave, but at vdsm automation scripts?
>
> We have this yam:
>
>  19 stages:
>  20   - build-artifacts:
>  21   substages:
>  22 - build-py27:
>  23 archs:
>  24   - ppc64le
>  25   - x86_64
>  26 - build-py37:
>  27 distributions:
>  28   - fc30
>
> And we get these jobs:
> -  build-artifacts.build-py27.el7.ppc64le
> - build-artifacts.build-py27.el7.x86_64
> - build-artifacts.build-py27.fc29.x86_64
> -  build-artifacts.build-py37.fc30.x86_64
> - build-artifacts.fc29.s390x
>
> The last job - s390x looks wrong - we should have only
> build-py27 and build-py37 jobs, using:
>
> - automation/build-artifacts.build-py27.sh
> - automation/build-artifacts.build-py37.sh
>
> But both scripts are using a symlinks:
> lrwxrwxrwx. 1 nsoffer nsoffer  18 Sep 29 00:55 automation/
> build-artifacts.build-py27.sh -> build-artifacts.sh
> lrwxrwxrwx. 1 nsoffer nsoffer  18 Sep 29 00:55 automation/
> build-artifacts.build-py37.sh -> build-artifacts.sh
> -rwxrwxr-x. 1 nsoffer nsoffer 346 Sep 17 02:54
> automation/build-artifacts.sh
>
> Is it possible the the CI find build-artifacts.sh and run it even when no
> sub stage is specified?
>
> I'll try to rename this script to avoid this.
>

Hopefully fixed by:
https://gerrit.ovirt.org/c/103655/



>
> We can enable the job when we have a reliable build slave again.
>>
>> Here are some failed jobs:
>> - http://jenkins.ovirt.org/job/standard-manual-runner/757/
>> - http://jenkins.ovirt.org/job/standard-manual-runner/758/
>> - http://jenkins.ovirt.org/job/standard-manual-runner/759/
>> - http://jenkins.ovirt.org/job/standard-manual-runner/762/
>> - http://jenkins.ovirt.org/job/standard-manual-runner/763/
>> - http://jenkins.ovirt.org/job/standard-manual-runner/764/
>>
>> Nir
>>
>

> build-artifacts job stuck on s390x - cannot run OST
> -----------
>
> Key: OVIRT-2805
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2805
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> Since Friday build-artifacts on s390x get stuck again, so we cannot run OST.
> This is not a new issue, we have issues on s390x every few weeks.
> I posted this patch to disable this job:
> https://gerrit.ovirt.org/c/97851
> We can enable the job when we have a reliable build slave again.
> Here are some failed jobs:
> - http://jenkins.ovirt.org/job/standard-manual-runner/757/
> - http://jenkins.ovirt.org/job/standard-manual-runner/758/
> - http://jenkins.ovirt.org/job/standard-manual-runner/759/
> - http://jenkins.ovirt.org/job/standard-manual-runner/762/
> - http://jenkins.ovirt.org/job/standard-manual-runner/763/
> - http://jenkins.ovirt.org/job/standard-manual-runner/764/
> Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100111)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/2QVNC3ADV4CNZLPLELGB6QUJRDJWDUIW/

[JIRA] (OVIRT-2805) build-artifacts job stuck on s390x - cannot run OST

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39871#comment-39871
 ] 

Nir Soffer commented on OVIRT-2805:
---

On Sun, Sep 29, 2019 at 1:13 AM Nir Soffer  wrote:

> Since Friday build-artifacts on s390x get stuck again, so we cannot run
> OST.
> This is not a new issue, we have issues on s390x every few weeks.
>
> I posted this patch to disable this job:
> https://gerrit.ovirt.org/c/97851
>

The patch does not work, it still runs the s390x job, and it run it
incorrectly.
Maybe the issue is not at the slave, but at vdsm automation scripts?

We have this yam:

 19 stages:
 20   - build-artifacts:
 21   substages:
 22 - build-py27:
 23 archs:
 24   - ppc64le
 25   - x86_64
 26 - build-py37:
 27 distributions:
 28   - fc30

And we get these jobs:
-  build-artifacts.build-py27.el7.ppc64le
- build-artifacts.build-py27.el7.x86_64
- build-artifacts.build-py27.fc29.x86_64
-  build-artifacts.build-py37.fc30.x86_64
- build-artifacts.fc29.s390x

The last job - s390x looks wrong - we should have only
build-py27 and build-py37 jobs, using:

- automation/build-artifacts.build-py27.sh
- automation/build-artifacts.build-py37.sh

But both scripts are using a symlinks:
lrwxrwxrwx. 1 nsoffer nsoffer  18 Sep 29 00:55 automation/
build-artifacts.build-py27.sh -> build-artifacts.sh
lrwxrwxrwx. 1 nsoffer nsoffer  18 Sep 29 00:55 automation/
build-artifacts.build-py37.sh -> build-artifacts.sh
-rwxrwxr-x. 1 nsoffer nsoffer 346 Sep 17 02:54 automation/build-artifacts.sh

Is it possible the the CI find build-artifacts.sh and run it even when no
sub stage is specified?

I'll try to rename this script to avoid this.

We can enable the job when we have a reliable build slave again.
>
> Here are some failed jobs:
> - http://jenkins.ovirt.org/job/standard-manual-runner/757/
> - http://jenkins.ovirt.org/job/standard-manual-runner/758/
> - http://jenkins.ovirt.org/job/standard-manual-runner/759/
> - http://jenkins.ovirt.org/job/standard-manual-runner/762/
> - http://jenkins.ovirt.org/job/standard-manual-runner/763/
> - http://jenkins.ovirt.org/job/standard-manual-runner/764/
>
> Nir
>

> build-artifacts job stuck on s390x - cannot run OST
> ---
>
> Key: OVIRT-2805
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2805
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> Since Friday build-artifacts on s390x get stuck again, so we cannot run OST.
> This is not a new issue, we have issues on s390x every few weeks.
> I posted this patch to disable this job:
> https://gerrit.ovirt.org/c/97851
> We can enable the job when we have a reliable build slave again.
> Here are some failed jobs:
> - http://jenkins.ovirt.org/job/standard-manual-runner/757/
> - http://jenkins.ovirt.org/job/standard-manual-runner/758/
> - http://jenkins.ovirt.org/job/standard-manual-runner/759/
> - http://jenkins.ovirt.org/job/standard-manual-runner/762/
> - http://jenkins.ovirt.org/job/standard-manual-runner/763/
> - http://jenkins.ovirt.org/job/standard-manual-runner/764/
> Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100111)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/4LYWCE65A6BRKNZ7E72VRDDPDMT2TF7R/

[JIRA] (OVIRT-2805) build-artifacts job stuck on s390x - cannot run OST

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2805:
-

 Summary: build-artifacts job stuck on s390x - cannot run OST
 Key: OVIRT-2805
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2805
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


Since Friday build-artifacts on s390x get stuck again, so we cannot run OST.
This is not a new issue, we have issues on s390x every few weeks.

I posted this patch to disable this job:
https://gerrit.ovirt.org/c/97851

We can enable the job when we have a reliable build slave again.

Here are some failed jobs:
- http://jenkins.ovirt.org/job/standard-manual-runner/757/
- http://jenkins.ovirt.org/job/standard-manual-runner/758/
- http://jenkins.ovirt.org/job/standard-manual-runner/759/
- http://jenkins.ovirt.org/job/standard-manual-runner/762/
- http://jenkins.ovirt.org/job/standard-manual-runner/763/
- http://jenkins.ovirt.org/job/standard-manual-runner/764/

Nir



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100111)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ADWQNFYBS4XSQGYIJQIND7S5JV5CPFAV/

Re: [CQ]: 103548,1 (vdsm) failed "ovirt-4.3" system tests

[Nir Soffer]

On Thu, Sep 26, 2019 at 3:55 PM oVirt Jenkins  wrote:

> Change 103548,1 (vdsm) is probably the reason behind recent system test
> failures in the "ovirt-4.3" change queue and needs to be fixed.
>
> This change had been removed from the testing queue. Artifacts build from
> this
> change will not be released until it is fixed.
>

This is a bug in the change queue, because

For further details about the change see:
> https://gerrit.ovirt.org/#/c/103548/1


this is a travis configuration change, and it cannot break system tests :-)

Do we have stats of such failures in the past?

For failed test results see:
> http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/2188/


Maybe we can add a simple file based condition to tell if a patch need to
run
system tests?

For example in vdsm we have this structure:
lib/
static/
tests/
automation/
.travis.ymal
...

Changes in tests/ and automation/ cannot break vdsm during runtime, so there
is no need to run system tests for patches modifying these directories.

Here are changes stats per directory since 4.2:

$ git shortlog -sn --since v4.20.0 ./tests ./automation | awk '{sum+=$1}
END {print sum}'
4579
$ git shortlog -sn --since v4.20.0 ./lib ./static | awk '{sum+=$1} END
{print sum}'
4413

So 50% of the changes cannot break the system and do not require system
tests.

SLOC Directory SLOC-by-Language (Sorted)
65205   lib python=64977,sh=228
62415   tests   python=62317,sh=98
3432vdsm_hooks  python=3432
803 contrib python=740,sh=63
468 automation  sh=454,python=14
313 static  python=210,sh=103
289 initsh=162,python=127
112 build-aux   sh=72,python=40
64  top_dir sh=64
30  doc python=30
30  helpers python=30
1   vdsm_logsh=1
0   docker  (none)
0   m4  (none)

We can see that the tests are about 50% of the code. This is pretty good
for legacy
project, but if we look in a more modern project like ovirt-imageio:

common:
SLOC Directory SLOC-by-Language (Sorted)
3641testpython=3641
2365ovirt_imageio_common python=2201,ansic=164
103 top_dir python=103

daemon:
SLOC Directory SLOC-by-Language (Sorted)
1588testpython=1588
792 ovirt_imageio_daemon python=792
18  top_dir python=18
0   data(none)

We see that test code ratio is close to 2/1.

So adding a filter per project that can tell if a patch needs system tests
very important,
and can speed our workflow by fact of 2-3.

Adding infra-support to file a bug, we need to work on this.

Nir
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/IMHBHVRPNCSFFBWWQBH6CDD5UOTLHOQ2/

Re: OST fails on 004_basic_sanity.verify_glance_import

[Nir Soffer]

On Thu, Sep 26, 2019 at 1:39 PM Eyal Edri  wrote:

>
>
> On Thu, Sep 26, 2019 at 1:22 PM Yedidyah Bar David 
> wrote:
>
>> Hi all,
>>
>> [1], which I ran for verifying [2], fails for me at
>> 004_basic_sanity.verify_glance_import . I can see several different
>> cases of failure there reported to devel@, last one 1.5 months ago. Is
>> this a (new) known issue?
>>
>
> Its not a known issue AFAIK, though it is dependent on the network to the
> glance instance,
> which sometimes can be unstable or slow, hence a test can fail.
> We've had a network outage this morning, it could be related to it.
> If you see the same test failure repeatedly, then we'll need to
> investigate further if something is wrong with the glance server.
>

Depending on external service in OST is big no no. How do you want to gate
patches
when a test can fail randomly because of environment?

We must run glance inside the closed OST environment. Same for yum repos,
we must use mirrors running in the closed environment.

Nir


>
>
>>
>> [1]
>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/5666/
>>
>> [2] https://gerrit.ovirt.org/97698
>>
>> Thanks and best regards,
>> --
>> Didi
>> ___
>> Infra mailing list -- infra@ovirt.org
>> To unsubscribe send an email to infra-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/OOP7647K4VLXGVSDWS4R7UERH5SMFQEA/
>>
>
>
> --
>
> Eyal edri
>
> He / Him / His
>
>
> MANAGER
>
> CONTINUOUS PRODUCTIZATION
>
> SYSTEM ENGINEERING
>
> Red Hat 
> 
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/YCZHTRJMCCGCTNUYNUXR3WYTO6I5HTKA/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/QRVN5LT6LFHDFQU3FGEFPPFDXYVVFIDM/

Re: OST fails on 004_basic_sanity.verify_glance_import

[Nir Soffer]

On Thu, Sep 26, 2019 at 1:22 PM Yedidyah Bar David  wrote:

> Hi all,
>
> [1], which I ran for verifying [2], fails for me at
> 004_basic_sanity.verify_glance_import . I can see several different
> cases of failure there reported to devel@, last one 1.5 months ago. Is
> this a (new) known issue?
>

In the past it was networking issues (accessing glance).

The error message does not contain any useful information:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/5666/testReport/junit/(root)/004_basic_sanity/verify_glance_import/

We run OST a lot recently and did not see this error, try to rebuild.

[1]
> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/5666/
>
> [2] https://gerrit.ovirt.org/97698
>
> Thanks and best regards,
> --
> Didi
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/OOP7647K4VLXGVSDWS4R7UERH5SMFQEA/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/RV6IT26XMJ2XRW2S6WU5AYMCDLP7VK5P/

[JIRA] (OVIRT-2803) Re: CI is not triggered for pushed gerrit updates

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2803:
-

 Summary: Re: CI is not triggered for pushed gerrit updates
 Key: OVIRT-2803
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2803
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


On Wed, Sep 25, 2019 at 12:42 PM Amit Bawer  wrote:

> CI has stopped from being triggered for pushed gerrit updates.
>

Adding infra-suppo...@ovirt.org - this file a bug in infra bug tracker.

Example at: https://gerrit.ovirt.org/#/c/103320/
> last PS did not trigger CI tests.
>

More examples:
https://gerrit.ovirt.org/c/103490/2
https://gerrit.ovirt.org/c/103455/4

Last triggered build was seen here:
https://gerrit.ovirt.org/c/103549/1
Sep 24 18:42.

"ci test" also did not trigger a build, but I tried now again using "ci
please test"
and it worked.
https://gerrit.ovirt.org/c/103490/2#message-8d40f6de_bd0564ff

Maybe someone changed the pattern?



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100111)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7DVT67CECHEFS4POEECT4GG7TPBLRHM6/

Re: CI is not triggered for pushed gerrit updates

[Nir Soffer]

On Wed, Sep 25, 2019 at 12:42 PM Amit Bawer  wrote:

> CI has stopped from being triggered for pushed gerrit updates.
>

Adding infra-suppo...@ovirt.org - this file a bug in infra bug tracker.

Example at: https://gerrit.ovirt.org/#/c/103320/
> last PS did not trigger CI tests.
>

More examples:
https://gerrit.ovirt.org/c/103490/2
https://gerrit.ovirt.org/c/103455/4

Last triggered build was seen here:
https://gerrit.ovirt.org/c/103549/1
Sep 24 18:42.

"ci test" also did not trigger a build, but I tried now again using "ci
please test"
and it worked.
https://gerrit.ovirt.org/c/103490/2#message-8d40f6de_bd0564ff

Maybe someone changed the pattern?
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/RFTM7SZDRP56SUGMO5J7Z6K5UYOMK5HY/

[JIRA] (OVIRT-2795) Re: [ovirt-devel] ovirt-vmconsole package issues

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2795:
-

 Summary: Re: [ovirt-devel] ovirt-vmconsole package issues
 Key: OVIRT-2795
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2795
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


On Wed, Sep 11, 2019 at 4:38 PM Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
>
> On 11 Sep 2019, at 11:19, Marcin Sobczyk  wrote:
>
> Hi,
>
> we get a lot of failures in the CI for el7 and fc29 regarding this package
> since this morning, i.e. for fc29:
>
>
> in which CI?
>

Here are some examples:
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11308/pipeline
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11307/pipeline/123

Adding infra-support to file a bug.


>
>
>  
> <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11231/pipeline/119#step-560-log-1150>[2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11231/pipeline/119#step-560-log-1151>[2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11231/pipeline/119#step-560-log-1152>[2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11231/pipeline/119#step-560-log-1153>[2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11231/pipeline/119#step-560-log-1154>[2019-09-11T07:55:42.107Z]
>  [FAILED] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: No more mirrors to try - 
> All mirrors were already tried without success
>  
> <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11231/pipeline/119#step-560-log-1155>[2019-09-11T07:55:42.107Z]
>  Error: Error downloading packages:
>  
> <https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11231/pipeline/119#step-560-log-1156>[2019-09-11T07:55:42.107Z]
>Cannot download noarch/ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: All 
> mirrors were tried
>
> Can someone take a look at this?
>
> as it looks like incomplete package or webserver issue it’s best to report
> this to infra
>
> Thanks, Marcin
>
>
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/GE3G2RZ3TPXHV6YFPE7HVQ5GFSCGVTHU/
>



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100109)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/UMKIJCKVIADZXBMUSKLKM2LH4DZTQTXY/

Re: [ovirt-devel] ovirt-vmconsole package issues

[Nir Soffer]

On Wed, Sep 11, 2019 at 4:38 PM Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
>
> On 11 Sep 2019, at 11:19, Marcin Sobczyk  wrote:
>
> Hi,
>
> we get a lot of failures in the CI for el7 and fc29 regarding this package
> since this morning, i.e. for fc29:
>
>
> in which CI?
>

Here are some examples:
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11308/pipeline
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/11307/pipeline/123

Adding infra-support to file a bug.


>
>
>  
> [2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> [2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> [2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> [2019-09-11T07:55:42.107Z]
>  [MIRROR] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: Interrupted by header 
> callback: Server reports Content-Length: 38712 but expected size is: 38804
>  
> [2019-09-11T07:55:42.107Z]
>  [FAILED] ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: No more mirrors to try - 
> All mirrors were already tried without success
>  
> [2019-09-11T07:55:42.107Z]
>  Error: Error downloading packages:
>  
> [2019-09-11T07:55:42.107Z]
>Cannot download noarch/ovirt-vmconsole-1.0.7-3.fc29.noarch.rpm: All 
> mirrors were tried
>
> Can someone take a look at this?
>
> as it looks like incomplete package or webserver issue it’s best to report
> this to infra
>
> Thanks, Marcin
>
>
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/GE3G2RZ3TPXHV6YFPE7HVQ5GFSCGVTHU/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/IOCBO6HATBVC3V2ZAPYYUSAWPQC76LYZ/

[JIRA] (OVIRT-2794) OST is broken since this morning - looks like infra issue

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2794:
-

 Summary: OST is broken since this morning - looks like infra issue
 Key: OVIRT-2794
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2794
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


The last successful build was today at 08:10:

Since then all builds fail very early with the error below - which is not
related to oVirt.

Removing image:
sha256:f8e5aa8e979155e074411bfef9adade6cdcdf3a5a2eb1d5ad2dbf0288d585ffa,
force=True
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/docker/api/client.py", line 222,
in _raise_for_status
response.raise_for_status()
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 893, in
raise_for_status
raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
http+docker://localunixsocket/v1.30/images/sha256:f8e5aa8e979155e074411bfef9adade6cdcdf3a5a2eb1d5ad2dbf0288d585ffa?force=True=False

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/home/jenkins/workspace/ovirt-system-tests_manual/jenkins/scripts/docker_cleanup.py",
line 349, in 
main()
  File
"/home/jenkins/workspace/ovirt-system-tests_manual/jenkins/scripts/docker_cleanup.py",
line 37, in main
safe_image_cleanup(client, whitelisted_repos)
  File
"/home/jenkins/workspace/ovirt-system-tests_manual/jenkins/scripts/docker_cleanup.py",
line 107, in safe_image_cleanup
_safe_rm(client, parent)
  File
"/home/jenkins/workspace/ovirt-system-tests_manual/jenkins/scripts/docker_cleanup.py",
line 329, in _safe_rm
client.images.remove(image_id, force=force)
  File "/usr/lib/python3.6/site-packages/docker/models/images.py", line
288, in remove
self.client.api.remove_image(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/docker/utils/decorators.py", line
19, in wrapped
return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/docker/api/image.py", line 481, in
remove_image
return self._result(res, True)
  File "/usr/lib/python3.6/site-packages/docker/api/client.py", line 228,
in _result
self._raise_for_status(response)
  File "/usr/lib/python3.6/site-packages/docker/api/client.py", line 224,
in _raise_for_status
raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3.6/site-packages/docker/errors.py", line 31, in
create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("reference does not
exist")

Aborting.

Build step 'Execute shell' marked build as failure


x

[image: Failed > Console Output]
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5542/console>
#5542 <https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5542/>
Sep 5, 2019 3:02 PM
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5542/>

[image: Failed > Console Output]
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5541/console>
#5541 <https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5541/>
Sep 5, 2019 3:02 PM
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5541/>

[image: Failed > Console Output]
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5540/console>
#5540 <https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5540/>
Sep 5, 2019 3:01 PM
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5540/>

[image: Failed > Console Output]
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5539/console>
#5539 <https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5539/>
Sep 5, 2019 2:13 PM
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5539/>

[image: Failed > Console Output]
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5538/console>
#5538 <https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5538/>
Sep 5, 2019 1:58 PM
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5538/>

[image: Failed > Console Output]
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5537/console>
#5537 <https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5537/>
Sep 5, 2019 1:50 PM
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5537/>

[image: Failed > Console Output]
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5536/console>
#5536 <https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5536/>
Sep 5, 2019 10:21 AM
<https://jenkins.ovirt.org/job/ovirt-system-tests_manual/5536/>
 [image: x]
<http://jenkins.ovirt.org/job/ovirt-system-tests_manu

[JIRA] (OVIRT-2783) CI fails - tox error

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39704#comment-39704
 ] 

Nir Soffer commented on OVIRT-2783:
---

On Sun, Aug 25, 2019, 15:19 Pavel Bar  wrote:

> Hi,
> We experience problems with failed CI on VDSM patches.
>

Does it work locally and in travis?

Examples:
> http://jenkins.ovirt.org/job/vdsm_standard-check-patch/10633/
> http://jenkins.ovirt.org/job/vdsm_standard-check-patch/10634/
> http://jenkins.ovirt.org/job/vdsm_standard-check-patch/10635/
>
> This is what we see in logs:
>
> out=`tox --version`; \
> if [ $? -ne 0 ]; then \
> echo "Error: cannot run tox, please install tox \
> 2.9.1 or later"; \
> exit 1; \
> fi; \
> version=`echo $out | cut -d' ' -f1`; \
> if python2.7 build-aux/vercmp $version 2.9.1; then \
> echo "Error: tox is too old, please install tox \
> 2.9.1 or later"; \
> exit 1; \
> fi
> Traceback (most recent call last):
>   File "/usr/bin/tox", line 7, in 
> from tox import cmdline
>   File "/usr/lib/python2.7/site-packages/tox/__init__.py", line 4, in
> 
> from .hookspecs import hookimpl
>   File "/usr/lib/python2.7/site-packages/tox/hookspecs.py", line 4, in
> 
> from pluggy import HookimplMarker
>   File "/usr/lib/python2.7/site-packages/pluggy/__init__.py", line 16, in
> 
> from .manager import PluginManager, PluginValidationError
>   File "/usr/lib/python2.7/site-packages/pluggy/manager.py", line 6, in
> 
> import importlib_metadata
>   File "/usr/lib/python2.7/site-packages/importlib_metadata/__init__.py",
> line 9, in 
> import zipp
>   File "/usr/lib/python2.7/site-packages/zipp.py", line 12, in 
> import more_itertools
>   File "/usr/lib/python2.7/site-packages/more_itertools/__init__.py", line
> 1, in 
> from more_itertools.more import *  # noqa
>   File "/usr/lib/python2.7/site-packages/more_itertools/more.py", line 340
> def _collate(*iterables, key=lambda a: a, reverse=False):
>^
> SyntaxError: invalid syntax
> Error: cannot run tox, please install tox 2.9.1 or later
> make: *** [tox] Error 1
> + teardown
> + res=2
> + '[' 2 -ne 0 ']'
> + echo '*** err: 2'
> *** err: 2
> + collect_logs
> + tar --directory /var/log --exclude 'journal/*' -czf
> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/exported-artifacts/mock_varlogs.tar.gz
> .
> + tar --directory /var/host_log --exclude 'journal/*' -czf
> /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/exported-artifacts/host_varlogs.tar.gz
> .
> + teardown_storage
> + python2 tests/storage/userstorage.py teardown
>
> Can someone look at it?
>
> Thank you in advance!
>
> Pavel
>
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/VADZM35IRVN4EY24MOM3JIHLWQKASLRG/
>

> CI fails - tox error
> 
>
> Key: OVIRT-2783
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2783
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Pavel Bar
>Assignee: infra
>
> Hi,
> We experience problems with failed CI on VDSM patches.
> Examples:
> http://jenkins.ovirt.org/job/vdsm_standard-check-patch/10633/
> http://jenkins.ovirt.org/job/vdsm_standard-check-patch/10634/
> http://jenkins.ovirt.org/job/vdsm_standard-check-patch/10635/
> This is what we see in logs:
> out=`tox --version`; \
> if [ $? -ne 0 ]; then \
> echo "Error: cannot run tox, please install tox \
> 2.9.1 or later"; \
> exit 1; \
> fi; \
> version=`echo $out | cut -d' ' -f1`; \
> if python2.7 build-aux/vercmp $version 2.9.1; then \
> echo "Error: tox is too old, please install tox \
> 2.9.1 or later"; \
> exit 1; \
> fi
> Traceback (most recent call last):
>   File "/usr/bin/tox", line 7, in 
> from tox import cmdline
>   File "/usr/lib/python2.7/site-packages/tox/__init__.py", line 4, in
> 
> from .hookspecs import hookimpl
>   File "/usr/lib/python2.7/site-packages/tox/hookspecs.py", line 4, in
> 
> from pluggy import HookimplMarker
>   File "/usr/lib/python2.7/site-packages/pluggy/__init__.py", line 16, in
>

[JIRA] (OVIRT-2778) Re: [ovirt-devel] vdsm_standard-check-patch is stuck on Archiving artifacts

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39664#comment-39664
 ] 

Nir Soffer commented on OVIRT-2778:
---

On Sat, Aug 17, 2019 at 3:06 AM Nir Soffer  wrote:

> This looks like a bug, so adding infra-support - this will open a ticket
> and someone will look into it.
>
> On Sat, Aug 17, 2019 at 1:12 AM Amit Bawer  wrote:
>
>> Hi
>> Unable to run CI builds and OSTs due to indefinite processing over fc29 
>> Archiving
>> artifacts phase
>> Example run:
>> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10453/console
>>
>
We have 2 stuck builds:
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10454/
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10453/

Both started 4 hours ago.

It possible to abort jobs from jenkins UI, but usually it cause more
trouble because
of partial cleanup, so lets let infra team handle this properly.


> ___
>> Devel mailing list -- de...@ovirt.org
>> To unsubscribe send an email to devel-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/JU3IZE2IQLVTVLRYCOOQZSQUC3QNLUD4/
>>
>

> Re: [ovirt-devel] vdsm_standard-check-patch is stuck on Archiving artifacts
> ---
>
> Key: OVIRT-2778
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2778
> Project: oVirt - virtualization made easy
>  Issue Type: By-EMAIL
>Reporter: Nir Soffer
>Assignee: infra
>
> This looks like a bug, so adding infra-support - this will open a ticket
> and someone will look into it.
> On Sat, Aug 17, 2019 at 1:12 AM Amit Bawer  wrote:
> > Hi
> > Unable to run CI builds and OSTs due to indefinite processing over fc29 
> > Archiving
> > artifacts phase
> > Example run:
> > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10453/console
> > ___
> > Devel mailing list -- de...@ovirt.org
> > To unsubscribe send an email to devel-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> > https://lists.ovirt.org/archives/list/de...@ovirt.org/message/JU3IZE2IQLVTVLRYCOOQZSQUC3QNLUD4/
> >



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100107)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/TBSGWBB7VJHQ5AOMGIJRWKSPO3UKELRO/

[JIRA] (OVIRT-2778) Re: [ovirt-devel] vdsm_standard-check-patch is stuck on Archiving artifacts

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2778:
-

 Summary: Re: [ovirt-devel] vdsm_standard-check-patch is stuck on 
Archiving artifacts
 Key: OVIRT-2778
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2778
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


This looks like a bug, so adding infra-support - this will open a ticket
and someone will look into it.

On Sat, Aug 17, 2019 at 1:12 AM Amit Bawer  wrote:

> Hi
> Unable to run CI builds and OSTs due to indefinite processing over fc29 
> Archiving
> artifacts phase
> Example run:
> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10453/console
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/JU3IZE2IQLVTVLRYCOOQZSQUC3QNLUD4/
>



--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100107)
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/J4FFCRXPMZ75NSR3GPPGF62JCX3RC355/

[JIRA] (OVIRT-2776) 51 jobs in the queue, all workers are idle

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2776:
-

 Summary: 51 jobs in the queue, all workers are idle
 Key: OVIRT-2776
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2776
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


After waiting for build-artifacts job for 36 minutes (typically finish in
15 minutes)
I found that jeknins seems to be broken.

We depend on CI and OST for merging patches.

Can someone check this?

---

[image: collapse]
<https://jenkins.ovirt.org/toggleCollapse?paneId=buildQueue>Build Queue (51)
vdsm_standard-on-merge
<https://jenkins.ovirt.org/job/vdsm_standard-on-merge/> [image: cancel this
build] <https://jenkins.ovirt.org/queue/cancelItem?id=30571>
vdsm_standard-on-merge
<https://jenkins.ovirt.org/job/vdsm_standard-on-merge/> [image: cancel this
build] <https://jenkins.ovirt.org/queue/cancelItem?id=30579>
vdsm_standard-on-merge
<https://jenkins.ovirt.org/job/vdsm_standard-on-merge/> [image: cancel this
build] <https://jenkins.ovirt.org/queue/cancelItem?id=30661>
vdsm_standard-on-merge
<https://jenkins.ovirt.org/job/vdsm_standard-on-merge/> [image: cancel this
build] <https://jenkins.ovirt.org/queue/cancelItem?id=30707>
part of standard-manual-runner #622
<https://jenkins.ovirt.org/job/standard-manual-runner/622/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30891>
part of jenkins_standard-check-patch #3199
<https://jenkins.ovirt.org/job/jenkins_standard-check-patch/3199/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30889>
part of standard-manual-runner #621
<https://jenkins.ovirt.org/job/standard-manual-runner/621/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30887>
part of jenkins_standard-check-patch #3197
<https://jenkins.ovirt.org/job/jenkins_standard-check-patch/3197/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30837>
part of vdsm_standard-check-patch #10405
<https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10405/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30832>
part of vdsm_standard-check-patch #10404
<https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10404/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30829>
part of vdsm_standard-check-patch #10403
<https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10403/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30826>
part of ovirt-system-tests_standard-check-patch #5206
<https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/5206/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30817>
part of ovirt-engine-ui-extensions_standard-on-merge #54
<https://jenkins.ovirt.org/job/ovirt-engine-ui-extensions_standard-on-merge/54/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30814>
part of nmstate_nmstate_standard-check-pr #2095
<https://jenkins.ovirt.org/job/nmstate_nmstate_standard-check-pr/2095/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30808>
part of standard-manual-runner #620
<https://jenkins.ovirt.org/job/standard-manual-runner/620/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30804>
part of standard-manual-runner #619
<https://jenkins.ovirt.org/job/standard-manual-runner/619/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30773>
part of vdsm_standard-check-patch #10402
<https://jenkins.ovirt.org/job/vdsm_standard-check-patch/10402/>
<https://jenkins.io/redirect/troubleshooting/executor-starvation> [image:
cancel this build] <https://jenkins.ovirt.org/queue/cancelItem?id=30769>
part of vdsm_standard-check-patch #1

Re: [ovirt-devel] Re: CI: vdsm-standard-check-patch fails

[Nir Soffer]

On Wed, Aug 7, 2019 at 3:18 PM Amit Bawer  wrote:

>
>
> On Wed, Aug 7, 2019 at 3:14 PM Nir Soffer  wrote:
>
>> On Wed, Aug 7, 2019 at 3:06 PM Amit Bawer  wrote:
>>
>>>
>>>
>>> On Wed, Aug 7, 2019 at 2:53 PM Nir Soffer  wrote:
>>>
>>>> On Wed, Aug 7, 2019 at 1:23 PM Amit Bawer  wrote:
>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 7, 2019 at 11:19 AM Amit Bawer  wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 6, 2019 at 5:07 PM Nir Soffer  wrote:
>>>>>>
>>>>>>> On Tue, Aug 6, 2019 at 5:01 PM Amit Bawer  wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 6, 2019 at 4:58 PM Nir Soffer 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Tue, Aug 6, 2019 at 11:27 AM Amit Bawer 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I have seen some improvement: when I re-trigger the CI per patch
>>>>>>>>>> I am able to pass or get the actual test errors if any (if not on 
>>>>>>>>>> first
>>>>>>>>>> try, then on second one).
>>>>>>>>>> Probably not a very useful information, but I have noticed that
>>>>>>>>>> when I push 30+ patches at the same
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do not do that, jenkins cannot handle 30 concurrent builds, and is
>>>>>>>>> it also bad for reviewers,
>>>>>>>>> getting several mails about every patch in your chain, for every
>>>>>>>>> rebase.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Is there is a way to prevent CI from running per gerrit push
>>>>>>>> (without working on 30 different branches) ?
>>>>>>>>
>>>>>>>
>>>>>>> I don't know about such way.
>>>>>>>
>>>>>>
>>>>>> A legit option could be adding the Skip CI plugin to jenkins plugins
>>>>>> [1]; with that devs can add "[skip ci]" to their commit messages to 
>>>>>> prevent
>>>>>> jenkins from automatically launching CI upon push.
>>>>>>
>>>>>
>>>> Do you want to modify the commit message for every patch to decide if
>>>> ci should run or not?
>>>>
>>>
>>> I think that having the option to knowingly disable automated CI in some
>>> cases is useful. We could always re-trigger when time is right [3].
>>> [3] https://jenkins.ovirt.org/login?from=%2Fgerrit_manual_trigger%2F
>>>
>>
>> This is too much work, but I think today we can add a comment to gerrit
>> like
>>
>> ci please test
>>
>> That will trigger a build of this patch.
>>
>
> Indeed, but it leaves the "Continuous-Integration" mark untouched in
> gerrit, giving the wrong impression this patch is still CI failed. The
> re-trigger UI takes care for that as well.
>

This is a bug in our CI infra that should be fix.



>
>
>>
>>
>>>
>>>
>>>>
>>>>> Another option is to emulate the behaviour in the existing gerrit
>>>>>> plugin (guess there is already such one in ovirt's jenkins), for example
>>>>>> skipping by a topic regex [2].
>>>>>>
>>>>>
>>>> Not clear how this will help.
>>>>
>>>
>>> If I make a gerrit topic with some name like "my_feature_skip_ci" I can
>>> control whether I want to have automated CI for its patches.
>>> When I want to go back to normal I can rename it to "my_feature" and
>>> have CI per push as usual.
>>>
>>>
>>>> I think a possible solution can be running only the top patch in a
>>>> changeset, same way we have in travis,
>>>> and the same way systems that grab patches from mailing lists work.
>>>> Every post to gerrit will trigger one
>>>> build, instead of one build per patch in the chain.
>>>>
>>>
>>> That could do as well.
>>>
>>>
>>>> Of course this will allow merging broken patches that are fixe

Re: [ovirt-devel] Re: CI: vdsm-standard-check-patch fails

[Nir Soffer]

On Wed, Aug 7, 2019 at 3:06 PM Amit Bawer  wrote:

>
>
> On Wed, Aug 7, 2019 at 2:53 PM Nir Soffer  wrote:
>
>> On Wed, Aug 7, 2019 at 1:23 PM Amit Bawer  wrote:
>>
>>>
>>>
>>> On Wed, Aug 7, 2019 at 11:19 AM Amit Bawer  wrote:
>>>
>>>>
>>>>
>>>> On Tue, Aug 6, 2019 at 5:07 PM Nir Soffer  wrote:
>>>>
>>>>> On Tue, Aug 6, 2019 at 5:01 PM Amit Bawer  wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 6, 2019 at 4:58 PM Nir Soffer  wrote:
>>>>>>
>>>>>>> On Tue, Aug 6, 2019 at 11:27 AM Amit Bawer 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have seen some improvement: when I re-trigger the CI per patch I
>>>>>>>> am able to pass or get the actual test errors if any (if not on first 
>>>>>>>> try,
>>>>>>>> then on second one).
>>>>>>>> Probably not a very useful information, but I have noticed that
>>>>>>>> when I push 30+ patches at the same
>>>>>>>>
>>>>>>>
>>>>>>> Do not do that, jenkins cannot handle 30 concurrent builds, and is
>>>>>>> it also bad for reviewers,
>>>>>>> getting several mails about every patch in your chain, for every
>>>>>>> rebase.
>>>>>>>
>>>>>>
>>>>>> Is there is a way to prevent CI from running per gerrit push (without
>>>>>> working on 30 different branches) ?
>>>>>>
>>>>>
>>>>> I don't know about such way.
>>>>>
>>>>
>>>> A legit option could be adding the Skip CI plugin to jenkins plugins
>>>> [1]; with that devs can add "[skip ci]" to their commit messages to prevent
>>>> jenkins from automatically launching CI upon push.
>>>>
>>>
>> Do you want to modify the commit message for every patch to decide if ci
>> should run or not?
>>
>
> I think that having the option to knowingly disable automated CI in some
> cases is useful. We could always re-trigger when time is right [3].
> [3] https://jenkins.ovirt.org/login?from=%2Fgerrit_manual_trigger%2F
>

This is too much work, but I think today we can add a comment to gerrit like

ci please test

That will trigger a build of this patch.


>
>
>>
>>> Another option is to emulate the behaviour in the existing gerrit plugin
>>>> (guess there is already such one in ovirt's jenkins), for example skipping
>>>> by a topic regex [2].
>>>>
>>>
>> Not clear how this will help.
>>
>
> If I make a gerrit topic with some name like "my_feature_skip_ci" I can
> control whether I want to have automated CI for its patches.
> When I want to go back to normal I can rename it to "my_feature" and have
> CI per push as usual.
>
>
>> I think a possible solution can be running only the top patch in a
>> changeset, same way we have in travis,
>> and the same way systems that grab patches from mailing lists work. Every
>> post to gerrit will trigger one
>> build, instead of one build per patch in the chain.
>>
>
> That could do as well.
>
>
>> Of course this will allow merging broken patches that are fixed by a
>> later patch in the chain, which
>> is also not ideal, but it is better given our restricted resources.
>>
>
> We can re-trigger CI manually in this case as part of the verification
> process.
>

>
>> +Anton Marchukov   I have been told you might be
>>> familiar with a similar solution.
>>>
>>>>
>>>> [1] https://plugins.jenkins.io/ci-skip
>>>> [2]
>>>> https://stackoverflow.com/questions/37807941/how-can-i-get-jenkins-gerrit-trigger-to-ignore-my-ci-users-commits
>>>>
>>>>
>>>>>
>>>>> I'm using keeping several small active branches. While you wait for
>>>>> reviews on one topic
>>>>> you can work on the other branches.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>> time the AWS connection issue arises constantly.
>>>>>>>>
>>>>>>>> On Sun, Aug 4, 2019 at 4:49 PM Eyal Edri  wrote:
>>>>>>>>
>>>>>>>>> This was reported already

Re: [ovirt-devel] Re: CI: vdsm-standard-check-patch fails

[Nir Soffer]

On Wed, Aug 7, 2019 at 1:23 PM Amit Bawer  wrote:

>
>
> On Wed, Aug 7, 2019 at 11:19 AM Amit Bawer  wrote:
>
>>
>>
>> On Tue, Aug 6, 2019 at 5:07 PM Nir Soffer  wrote:
>>
>>> On Tue, Aug 6, 2019 at 5:01 PM Amit Bawer  wrote:
>>>
>>>>
>>>>
>>>> On Tue, Aug 6, 2019 at 4:58 PM Nir Soffer  wrote:
>>>>
>>>>> On Tue, Aug 6, 2019 at 11:27 AM Amit Bawer  wrote:
>>>>>
>>>>>> I have seen some improvement: when I re-trigger the CI per patch I am
>>>>>> able to pass or get the actual test errors if any (if not on first try,
>>>>>> then on second one).
>>>>>> Probably not a very useful information, but I have noticed that when
>>>>>> I push 30+ patches at the same
>>>>>>
>>>>>
>>>>> Do not do that, jenkins cannot handle 30 concurrent builds, and is it
>>>>> also bad for reviewers,
>>>>> getting several mails about every patch in your chain, for every
>>>>> rebase.
>>>>>
>>>>
>>>> Is there is a way to prevent CI from running per gerrit push (without
>>>> working on 30 different branches) ?
>>>>
>>>
>>> I don't know about such way.
>>>
>>
>> A legit option could be adding the Skip CI plugin to jenkins plugins [1];
>> with that devs can add "[skip ci]" to their commit messages to prevent
>> jenkins from automatically launching CI upon push.
>>
>
Do you want to modify the commit message for every patch to decide if ci
should run or not?


> Another option is to emulate the behaviour in the existing gerrit plugin
>> (guess there is already such one in ovirt's jenkins), for example skipping
>> by a topic regex [2].
>>
>
Not clear how this will help.

I think a possible solution can be running only the top patch in a
changeset, same way we have in travis,
and the same way systems that grab patches from mailing lists work. Every
post to gerrit will trigger one
build, instead of one build per patch in the chain.

Of course this will allow merging broken patches that are fixed by a later
patch in the chain, which
is also not ideal, but it is better given our restricted resources.

+Anton Marchukov   I have been told you might be
> familiar with a similar solution.
>
>>
>> [1] https://plugins.jenkins.io/ci-skip
>> [2]
>> https://stackoverflow.com/questions/37807941/how-can-i-get-jenkins-gerrit-trigger-to-ignore-my-ci-users-commits
>>
>>
>>>
>>> I'm using keeping several small active branches. While you wait for
>>> reviews on one topic
>>> you can work on the other branches.
>>>
>>>
>>>
>>>>
>>>>>
>>>>>> time the AWS connection issue arises constantly.
>>>>>>
>>>>>> On Sun, Aug 4, 2019 at 4:49 PM Eyal Edri  wrote:
>>>>>>
>>>>>>> This was reported already and AFAIK its a network issue between AWS
>>>>>>> and PHX which is still being investigated.
>>>>>>> Evgheni, any insights or update on the issue? should we involve
>>>>>>> debugging from amazon side?
>>>>>>>
>>>>>>> On Sun, Aug 4, 2019 at 4:46 PM Amit Bawer  wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> CI seems to fail constantly for unavailable remote gerrit
>>>>>>>> repository.
>>>>>>>> Example can be seen here:
>>>>>>>> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9415/console
>>>>>>>> ___
>>>>>>>> Devel mailing list -- de...@ovirt.org
>>>>>>>> To unsubscribe send an email to devel-le...@ovirt.org
>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>>>>> oVirt Code of Conduct:
>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>>>>> List Archives:
>>>>>>>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/AHPHUZAABAQNWEMD2JQ6WARHJRDTYCPI/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Eyal edri
>>>>>>>
>>>>>>> He / Him / His
>>>>>>>
>>>>>>>
>>>>>>> MANAGER
>>>>>>>
>>>>>>> CONTINUOUS PRODUCTIZATION
>>>>>>>
>>>>>>> SYSTEM ENGINEERING
>>>>>>>
>>>>>>> Red Hat <https://www.redhat.com/>
>>>>>>> <https://red.ht/sig>
>>>>>>> phone: +972-9-7692018
>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)
>>>>>>>
>>>>>> ___
>>>>>> Devel mailing list -- de...@ovirt.org
>>>>>> To unsubscribe send an email to devel-le...@ovirt.org
>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>>> oVirt Code of Conduct:
>>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>>> List Archives:
>>>>>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/W6DUMIUSN5DPUVGUFUNHF2ZALB5I4JPZ/
>>>>>>
>>>>>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/6NNLAWOI3FIMVCJF6ISCNHXYQGZRHCIC/

Re: [ovirt-devel] Re: CI: vdsm-standard-check-patch fails

[Nir Soffer]

On Tue, Aug 6, 2019 at 5:01 PM Amit Bawer  wrote:

>
>
> On Tue, Aug 6, 2019 at 4:58 PM Nir Soffer  wrote:
>
>> On Tue, Aug 6, 2019 at 11:27 AM Amit Bawer  wrote:
>>
>>> I have seen some improvement: when I re-trigger the CI per patch I am
>>> able to pass or get the actual test errors if any (if not on first try,
>>> then on second one).
>>> Probably not a very useful information, but I have noticed that when I
>>> push 30+ patches at the same
>>>
>>
>> Do not do that, jenkins cannot handle 30 concurrent builds, and is it
>> also bad for reviewers,
>> getting several mails about every patch in your chain, for every rebase.
>>
>
> Is there is a way to prevent CI from running per gerrit push (without
> working on 30 different branches) ?
>

I don't know about such way.

I'm using keeping several small active branches. While you wait for reviews
on one topic
you can work on the other branches.



>
>>
>>> time the AWS connection issue arises constantly.
>>>
>>> On Sun, Aug 4, 2019 at 4:49 PM Eyal Edri  wrote:
>>>
>>>> This was reported already and AFAIK its a network issue between AWS and
>>>> PHX which is still being investigated.
>>>> Evgheni, any insights or update on the issue? should we involve
>>>> debugging from amazon side?
>>>>
>>>> On Sun, Aug 4, 2019 at 4:46 PM Amit Bawer  wrote:
>>>>
>>>>> Hi,
>>>>> CI seems to fail constantly for unavailable remote gerrit repository.
>>>>> Example can be seen here:
>>>>> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9415/console
>>>>> ___
>>>>> Devel mailing list -- de...@ovirt.org
>>>>> To unsubscribe send an email to devel-le...@ovirt.org
>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>> oVirt Code of Conduct:
>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>> List Archives:
>>>>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/AHPHUZAABAQNWEMD2JQ6WARHJRDTYCPI/
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Eyal edri
>>>>
>>>> He / Him / His
>>>>
>>>>
>>>> MANAGER
>>>>
>>>> CONTINUOUS PRODUCTIZATION
>>>>
>>>> SYSTEM ENGINEERING
>>>>
>>>> Red Hat <https://www.redhat.com/>
>>>> <https://red.ht/sig>
>>>> phone: +972-9-7692018
>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)
>>>>
>>> ___
>>> Devel mailing list -- de...@ovirt.org
>>> To unsubscribe send an email to devel-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/W6DUMIUSN5DPUVGUFUNHF2ZALB5I4JPZ/
>>>
>>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/IMLFBNCO6VQH55I3BMN5MOMMPFA5SBW7/

Re: [ovirt-devel] Re: CI: vdsm-standard-check-patch fails

[Nir Soffer]

On Tue, Aug 6, 2019 at 11:27 AM Amit Bawer  wrote:

> I have seen some improvement: when I re-trigger the CI per patch I am able
> to pass or get the actual test errors if any (if not on first try, then on
> second one).
> Probably not a very useful information, but I have noticed that when I
> push 30+ patches at the same
>

Do not do that, jenkins cannot handle 30 concurrent builds, and is it also
bad for reviewers,
getting several mails about every patch in your chain, for every rebase.


> time the AWS connection issue arises constantly.
>
> On Sun, Aug 4, 2019 at 4:49 PM Eyal Edri  wrote:
>
>> This was reported already and AFAIK its a network issue between AWS and
>> PHX which is still being investigated.
>> Evgheni, any insights or update on the issue? should we involve debugging
>> from amazon side?
>>
>> On Sun, Aug 4, 2019 at 4:46 PM Amit Bawer  wrote:
>>
>>> Hi,
>>> CI seems to fail constantly for unavailable remote gerrit repository.
>>> Example can be seen here:
>>> https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9415/console
>>> ___
>>> Devel mailing list -- de...@ovirt.org
>>> To unsubscribe send an email to devel-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/AHPHUZAABAQNWEMD2JQ6WARHJRDTYCPI/
>>>
>>
>>
>> --
>>
>> Eyal edri
>>
>> He / Him / His
>>
>>
>> MANAGER
>>
>> CONTINUOUS PRODUCTIZATION
>>
>> SYSTEM ENGINEERING
>>
>> Red Hat 
>> 
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel)
>>
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/W6DUMIUSN5DPUVGUFUNHF2ZALB5I4JPZ/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/WSSEWXCHCU6BYQCHRBMIVKLHEF3JYGYU/

[JIRA] (OVIRT-2768) All loader-container nodes are offline, 99 jobs in the queue, jobs stuck for 2 hours

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2768:
-

 Summary: All loader-container nodes are offline, 99 jobs in the 
queue, jobs stuck for 2 hours
 Key: OVIRT-2768
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2768
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


We depend on jenkins to run our tests and build artifacts in a timely
manner.

Here is example job, stuck for 2 hours:
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/9212/

This looks like the issue:

[image: collapse]
<https://jenkins.ovirt.org/toggleCollapse?paneId=executors>Build Executor
Status <https://jenkins.ovirt.org/computer/>
 master <https://jenkins.ovirt.org/computer/(master)/>
1 Idle
2 Idle
3 Idle
4 Idle
 lfedora1.lf-dev.marist.edu
<https://jenkins.ovirt.org/computer/lfedora1.lf-dev.marist.edu/>
1 Idle
 loader-container-28ljp
<https://jenkins.ovirt.org/computer/loader-container-28ljp/>  (offline)
(suspended)
 loader-container-35tlc
<https://jenkins.ovirt.org/computer/loader-container-35tlc/>  (offline)
(suspended)
 loader-container-3j0ps
<https://jenkins.ovirt.org/computer/loader-container-3j0ps/>  (offline)
(suspended)
 loader-container-4391q
<https://jenkins.ovirt.org/computer/loader-container-4391q/>  (offline)
(suspended)
 loader-container-58v3d
<https://jenkins.ovirt.org/computer/loader-container-58v3d/>  (offline)
(suspended)
 loader-container-66r5r
<https://jenkins.ovirt.org/computer/loader-container-66r5r/>  (offline)
(suspended)
 loader-container-6k57j
<https://jenkins.ovirt.org/computer/loader-container-6k57j/>  (offline)
(suspended)
 loader-container-7813f
<https://jenkins.ovirt.org/computer/loader-container-7813f/>  (offline)
(suspended)
 loader-container-8dkj1
<https://jenkins.ovirt.org/computer/loader-container-8dkj1/>  (offline)
(suspended)
 loader-container-8hb98
<https://jenkins.ovirt.org/computer/loader-container-8hb98/>  (offline)
(suspended)
 loader-container-8v609
<https://jenkins.ovirt.org/computer/loader-container-8v609/>  (offline)
(suspended)
 loader-container-99f36
<https://jenkins.ovirt.org/computer/loader-container-99f36/>  (offline)
(suspended)
 loader-container-c16rq
<https://jenkins.ovirt.org/computer/loader-container-c16rq/>  (offline)
(suspended)
 loader-container-c1vkj
<https://jenkins.ovirt.org/computer/loader-container-c1vkj/>  (offline)
(suspended)
 loader-container-c4g8c
<https://jenkins.ovirt.org/computer/loader-container-c4g8c/>  (offline)
(suspended)
 loader-container-cqrx5
<https://jenkins.ovirt.org/computer/loader-container-cqrx5/>  (offline)
(suspended)
 loader-container-d1r8d
<https://jenkins.ovirt.org/computer/loader-container-d1r8d/>  (offline)
(suspended)
 loader-container-dpp70
<https://jenkins.ovirt.org/computer/loader-container-dpp70/>  (offline)
(suspended)
 loader-container-gz52q
<https://jenkins.ovirt.org/computer/loader-container-gz52q/>  (offline)
(suspended)
 loader-container-h9kcf
<https://jenkins.ovirt.org/computer/loader-container-h9kcf/>  (offline)
(suspended)
 loader-container-hjq0g
<https://jenkins.ovirt.org/computer/loader-container-hjq0g/>  (offline)
(suspended)
 loader-container-ht75p
<https://jenkins.ovirt.org/computer/loader-container-ht75p/>  (offline)
(suspended)
 loader-container-htstk
<https://jenkins.ovirt.org/computer/loader-container-htstk/>  (offline)
(suspended)
 loader-container-j47s3
<https://jenkins.ovirt.org/computer/loader-container-j47s3/>  (offline)
(suspended)
 loader-container-k2p10
<https://jenkins.ovirt.org/computer/loader-container-k2p10/>  (offline)
(suspended)
 loader-container-kc066
<https://jenkins.ovirt.org/computer/loader-container-kc066/>  (offline)
(suspended)
 loader-container-l1vd9
<https://jenkins.ovirt.org/computer/loader-container-l1vd9/>  (offline)
(suspended)
 loader-container-l7lqn
<https://jenkins.ovirt.org/computer/loader-container-l7lqn/>  (offline)
(suspended)
 loader-container-lqqkj
<https://jenkins.ovirt.org/computer/loader-container-lqqkj/>  (offline)
(suspended)
 loader-container-m8fkq
<https://jenkins.ovirt.org/computer/loader-container-m8fkq/>  (offline)
(suspended)
 loader-container-n2zml
<https://jenkins.ovirt.org/computer/loader-container-n2zml/>  (offline)
(suspended)
 loader-container-p7rxw
<https://jenkins.ovirt.org/computer/loader-container-p7rxw/>  (offline)
(suspended)
 loader-container-pqg56
<https://jenkins.ovirt.org/computer/loader-container-pqg56/>  (offline)
(suspended)
 loader-container-qtsdr
<https://jenkins.ovirt.org/computer/loader-container-qtsdr/>  (offline)
(suspended)
 loader-container-qxwnp
<https://jenkins.ovirt.org/computer/loader-container-qxwnp/>  (offline)
(suspended)
 loader-cont

[JIRA] (OVIRT-2741) Most el7 vdsm build fail to fetch source - bad slave?

[Nir Soffer (oVirt JIRA)]

[ 
https://ovirt-jira.atlassian.net/browse/OVIRT-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=39416#comment-39416
 ] 

Nir Soffer commented on OVIRT-2741:
---

On Fri, Jun 21, 2019 at 2:16 AM Nir Soffer  wrote:

> I have see many such failures today, here is 2 bulids - both on same slave
> (slave issue?)
>
> 1. [2019-06-20T22:23:12.401Z] Running on node:
> vm0015.workers-phx.ovirt.org (el7 phx nested local_disk)
>
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/6511/pipeline
>
> 2. [2019-06-20T22:20:50.288Z] Running on node:
> vm0015.workers-phx.ovirt.org (el7 phx nested local_disk)
>
> https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/6509/pipeline
>

Another one:
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/6515/pipeline

Same slave again:
[2019-06-20T23:45:38.183Z] Running on node: vm0015.workers-phx.ovirt.org
(el7 phx nested local_disk)


>
> Looks like there was no build on this slave in the last 2 days:
> https://jenkins.ovirt.org/computer/vm0015.workers-phx.ovirt.org/builds
>
> The failure looks like this:
>
> [2019-06-20T22:23:16.416Z] No credentials specified
>
> [2019-06-20T22:23:16.440Z] Fetching changes from the remote Git repository
>
> [2019-06-20T22:23:16.456Z] Cleaning workspace
>
> [2019-06-20T22:23:16.566Z] ERROR: Error fetching remote repo 'origin'
>
> [2019-06-20T22:23:16.566Z] hudson.plugins.git.GitException: Failed to
> fetch from https://gerrit.ovirt.org/vdsm
>
> [2019-06-20T22:23:16.566Z] at
> hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
>
> [2019-06-20T22:23:16.566Z] at
> hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
>
> [2019-06-20T22:23:16.566Z] at
> hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
>
> [2019-06-20T22:23:16.566Z] at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
>
> [2019-06-20T22:23:16.566Z] at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
>
> [2019-06-20T22:23:16.566Z] at
> org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
>
> [2019-06-20T22:23:16.566Z] at
> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
>
> [2019-06-20T22:23:16.566Z] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> [2019-06-20T22:23:16.566Z] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> [2019-06-20T22:23:16.566Z] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> [2019-06-20T22:23:16.566Z] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> [2019-06-20T22:23:16.566Z] at java.lang.Thread.run(Thread.java:748)
>
> [2019-06-20T22:23:16.566Z] Caused by: hudson.plugins.git.GitException:
> Command "git clean -fdx" returned status code 1:
>
> [2019-06-20T22:23:16.566Z] stdout:
>
> [2019-06-20T22:23:16.566Z] stderr: warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage___init___py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncevent_py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncutils_py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blkdiscard_py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockSD_py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockVolume_py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockdev_py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_check_py.html
>
> [2019-06-20T22:23:16.566Z] warning: failed to remove
> tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vd

[JIRA] (OVIRT-2741) Most el7 vdsm build fail to fetch source - bad slave?

[Nir Soffer (oVirt JIRA)]

Nir Soffer created OVIRT-2741:
-

 Summary: Most el7 vdsm build fail to fetch source - bad slave?
 Key: OVIRT-2741
 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2741
 Project: oVirt - virtualization made easy
  Issue Type: By-EMAIL
Reporter: Nir Soffer
Assignee: infra


I have see many such failures today, here is 2 bulids - both on same slave
(slave issue?)

1. [2019-06-20T22:23:12.401Z] Running on node: vm0015.workers-phx.ovirt.org
(el7 phx nested local_disk)

https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/6511/pipeline

2. [2019-06-20T22:20:50.288Z] Running on node: vm0015.workers-phx.ovirt.org
(el7 phx nested local_disk)

https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-patch/detail/vdsm_standard-check-patch/6509/pipeline

Looks like there was no build on this slave in the last 2 days:
https://jenkins.ovirt.org/computer/vm0015.workers-phx.ovirt.org/builds

The failure looks like this:

[2019-06-20T22:23:16.416Z] No credentials specified

[2019-06-20T22:23:16.440Z] Fetching changes from the remote Git repository

[2019-06-20T22:23:16.456Z] Cleaning workspace

[2019-06-20T22:23:16.566Z] ERROR: Error fetching remote repo 'origin'

[2019-06-20T22:23:16.566Z] hudson.plugins.git.GitException: Failed to fetch
from https://gerrit.ovirt.org/vdsm

[2019-06-20T22:23:16.566Z] at
hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)

[2019-06-20T22:23:16.566Z] at
hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)

[2019-06-20T22:23:16.566Z] at
hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)

[2019-06-20T22:23:16.566Z] at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)

[2019-06-20T22:23:16.566Z] at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)

[2019-06-20T22:23:16.566Z] at
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)

[2019-06-20T22:23:16.566Z] at
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)

[2019-06-20T22:23:16.566Z] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

[2019-06-20T22:23:16.566Z] at
java.util.concurrent.FutureTask.run(FutureTask.java:266)

[2019-06-20T22:23:16.566Z] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

[2019-06-20T22:23:16.566Z] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

[2019-06-20T22:23:16.566Z] at java.lang.Thread.run(Thread.java:748)

[2019-06-20T22:23:16.566Z] Caused by: hudson.plugins.git.GitException:
Command "git clean -fdx" returned status code 1:

[2019-06-20T22:23:16.566Z] stdout:

[2019-06-20T22:23:16.566Z] stderr: warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage___init___py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncevent_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_asyncutils_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blkdiscard_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockSD_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockVolume_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_blockdev_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_check_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_clusterlock_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_compat_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_constants_py.html

[2019-06-20T22:23:16.566Z] warning: failed to remove
tests/htmlcov-storage-py27/_home_jenkins_workspace_vdsm_standard-check-patch_vdsm_lib_vdsm_storage_curlImgWrap_py.html

[2019-06-20T22:23:16.566Z] warning: failed

Re: [CQ]: 100778, 6 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause

[Nir Soffer]

On Wed, Jun 19, 2019 at 7:03 PM Dafna Ron  wrote:

> this was a failed build-artifacts.
> its now fixed as there are several passing builds after this one
>

Why build-artifacts failed?

Do we have a retry on failures?


> On Wed, Jun 19, 2019 at 3:16 AM oVirt Jenkins  wrote:
>
>> A system test invoked by the "ovirt-master" change queue including change
>> 100778,6 (vdsm) failed. However, this change seems not to be the root
>> cause for
>> this failure. Change 100783,7 (vdsm) that this change depends on or is
>> based
>> on, was detected as the cause of the testing failures.
>>
>> This change had been removed from the testing queue. Artifacts built from
>> this
>> change will not be released until either change 100783,7 (vdsm) is fixed
>> and
>> this change is updated to refer to or rebased on the fixed version, or
>> this
>> change is modified to no longer depend on it.
>>
>> For further details about the change see:
>> https://gerrit.ovirt.org/#/c/100778/6
>>
>> For further details about the change that seems to be the root cause
>> behind the
>> testing failures see:
>> https://gerrit.ovirt.org/#/c/100783/7
>>
>> For failed test results see:
>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14624/
>> ___
>> Infra mailing list -- infra@ovirt.org
>> To unsubscribe send an email to infra-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CHXVUBUQZO3OS2OQ2FWEGSFB2KS7F33V/
>>
> ___
> Infra mailing list -- infra@ovirt.org
> To unsubscribe send an email to infra-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/FNSPEESXJOPRYBRKKABJ3PUGILZ2R53X/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/IQNGNT2ZAROG53TAFERBJE7A6BMFHARW/