[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-25 Thread Eitan Raviv 
ovirt-engine change queue and ovit-system-tests check_patch also look good
with ovirt-engine rpm including Ahmed's revert.

Thanks

On Mon, Mar 25, 2019 at 9:38 PM Dominik Holler  wrote:

> This issue seems to be solved now, please let me know if not.
>
> The build
> https://jenkins.ovirt.org/view/oVirt system
> tests/job/ovirt-system-tests_manual/4408/
> with ovirt-engine d0a215d862eb819f0bbdd51fed012f9b972c1bdf
> which includes  Ahmed's commit a236c90d54652503d43d8315582effb74050d22e
> succeeded.
> Also the build
> https://jenkins.ovirt.org/view/oVirt system
> tests/job/ovirt-system-tests_manual/4407/
> with default repos succeeded.
>
> On Mon, 25 Mar 2019 08:58:25 +0200
> Eitan Raviv  wrote:
>
> > the patch was merged on 4.3 but test run [1] has build [2] which is one
> > patch before Ahmed's merge...
> >
> >
> > [1]
> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> > [2]
> ovirt-engine-4.3.2.2-0.0.master.20190324105929.git8b0969c.el7.noarch.rpm
> >
> >
> > On Mon, Mar 25, 2019 at 8:42 AM Dan Kenigsberg 
> wrote:
> >
> > > but
> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> > > is still failing
> > > Was your patch merged?
> > >
> > > On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet 
> wrote:
> > >
> > >> patched 4.3!
> > >>
> > >> On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv 
> wrote:
> > >>
> > >>> After some off line discussions it seems that the change that should
> be
> > >>> implemented in order to solve the original problem (remove host
> fails due
> > >>> to disconnect storage in progress) is to leave the host in
> 'preparing for
> > >>> maintenance' until all relevant storage operations are completed.
> > >>>
> > >>> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg 
> > >>> wrote:
> > >>>
> >  Unfortunately, the network suite is still failing on
> > 
> >  Cannot edit Host. Related operation is currently in progress. Please
> >  try again later.
> > 
> >  Can you check if that's the same issue? Did you revert from 4.3 too?
> > 
> > 
> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
> > 
> > 
> > 
> >  On Sat, 23 Mar 2019, 21:27 Benny Zlotnik, 
> wrote:
> > 
> > > The patch was reverted on Thursday
> > >
> > > On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg 
> > > wrote:
> > >
> > >> I was told that intervening in the host state machine is
> delicate, but
> > >> I think that this is the only correct approach.
> > >>
> > >> Benny, Ahmad, Tal: do you have a plan to resolve this? We are
> entering
> > >> a third week with this constant failure.
> > >>
> > >>
> > >> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv 
> > >> wrote:
> > >> >
> > >> > I am not sure that locking both groups would be sufficient,
> because
> > >> there is still a chance that the removeNetowrks request will
> start and
> > >> acquire the lock before the DisconnectStorage operation starts.
> > >> > So probably the correct and full proof solution is to not move
> the
> > >> host to maintenance until all related storage ops terminate.
> > >> >
> > >> >
> > >> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
> > >> stira...@redhat.com> wrote:
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri 
> > >> wrote:
> > >> >>>
> > >> >>> Not sure if all the same issue, but seems to failing around
> the
> > >> same time:
> > >> >>>
> > >> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824
> 12
> > >> hr - #825 57 min  integ-tests
> > >> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr -
> #1043
> > >> 12 hr - #1045 51 min  integ-tests
> > >> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr -
> #11
> > >> 24 sec  integ-tests
> > >> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr -
> #11
> > >> 10 min  integ-tests
> > >> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10
> hr -
> > >> #871 10 hr - #873 1 hr 36 min  integ-tests
> > >> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr -
> #1147
> > >> 11 hr - #1149 1 hr 26 min  integ-tests
> > >> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr -
> > >> #727 12 hr - #729 1 hr 54 min  integ-tests
> > >> >>
> > >> >>
> > >> >> It's a NullPointerException on engine side:
> > >> >> I opened a bug here:
> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not
> on
> > >> POST
> > >> >>
> > >> >>
> > >> >>>
> > >> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr -
> #926
> > >> 12 hr - #929 45 min  integ-tests
> > >> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10
> hr
> > >> - #187 10 hr - #190 45 min  integ-tests
> > >> >>>
> > >> >>> Links to jobs can be

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-25 Thread Dominik Holler 
This issue seems to be solved now, please let me know if not.

The build
https://jenkins.ovirt.org/view/oVirt system 
tests/job/ovirt-system-tests_manual/4408/
with ovirt-engine d0a215d862eb819f0bbdd51fed012f9b972c1bdf
which includes  Ahmed's commit a236c90d54652503d43d8315582effb74050d22e
succeeded.
Also the build
https://jenkins.ovirt.org/view/oVirt system 
tests/job/ovirt-system-tests_manual/4407/
with default repos succeeded.

On Mon, 25 Mar 2019 08:58:25 +0200
Eitan Raviv  wrote:

> the patch was merged on 4.3 but test run [1] has build [2] which is one
> patch before Ahmed's merge...
> 
> 
> [1] http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> [2] ovirt-engine-4.3.2.2-0.0.master.20190324105929.git8b0969c.el7.noarch.rpm
> 
> 
> On Mon, Mar 25, 2019 at 8:42 AM Dan Kenigsberg  wrote:
> 
> > but http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> > is still failing
> > Was your patch merged?
> >
> > On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet  wrote:
> >
> >> patched 4.3!
> >>
> >> On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv  wrote:
> >>
> >>> After some off line discussions it seems that the change that should be
> >>> implemented in order to solve the original problem (remove host fails due
> >>> to disconnect storage in progress) is to leave the host in 'preparing for
> >>> maintenance' until all relevant storage operations are completed.
> >>>
> >>> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg 
> >>> wrote:
> >>>
>  Unfortunately, the network suite is still failing on
> 
>  Cannot edit Host. Related operation is currently in progress. Please
>  try again later.
> 
>  Can you check if that's the same issue? Did you revert from 4.3 too?
> 
>  http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
> 
> 
> 
>  On Sat, 23 Mar 2019, 21:27 Benny Zlotnik,  wrote:
> 
> > The patch was reverted on Thursday
> >
> > On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg 
> > wrote:
> >
> >> I was told that intervening in the host state machine is delicate, but
> >> I think that this is the only correct approach.
> >>
> >> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
> >> a third week with this constant failure.
> >>
> >>
> >> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv 
> >> wrote:
> >> >
> >> > I am not sure that locking both groups would be sufficient, because
> >> there is still a chance that the removeNetowrks request will start and
> >> acquire the lock before the DisconnectStorage operation starts.
> >> > So probably the correct and full proof solution is to not move the
> >> host to maintenance until all related storage ops terminate.
> >> >
> >> >
> >> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
> >> stira...@redhat.com> wrote:
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri 
> >> wrote:
> >> >>>
> >> >>> Not sure if all the same issue, but seems to failing around the
> >> same time:
> >> >>>
> >> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12
> >> hr - #825 57 min  integ-tests
> >> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043
> >> 12 hr - #1045 51 min  integ-tests
> >> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11
> >> 24 sec  integ-tests
> >> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11
> >> 10 min  integ-tests
> >> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
> >> #871 10 hr - #873 1 hr 36 min  integ-tests
> >> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147
> >> 11 hr - #1149 1 hr 26 min  integ-tests
> >> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr -
> >> #727 12 hr - #729 1 hr 54 min  integ-tests
> >> >>
> >> >>
> >> >> It's a NullPointerException on engine side:
> >> >> I opened a bug here:
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on
> >> POST
> >> >>
> >> >>
> >> >>>
> >> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926
> >> 12 hr - #929 45 min  integ-tests
> >> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr
> >> - #187 10 hr - #190 45 min  integ-tests
> >> >>>
> >> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
> >> >>>
> >> >>> --
> >> >>>
> >> >>> Eyal edri
> >> >>>
> >> >>>
> >> >>> MANAGER
> >> >>>
> >> >>> RHV/CNV DevOps
> >> >>>
> >> >>> EMEA VIRTUALIZATION R
> >> >>>
> >> >>>
> >> >>> Red Hat EMEA
> >> >>>
> >> >>> TRIED. TESTED. TRUSTED.
> >> >>> phone: +972-9-7692018
> >> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >> >>
> >> >>

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-25 Thread Eitan Raviv 
the patch was merged on 4.3 but test run [1] has build [2] which is one
patch before Ahmed's merge...


[1] http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
[2] ovirt-engine-4.3.2.2-0.0.master.20190324105929.git8b0969c.el7.noarch.rpm


On Mon, Mar 25, 2019 at 8:42 AM Dan Kenigsberg  wrote:

> but http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
> is still failing
> Was your patch merged?
>
> On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet  wrote:
>
>> patched 4.3!
>>
>> On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv  wrote:
>>
>>> After some off line discussions it seems that the change that should be
>>> implemented in order to solve the original problem (remove host fails due
>>> to disconnect storage in progress) is to leave the host in 'preparing for
>>> maintenance' until all relevant storage operations are completed.
>>>
>>> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg 
>>> wrote:
>>>
 Unfortunately, the network suite is still failing on

 Cannot edit Host. Related operation is currently in progress. Please
 try again later.

 Can you check if that's the same issue? Did you revert from 4.3 too?

 http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/



 On Sat, 23 Mar 2019, 21:27 Benny Zlotnik,  wrote:

> The patch was reverted on Thursday
>
> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg 
> wrote:
>
>> I was told that intervening in the host state machine is delicate, but
>> I think that this is the only correct approach.
>>
>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
>> a third week with this constant failure.
>>
>>
>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv 
>> wrote:
>> >
>> > I am not sure that locking both groups would be sufficient, because
>> there is still a chance that the removeNetowrks request will start and
>> acquire the lock before the DisconnectStorage operation starts.
>> > So probably the correct and full proof solution is to not move the
>> host to maintenance until all related storage ops terminate.
>> >
>> >
>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
>> stira...@redhat.com> wrote:
>> >>
>> >>
>> >>
>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri 
>> wrote:
>> >>>
>> >>> Not sure if all the same issue, but seems to failing around the
>> same time:
>> >>>
>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12
>> hr - #825 57 min  integ-tests
>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043
>> 12 hr - #1045 51 min  integ-tests
>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11
>> 24 sec  integ-tests
>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11
>> 10 min  integ-tests
>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
>> #871 10 hr - #873 1 hr 36 min  integ-tests
>> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147
>> 11 hr - #1149 1 hr 26 min  integ-tests
>> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr -
>> #727 12 hr - #729 1 hr 54 min  integ-tests
>> >>
>> >>
>> >> It's a NullPointerException on engine side:
>> >> I opened a bug here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on
>> POST
>> >>
>> >>
>> >>>
>> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926
>> 12 hr - #929 45 min  integ-tests
>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr
>> - #187 10 hr - #190 45 min  integ-tests
>> >>>
>> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
>> >>>
>> >>> --
>> >>>
>> >>> Eyal edri
>> >>>
>> >>>
>> >>> MANAGER
>> >>>
>> >>> RHV/CNV DevOps
>> >>>
>> >>> EMEA VIRTUALIZATION R
>> >>>
>> >>>
>> >>> Red Hat EMEA
>> >>>
>> >>> TRIED. TESTED. TRUSTED.
>> >>> phone: +972-9-7692018
>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >>
>> >> ___
>> >> Devel mailing list -- devel@ovirt.org
>> >> To unsubscribe send an email to devel-le...@ovirt.org
>> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> >> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> >> List Archives:
>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
>> >
>> > ___
>> > Devel mailing list -- devel@ovirt.org
>> > To unsubscribe send an email to devel-le...@ovirt.org
>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> > oVirt Code of Conduct:
>>

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-25 Thread Dan Kenigsberg 
but http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/21/
is still failing
Was your patch merged?

On Sun, Mar 24, 2019 at 10:14 AM Ahmad Khiet  wrote:

> patched 4.3!
>
> On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv  wrote:
>
>> After some off line discussions it seems that the change that should be
>> implemented in order to solve the original problem (remove host fails due
>> to disconnect storage in progress) is to leave the host in 'preparing for
>> maintenance' until all relevant storage operations are completed.
>>
>> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg 
>> wrote:
>>
>>> Unfortunately, the network suite is still failing on
>>>
>>> Cannot edit Host. Related operation is currently in progress. Please try
>>> again later.
>>>
>>> Can you check if that's the same issue? Did you revert from 4.3 too?
>>>
>>> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
>>>
>>>
>>>
>>> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik,  wrote:
>>>
 The patch was reverted on Thursday

 On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg 
 wrote:

> I was told that intervening in the host state machine is delicate, but
> I think that this is the only correct approach.
>
> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
> a third week with this constant failure.
>
>
> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv  wrote:
> >
> > I am not sure that locking both groups would be sufficient, because
> there is still a chance that the removeNetowrks request will start and
> acquire the lock before the DisconnectStorage operation starts.
> > So probably the correct and full proof solution is to not move the
> host to maintenance until all related storage ops terminate.
> >
> >
> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
> stira...@redhat.com> wrote:
> >>
> >>
> >>
> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:
> >>>
> >>> Not sure if all the same issue, but seems to failing around the
> same time:
> >>>
> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr
> - #825 57 min  integ-tests
> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043
> 12 hr - #1045 51 min  integ-tests
> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11
> 24 sec  integ-tests
> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11
> 10 min  integ-tests
> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
> #871 10 hr - #873 1 hr 36 min  integ-tests
> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147
> 11 hr - #1149 1 hr 26 min  integ-tests
> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727
> 12 hr - #729 1 hr 54 min  integ-tests
> >>
> >>
> >> It's a NullPointerException on engine side:
> >> I opened a bug here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on
> POST
> >>
> >>
> >>>
> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12
> hr - #929 45 min  integ-tests
> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
> #187 10 hr - #190 45 min  integ-tests
> >>>
> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
> >>>
> >>> --
> >>>
> >>> Eyal edri
> >>>
> >>>
> >>> MANAGER
> >>>
> >>> RHV/CNV DevOps
> >>>
> >>> EMEA VIRTUALIZATION R
> >>>
> >>>
> >>> Red Hat EMEA
> >>>
> >>> TRIED. TESTED. TRUSTED.
> >>> phone: +972-9-7692018
> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>
> >> ___
> >> Devel mailing list -- devel@ovirt.org
> >> To unsubscribe send an email to devel-le...@ovirt.org
> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> >> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> >> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
> >
> > ___
> > Devel mailing list -- devel@ovirt.org
> > To unsubscribe send an email to devel-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54EQ2EGC5K25IJ47N3GWSHM/
>

___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-24 Thread Ahmad Khiet 
patched 4.3!

On Sun, Mar 24, 2019 at 9:06 AM Eitan Raviv  wrote:

> After some off line discussions it seems that the change that should be
> implemented in order to solve the original problem (remove host fails due
> to disconnect storage in progress) is to leave the host in 'preparing for
> maintenance' until all relevant storage operations are completed.
>
> On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg  wrote:
>
>> Unfortunately, the network suite is still failing on
>>
>> Cannot edit Host. Related operation is currently in progress. Please try
>> again later.
>>
>> Can you check if that's the same issue? Did you revert from 4.3 too?
>>
>> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
>>
>>
>>
>> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik,  wrote:
>>
>>> The patch was reverted on Thursday
>>>
>>> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg 
>>> wrote:
>>>
 I was told that intervening in the host state machine is delicate, but
 I think that this is the only correct approach.

 Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
 a third week with this constant failure.


 On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv  wrote:
 >
 > I am not sure that locking both groups would be sufficient, because
 there is still a chance that the removeNetowrks request will start and
 acquire the lock before the DisconnectStorage operation starts.
 > So probably the correct and full proof solution is to not move the
 host to maintenance until all related storage ops terminate.
 >
 >
 > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi <
 stira...@redhat.com> wrote:
 >>
 >>
 >>
 >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:
 >>>
 >>> Not sure if all the same issue, but seems to failing around the
 same time:
 >>>
 >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr
 - #825 57 min  integ-tests
 >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043
 12 hr - #1045 51 min  integ-tests
 >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11
 24 sec  integ-tests
 >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11
 10 min  integ-tests
 >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
 #871 10 hr - #873 1 hr 36 min  integ-tests
 >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147
 11 hr - #1149 1 hr 26 min  integ-tests
 >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727
 12 hr - #729 1 hr 54 min  integ-tests
 >>
 >>
 >> It's a NullPointerException on engine side:
 >> I opened a bug here:
 https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on
 POST
 >>
 >>
 >>>
 >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12
 hr - #929 45 min  integ-tests
 >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
 #187 10 hr - #190 45 min  integ-tests
 >>>
 >>> Links to jobs can be found here: https://jenkins.ovirt.org/
 >>>
 >>> --
 >>>
 >>> Eyal edri
 >>>
 >>>
 >>> MANAGER
 >>>
 >>> RHV/CNV DevOps
 >>>
 >>> EMEA VIRTUALIZATION R
 >>>
 >>>
 >>> Red Hat EMEA
 >>>
 >>> TRIED. TESTED. TRUSTED.
 >>> phone: +972-9-7692018
 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
 >>
 >> ___
 >> Devel mailing list -- devel@ovirt.org
 >> To unsubscribe send an email to devel-le...@ovirt.org
 >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
 >> oVirt Code of Conduct:
 https://www.ovirt.org/community/about/community-guidelines/
 >> List Archives:
 https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
 >
 > ___
 > Devel mailing list -- devel@ovirt.org
 > To unsubscribe send an email to devel-le...@ovirt.org
 > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
 > oVirt Code of Conduct:
 https://www.ovirt.org/community/about/community-guidelines/
 > List Archives:
 https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54EQ2EGC5K25IJ47N3GWSHM/

>>>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/UXZL5HUMSEVOKOXCBQ6H3KT62PJHINVQ/

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-24 Thread Eitan Raviv 
After some off line discussions it seems that the change that should be
implemented in order to solve the original problem (remove host fails due
to disconnect storage in progress) is to leave the host in 'preparing for
maintenance' until all relevant storage operations are completed.

On Sat, Mar 23, 2019 at 11:36 PM Dan Kenigsberg  wrote:

> Unfortunately, the network suite is still failing on
>
> Cannot edit Host. Related operation is currently in progress. Please try
> again later.
>
> Can you check if that's the same issue? Did you revert from 4.3 too?
>
> http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/
>
>
>
> On Sat, 23 Mar 2019, 21:27 Benny Zlotnik,  wrote:
>
>> The patch was reverted on Thursday
>>
>> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg  wrote:
>>
>>> I was told that intervening in the host state machine is delicate, but
>>> I think that this is the only correct approach.
>>>
>>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
>>> a third week with this constant failure.
>>>
>>>
>>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv  wrote:
>>> >
>>> > I am not sure that locking both groups would be sufficient, because
>>> there is still a chance that the removeNetowrks request will start and
>>> acquire the lock before the DisconnectStorage operation starts.
>>> > So probably the correct and full proof solution is to not move the
>>> host to maintenance until all related storage ops terminate.
>>> >
>>> >
>>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi 
>>> wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:
>>> >>>
>>> >>> Not sure if all the same issue, but seems to failing around the same
>>> time:
>>> >>>
>>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr -
>>> #825 57 min  integ-tests
>>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12
>>> hr - #1045 51 min  integ-tests
>>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24
>>> sec  integ-tests
>>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10
>>> min  integ-tests
>>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
>>> #871 10 hr - #873 1 hr 36 min  integ-tests
>>> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11
>>> hr - #1149 1 hr 26 min  integ-tests
>>> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727
>>> 12 hr - #729 1 hr 54 min  integ-tests
>>> >>
>>> >>
>>> >> It's a NullPointerException on engine side:
>>> >> I opened a bug here:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on POST
>>> >>
>>> >>
>>> >>>
>>> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12
>>> hr - #929 45 min  integ-tests
>>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
>>> #187 10 hr - #190 45 min  integ-tests
>>> >>>
>>> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>> >>>
>>> >>> --
>>> >>>
>>> >>> Eyal edri
>>> >>>
>>> >>>
>>> >>> MANAGER
>>> >>>
>>> >>> RHV/CNV DevOps
>>> >>>
>>> >>> EMEA VIRTUALIZATION R
>>> >>>
>>> >>>
>>> >>> Red Hat EMEA
>>> >>>
>>> >>> TRIED. TESTED. TRUSTED.
>>> >>> phone: +972-9-7692018
>>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>> >>
>>> >> ___
>>> >> Devel mailing list -- devel@ovirt.org
>>> >> To unsubscribe send an email to devel-le...@ovirt.org
>>> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> >> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> >> List Archives:
>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
>>> >
>>> > ___
>>> > Devel mailing list -- devel@ovirt.org
>>> > To unsubscribe send an email to devel-le...@ovirt.org
>>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> > oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> > List Archives:
>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54EQ2EGC5K25IJ47N3GWSHM/
>>>
>>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/TPXBV4O4VT34FLX5XB4L67DX3CABKFSD/

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-23 Thread Dan Kenigsberg 
Unfortunately, the network suite is still failing on

Cannot edit Host. Related operation is currently in progress. Please try
again later.

Can you check if that's the same issue? Did you revert from 4.3 too?

http://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.3/19/



On Sat, 23 Mar 2019, 21:27 Benny Zlotnik,  wrote:

> The patch was reverted on Thursday
>
> On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg  wrote:
>
>> I was told that intervening in the host state machine is delicate, but
>> I think that this is the only correct approach.
>>
>> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
>> a third week with this constant failure.
>>
>>
>> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv  wrote:
>> >
>> > I am not sure that locking both groups would be sufficient, because
>> there is still a chance that the removeNetowrks request will start and
>> acquire the lock before the DisconnectStorage operation starts.
>> > So probably the correct and full proof solution is to not move the host
>> to maintenance until all related storage ops terminate.
>> >
>> >
>> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi 
>> wrote:
>> >>
>> >>
>> >>
>> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:
>> >>>
>> >>> Not sure if all the same issue, but seems to failing around the same
>> time:
>> >>>
>> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr -
>> #825 57 min  integ-tests
>> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12
>> hr - #1045 51 min  integ-tests
>> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24
>> sec  integ-tests
>> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10
>> min  integ-tests
>> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr -
>> #871 10 hr - #873 1 hr 36 min  integ-tests
>> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11
>> hr - #1149 1 hr 26 min  integ-tests
>> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12
>> hr - #729 1 hr 54 min  integ-tests
>> >>
>> >>
>> >> It's a NullPointerException on engine side:
>> >> I opened a bug here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on POST
>> >>
>> >>
>> >>>
>> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr
>> - #929 45 min  integ-tests
>> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
>> #187 10 hr - #190 45 min  integ-tests
>> >>>
>> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
>> >>>
>> >>> --
>> >>>
>> >>> Eyal edri
>> >>>
>> >>>
>> >>> MANAGER
>> >>>
>> >>> RHV/CNV DevOps
>> >>>
>> >>> EMEA VIRTUALIZATION R
>> >>>
>> >>>
>> >>> Red Hat EMEA
>> >>>
>> >>> TRIED. TESTED. TRUSTED.
>> >>> phone: +972-9-7692018
>> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> >>
>> >> ___
>> >> Devel mailing list -- devel@ovirt.org
>> >> To unsubscribe send an email to devel-le...@ovirt.org
>> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> >> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> >> List Archives:
>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
>> >
>> > ___
>> > Devel mailing list -- devel@ovirt.org
>> > To unsubscribe send an email to devel-le...@ovirt.org
>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> > oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> > List Archives:
>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54EQ2EGC5K25IJ47N3GWSHM/
>>
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Z7BLMZXU3T463N6L36YFUKXCOULLFHXK/

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-23 Thread Benny Zlotnik 
The patch was reverted on Thursday

On Sat, Mar 23, 2019 at 8:48 PM Dan Kenigsberg  wrote:

> I was told that intervening in the host state machine is delicate, but
> I think that this is the only correct approach.
>
> Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
> a third week with this constant failure.
>
>
> On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv  wrote:
> >
> > I am not sure that locking both groups would be sufficient, because
> there is still a chance that the removeNetowrks request will start and
> acquire the lock before the DisconnectStorage operation starts.
> > So probably the correct and full proof solution is to not move the host
> to maintenance until all related storage ops terminate.
> >
> >
> > On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi 
> wrote:
> >>
> >>
> >>
> >> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:
> >>>
> >>> Not sure if all the same issue, but seems to failing around the same
> time:
> >>>
> >>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr -
> #825 57 min  integ-tests
> >>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12
> hr - #1045 51 min  integ-tests
> >>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24
> sec  integ-tests
> >>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10
> min  integ-tests
> >>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871
> 10 hr - #873 1 hr 36 min  integ-tests
> >>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11
> hr - #1149 1 hr 26 min  integ-tests
> >>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12
> hr - #729 1 hr 54 min  integ-tests
> >>
> >>
> >> It's a NullPointerException on engine side:
> >> I opened a bug here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which is not on POST
> >>
> >>
> >>>
> >>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr
> - #929 45 min  integ-tests
> >>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr -
> #187 10 hr - #190 45 min  integ-tests
> >>>
> >>> Links to jobs can be found here: https://jenkins.ovirt.org/
> >>>
> >>> --
> >>>
> >>> Eyal edri
> >>>
> >>>
> >>> MANAGER
> >>>
> >>> RHV/CNV DevOps
> >>>
> >>> EMEA VIRTUALIZATION R
> >>>
> >>>
> >>> Red Hat EMEA
> >>>
> >>> TRIED. TESTED. TRUSTED.
> >>> phone: +972-9-7692018
> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> >>
> >> ___
> >> Devel mailing list -- devel@ovirt.org
> >> To unsubscribe send an email to devel-le...@ovirt.org
> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> >> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> >> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
> >
> > ___
> > Devel mailing list -- devel@ovirt.org
> > To unsubscribe send an email to devel-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54EQ2EGC5K25IJ47N3GWSHM/
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CWXKEVEU7M6F25AG46OHOHADPFXHUYB6/

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-23 Thread Dan Kenigsberg 
I was told that intervening in the host state machine is delicate, but
I think that this is the only correct approach.

Benny, Ahmad, Tal: do you have a plan to resolve this? We are entering
a third week with this constant failure.


On Wed, Mar 20, 2019 at 2:42 PM Eitan Raviv  wrote:
>
> I am not sure that locking both groups would be sufficient, because there is 
> still a chance that the removeNetowrks request will start and acquire the 
> lock before the DisconnectStorage operation starts.
> So probably the correct and full proof solution is to not move the host to 
> maintenance until all related storage ops terminate.
>
>
> On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi  wrote:
>>
>>
>>
>> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:
>>>
>>> Not sure if all the same issue, but seems to failing around the same time:
>>>
>>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 
>>> 57 min  integ-tests
>>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr - 
>>> #1045 51 min  integ-tests
>>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec  
>>> integ-tests
>>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min  
>>> integ-tests
>>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10 
>>> hr - #873 1 hr 36 min  integ-tests
>>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr - 
>>> #1149 1 hr 26 min  integ-tests
>>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr - 
>>> #729 1 hr 54 min  integ-tests
>>
>>
>> It's a NullPointerException on engine side:
>> I opened a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1690159 
>> which is not on POST
>>
>>
>>>
>>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr - 
>>> #929 45 min  integ-tests
>>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10 
>>> hr - #190 45 min  integ-tests
>>>
>>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> MANAGER
>>>
>>> RHV/CNV DevOps
>>>
>>> EMEA VIRTUALIZATION R
>>>
>>>
>>> Red Hat EMEA
>>>
>>> TRIED. TESTED. TRUSTED.
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>> ___
>> Devel mailing list -- devel@ovirt.org
>> To unsubscribe send an email to devel-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct: 
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives: 
>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
>
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54EQ2EGC5K25IJ47N3GWSHM/
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MTARNSFLX2GXBGSVJQ2COHWWRWGTQEU5/

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-20 Thread Eitan Raviv 
I am not sure that locking both groups would be sufficient, because there
is still a chance that the removeNetowrks request will start and acquire
the lock before the DisconnectStorage operation starts.
So probably the correct and full proof solution is to not move the host to
maintenance until all related storage ops terminate.


On Wed, Mar 20, 2019 at 2:07 PM Simone Tiraboschi 
wrote:

>
>
> On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:
>
>> Not sure if all the same issue, but seems to failing around the same time:
>>
>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
>> min  integ-tests
>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr
>> - #1045 51 min  integ-tests
>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec
>>  integ-tests
>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min
>>  integ-tests
>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
>> hr - #873 1 hr 36 min  integ-tests
>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr
>> - #1149 1 hr 26 min  integ-tests
>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
>> - #729 1 hr 54 min  integ-tests
>>
>
> It's a NullPointerException on engine side:
> I opened a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which
> is not on POST
>
>
>
>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr -
>> #929 45 min  integ-tests
>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
>> hr - #190 45 min  integ-tests
>>
>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>
>> --
>>
>> Eyal edri
>>
>>
>> MANAGER
>>
>> RHV/CNV DevOps
>>
>> EMEA VIRTUALIZATION R
>>
>>
>> Red Hat EMEA 
>>  TRIED. TESTED. TRUSTED. 
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3XP6WDOAL54EQ2EGC5K25IJ47N3GWSHM/

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-20 Thread Simone Tiraboschi 
On Sun, Mar 17, 2019 at 3:04 PM Eyal Edri  wrote:

> Not sure if all the same issue, but seems to failing around the same time:
>
>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
> min  integ-tests
>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr -
> #1045 51 min  integ-tests
>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec 
> integ-tests
>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min 
> integ-tests
>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
> hr - #873 1 hr 36 min  integ-tests
>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr -
> #1149 1 hr 26 min  integ-tests
>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
> - #729 1 hr 54 min  integ-tests
>

It's a NullPointerException on engine side:
I opened a bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1690159 which
is not on POST



>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr -
> #929 45 min  integ-tests
>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
> hr - #190 45 min  integ-tests
>
> Links to jobs can be found here: https://jenkins.ovirt.org/
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV/CNV DevOps
>
> EMEA VIRTUALIZATION R
>
>
> Red Hat EMEA 
>  TRIED. TESTED. TRUSTED. 
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CQP4YDTMT53J5JKZQLYSYOELIVCHKVKE/

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-20 Thread Benny Zlotnik 
We should probably lock both groups in
VdsEventListener#processStorageOnVdsInactive or in RemoveVdsCommand
Ahmad, please evaluate and adjust

On Wed, Mar 20, 2019 at 11:25 AM Eitan Raviv  wrote:

> At least for the network suite these failures are due to the fact that 
> DisconnectStoragePoolVDSCommand
> is not finished and a change_cluster request is issued by the test but
> revoked by engine which fails the test setup.
>
> Ahmed,
> Can you please have a look at the below engine log [2]- currently
> change_cluster requests from the test are being revoked by engine. I
> believe this is because the locking group of DisconnectStoragePoolVDSCommand
> was moved to VDS in patch [1]
> Is this a desired outcome and should the tests be changed to reflect it
> (wait for which status notification from engine?)?
>
> Thanks
>
> P.S. please note that due to this patch 
> LockingGroup.VDS_POOL_AND_STORAGE_CONNECTIONS
> has only one consumer, which means it does not lock against anything. Is
> this a desired outcome of the patch?
> [1] https://gerrit.ovirt.org/#/c/98405
> [2]
> Network-suite-master.tests.test_mac_pools.test_mac_pools_in_different_clusters_dont_overlap
> Error Message
>
> test setup failure
> Stacktrace
>
> default_data_center =  0x7fbe88aa8790>
> host_1_up = 
> cluster_0 = 
>
>@pytest.fixture(scope='module')
>def host_1_in_cluster_0(default_data_center, host_1_up, cluster_0):
>current_cluster = host_1_up.get_cluster()
>
> >   host_1_up.change_cluster(cluster_0)
>
> network-suite-master/tests/test_mac_pools.py:49
>
>
> 2019-03-17 07:16:40,493-04 INFO
>  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] START,
> DisconnectStoragePoolVDSCommand(HostName =
> lago-network-suite-master-host-1,
> DisconnectStoragePoolVDSCommandParameters:{hostId='2ab16b7a-fd38-41c5-98be-c8db1fdece42',
> storagePoolId='239d1be2-48a4-11e9-bec6-5452c0a8c902', vds_spm_id='2'}), log
> id: 44ad4438
>
> 2019-03-17 07:16:43,635-04 INFO
>  [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
> [eded9adc-b780-4640-ba5e-ec989f620d57] Failed to Acquire Lock to object
> 'EngineLock:{exclusiveLocks='[2ab16b7a-fd38-41c5-98be-c8db1fdece42=VDS]',
> sharedLocks=''}'
> 2019-03-17 07:16:43,635-04 WARN
>  [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
> [eded9adc-b780-4640-ba5e-ec989f620d57] Validation of action
> 'ChangeVDSCluster' failed for user admin@internal-authz. Reasons:
> VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
> 2019-03-17 07:16:43,636-04 DEBUG
> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
> (default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] method: runAction,
> params: [ChangeVDSCluster,
> ChangeVDSClusterParameters:{commandId='00522716-6557-4da0-8d02-802fbf402c24',
> user='null', commandType='Unknown'}], timeElapsed: 18ms
> 2019-03-17 07:16:43,637-04 ERROR
> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
> task-2) [] Operation Failed: [Cannot edit Host. Related operation is
> currently in progress. Please try again later.]
>
>
> 2019-03-17 07:16:44,575-04 INFO
>  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] FINISH,
> DisconnectStoragePoolVDSCommand, return: , log id: 44ad4438
>
> On Sun, Mar 17, 2019 at 4:05 PM Eyal Edri  wrote:
>
>> Not sure if all the same issue, but seems to failing around the same time:
>>
>>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
>> min  integ-tests
>>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr
>> - #1045 51 min  integ-tests
>>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec
>>  integ-tests
>>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min
>>  integ-tests
>>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
>> hr - #873 1 hr 36 min  integ-tests
>>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr
>> - #1149 1 hr 26 min  integ-tests
>>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
>> - #729 1 hr 54 min  integ-tests
>>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr -
>> #929 45 min  integ-tests
>>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
>> hr - #190 45 min  integ-tests
>>
>> Links to jobs can be found here: https://jenkins.ovirt.org/
>>
>> --
>>
>> Eyal edri
>>
>>
>> MANAGER
>>
>> RHV/CNV DevOps
>>
>> EMEA VIRTUALIZATION R
>>
>>
>> Red Hat EMEA 
>>  TRIED. TESTED. TRUSTED. 
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>> ___
>> Devel mailing list -- devel@ovirt.org
>> To unsubscribe send an email to

[ovirt-devel] Re: FYI - Many nightly suites, including Network, HE and HC are failing since 2-3 days ago

2019-03-20 Thread Eitan Raviv 
At least for the network suite these failures are due to the fact that
DisconnectStoragePoolVDSCommand
is not finished and a change_cluster request is issued by the test but
revoked by engine which fails the test setup.

Ahmed,
Can you please have a look at the below engine log [2]- currently
change_cluster requests from the test are being revoked by engine. I
believe this is because the locking group of DisconnectStoragePoolVDSCommand
was moved to VDS in patch [1]
Is this a desired outcome and should the tests be changed to reflect it
(wait for which status notification from engine?)?

Thanks

P.S. please note that due to this patch
LockingGroup.VDS_POOL_AND_STORAGE_CONNECTIONS
has only one consumer, which means it does not lock against anything. Is
this a desired outcome of the patch?
[1] https://gerrit.ovirt.org/#/c/98405
[2]
Network-suite-master.tests.test_mac_pools.test_mac_pools_in_different_clusters_dont_overlap
Error Message

test setup failure
Stacktrace

default_data_center = 
host_1_up = 
cluster_0 = 

   @pytest.fixture(scope='module')
   def host_1_in_cluster_0(default_data_center, host_1_up, cluster_0):
   current_cluster = host_1_up.get_cluster()

>   host_1_up.change_cluster(cluster_0)

network-suite-master/tests/test_mac_pools.py:49


2019-03-17 07:16:40,493-04 INFO
 [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-65) [] START,
DisconnectStoragePoolVDSCommand(HostName =
lago-network-suite-master-host-1,
DisconnectStoragePoolVDSCommandParameters:{hostId='2ab16b7a-fd38-41c5-98be-c8db1fdece42',
storagePoolId='239d1be2-48a4-11e9-bec6-5452c0a8c902', vds_spm_id='2'}), log
id: 44ad4438

2019-03-17 07:16:43,635-04 INFO
 [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
[eded9adc-b780-4640-ba5e-ec989f620d57] Failed to Acquire Lock to object
'EngineLock:{exclusiveLocks='[2ab16b7a-fd38-41c5-98be-c8db1fdece42=VDS]',
sharedLocks=''}'
2019-03-17 07:16:43,635-04 WARN
 [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (default task-2)
[eded9adc-b780-4640-ba5e-ec989f620d57] Validation of action
'ChangeVDSCluster' failed for user admin@internal-authz. Reasons:
VAR__ACTION__UPDATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
2019-03-17 07:16:43,636-04 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-2) [eded9adc-b780-4640-ba5e-ec989f620d57] method: runAction,
params: [ChangeVDSCluster,
ChangeVDSClusterParameters:{commandId='00522716-6557-4da0-8d02-802fbf402c24',
user='null', commandType='Unknown'}], timeElapsed: 18ms
2019-03-17 07:16:43,637-04 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-2) [] Operation Failed: [Cannot edit Host. Related operation is
currently in progress. Please try again later.]


2019-03-17 07:16:44,575-04 INFO
 [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-65) [] FINISH,
DisconnectStoragePoolVDSCommand, return: , log id: 44ad4438

On Sun, Mar 17, 2019 at 4:05 PM Eyal Edri  wrote:

> Not sure if all the same issue, but seems to failing around the same time:
>
>   ovirt-system-tests_hc-basic-suite-4.2 1 day 12 hr - #824 12 hr - #825 57
> min  integ-tests
>   ovirt-system-tests_hc-basic-suite-master 2 days 12 hr - #1043 12 hr -
> #1045 51 min  integ-tests
>   ovirt-system-tests_he-basic-ansible-suite-4.3 N/A 11 hr - #11 24 sec 
> integ-tests
>   ovirt-system-tests_he-basic-ipv6-suite-master N/A 12 hr - #11 10 min 
> integ-tests
>   ovirt-system-tests_he-basic-iscsi-suite-master 2 days 10 hr - #871 10
> hr - #873 1 hr 36 min  integ-tests
>   ovirt-system-tests_he-basic-suite-master 2 days 11 hr - #1147 11 hr -
> #1149 1 hr 26 min  integ-tests
>   ovirt-system-tests_he-node-ng-suite-master 2 days 12 hr - #727 12 hr
> - #729 1 hr 54 min  integ-tests
>   ovirt-system-tests_network-suite-master 3 days 12 hr - #926 12 hr -
> #929 45 min  integ-tests
>   ovirt-system-tests_openshift-on-ovirt-suite-4.2 3 days 10 hr - #187 10
> hr - #190 45 min  integ-tests
>
> Links to jobs can be found here: https://jenkins.ovirt.org/
>
> --
>
> Eyal edri
>
>
> MANAGER
>
> RHV/CNV DevOps
>
> EMEA VIRTUALIZATION R
>
>
> Red Hat EMEA 
>  TRIED. TESTED. TRUSTED. 
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/I75ZI4LYD3IKZCSZEHNU7IR6K7DDRU3K/
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to