Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288

2016-09-20 Thread Nir Soffer
On Mon, Sep 19, 2016 at 8:49 PM, Eyal Edri  wrote:
> Is this the usual Sanlock issue?

"Usual" sanlock issue is alarming. I don't know about any issues with Sanlock.

Do we have a bug about this?

>
> Error Message
>
> status: 400
> reason: Bad Request
> detail: Cannot add VM: Storage Domain cannot be accessed.
> -Please check that at least one Host is operational and Data Center state is
> up.
>
> Stacktrace
>
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/unittest/case.py", line 367, in run
> testMethod()
>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
> self.test(*self.arg)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in
> wrapped_test
> return test()
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in
> wrapper
> return func(get_test_prefix(), *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in
> wrapper
> return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
>   File
> "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py",
> line 78, in add_vm_blank
> api.vms.add(vm_params)
>   File
> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line
> 35701, in add
> headers={"Correlation-Id":correlation_id, "Expect":expect}
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",
> line 79, in add
> return self.request('POST', url, body, headers, cls=cls)
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",
> line 122, in request
> persistent_auth=self.__persistent_auth
>   File
> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
> line 79, in do_request
> persistent_auth)
>   File
> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
> line 162, in __do_request
> raise errors.RequestError(response_code, response_reason, response_body)
> RequestError:
> status: 400
> reason: Bad Request
> detail: Cannot add VM: Storage Domain cannot be accessed.
> -Please check that at least one Host is operational and Data Center state is
> up.
>
>
> On Mon, Sep 19, 2016 at 8:22 PM,  wrote:
>>
>> See 
>>
>> ___
>> Infra mailing list
>> Infra@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>
>
>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
> ___
> Infra mailing list
> Infra@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra


Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288

2016-09-20 Thread Simone Tiraboschi
On Tue, Sep 20, 2016 at 10:44 AM, Lev Veyde <lve...@redhat.com> wrote:

> Hi Eyal,
>
> I'm not 100% sure about that one.
> Checking the logs I see that the first host has been setup without any
> issues, yet setting up the second host gives:
>
> 17:16:31 [ ERROR ] Failed to execute stage 'Closing up': HTTP Error 500:
> Internal Server Error
> 17:16:31 [ INFO  ] Stage: Clean up
> 17:16:31 [ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-
> setup/answers/answers-20160919131631.conf'
> 17:16:31 [ INFO  ] Stage: Pre-termination
> 17:16:31 [ INFO  ] Stage: Termination
> 17:16:31 [ ERROR ] Hosted Engine deployment failed: this system is not
> reliable, please check the issue,fix and redeploy
> 17:16:31   Log file is located at /var/log/ovirt-hosted-engine-
> setup/ovirt-hosted-engine-setup-20160919131556-qufq5z.log
>
> However as we don't have the ovirt-engine and ovirt-hosted-engine-setup
> logs we can't be 100% certain.
>

We have them,
they are in
http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/artifact/*zip*/archive.zip
under /archive/exported-artifacts/test_logs/he_basic_suite_4.0/
post-002_bootstrap.py/


>
> Thanks in advance,
> Lev Veyde.
>
> - Original Message -
> From: "Eyal Edri" <ee...@redhat.com>
> To: "Lev Veyde" <lve...@redhat.com>
> Cc: "Sandro Bonazzola" <sbona...@redhat.com>, "infra" <infra@ovirt.org>,
> "Simone Tiraboschi" <stira...@redhat.com>
> Sent: Monday, September 19, 2016 8:49:59 PM
> Subject: Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288
>
> Is this the usual Sanlock issue?
>
> Error Message
>
> status: 400
> reason: Bad Request
> detail: Cannot add VM: Storage Domain cannot be accessed.
> -Please check that at least one Host is operational and Data Center state
> is up.
>
> Stacktrace
>
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/unittest/case.py", line 367, in run
> testMethod()
>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in
> runTest
> self.test(*self.arg)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
> 120, in wrapped_test
> return test()
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
> 52, in wrapper
> return func(get_test_prefix(), *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
> 61, in wrapper
> return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
>   File "/home/jenkins/workspace/ovirt_4.0_he-system-tests/
> ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py",
> line 78, in add_vm_blank
> api.vms.add(vm_params)
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/
> infrastructure/brokers.py",
> line 35701, in add
> headers={"Correlation-Id":correlation_id, "Expect":expect}
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/
> infrastructure/proxy.py",
> line 79, in add
> return self.request('POST', url, body, headers, cls=cls)
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/
> infrastructure/proxy.py",
> line 122, in request
> persistent_auth=self.__persistent_auth
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/
> connectionspool.py",
> line 79, in do_request
> persistent_auth)
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/
> connectionspool.py",
> line 162, in __do_request
> raise errors.RequestError(response_code, response_reason,
> response_body)
> RequestError:
> status: 400
> reason: Bad Request
> detail: Cannot add VM: Storage Domain cannot be accessed.
> -Please check that at least one Host is operational and Data Center state
> is up.
>
>
> On Mon, Sep 19, 2016 at 8:22 PM, <jenk...@jenkins.phx.ovirt.org> wrote:
>
> > See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
> >
> > ___
> > Infra mailing list
> > Infra@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
>
>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra


Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288

2016-09-20 Thread Lev Veyde
Hi Eyal,

I'm not 100% sure about that one.
Checking the logs I see that the first host has been setup without any issues, 
yet setting up the second host gives:

17:16:31 [ ERROR ] Failed to execute stage 'Closing up': HTTP Error 500: 
Internal Server Error
17:16:31 [ INFO  ] Stage: Clean up
17:16:31 [ INFO  ] Generating answer file 
'/var/lib/ovirt-hosted-engine-setup/answers/answers-20160919131631.conf'
17:16:31 [ INFO  ] Stage: Pre-termination
17:16:31 [ INFO  ] Stage: Termination
17:16:31 [ ERROR ] Hosted Engine deployment failed: this system is not 
reliable, please check the issue,fix and redeploy
17:16:31   Log file is located at 
/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160919131556-qufq5z.log

However as we don't have the ovirt-engine and ovirt-hosted-engine-setup logs we 
can't be 100% certain.

Thanks in advance,
Lev Veyde.

- Original Message -
From: "Eyal Edri" <ee...@redhat.com>
To: "Lev Veyde" <lve...@redhat.com>
Cc: "Sandro Bonazzola" <sbona...@redhat.com>, "infra" <infra@ovirt.org>, 
"Simone Tiraboschi" <stira...@redhat.com>
Sent: Monday, September 19, 2016 8:49:59 PM
Subject: Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288

Is this the usual Sanlock issue?

Error Message

status: 400
reason: Bad Request
detail: Cannot add VM: Storage Domain cannot be accessed.
-Please check that at least one Host is operational and Data Center state is up.

Stacktrace

Traceback (most recent call last):
  File "/usr/lib64/python2.7/unittest/case.py", line 367, in run
testMethod()
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
120, in wrapped_test
return test()
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
52, in wrapper
return func(get_test_prefix(), *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
61, in wrapper
return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
  File 
"/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py",
line 78, in add_vm_blank
api.vms.add(vm_params)
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py",
line 35701, in add
headers={"Correlation-Id":correlation_id, "Expect":expect}
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",
line 79, in add
return self.request('POST', url, body, headers, cls=cls)
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",
line 122, in request
persistent_auth=self.__persistent_auth
  File 
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
line 79, in do_request
persistent_auth)
  File 
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body)
RequestError:
status: 400
reason: Bad Request
detail: Cannot add VM: Storage Domain cannot be accessed.
-Please check that at least one Host is operational and Data Center state is up.


On Mon, Sep 19, 2016 at 8:22 PM, <jenk...@jenkins.phx.ovirt.org> wrote:

> See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/288/changes>
>
> ___
> Infra mailing list
> Infra@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>
>


-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra


Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288

2016-09-20 Thread Eyal Edri
I really want to add this job to the experimental flow to catch regressions
early, but we have to iron out any false positives or errors we're seeing
before that.
Until now I was only aware of the Sanlock issue, which should be resolved
soon once its available in centos.

Are we aware of other known issues in the tests?

On Tue, Sep 20, 2016 at 11:22 AM, Simone Tiraboschi 
wrote:

>
>
> On Mon, Sep 19, 2016 at 7:49 PM, Eyal Edri  wrote:
>
>> Is this the usual Sanlock issue?
>>
>
> Not that sure, however build 289 come back to stable by itself.
>
>
>
>>
>> Error Message
>>
>> status: 400
>> reason: Bad Request
>> detail: Cannot add VM: Storage Domain cannot be accessed.
>> -Please check that at least one Host is operational and Data Center state is 
>> up.
>>
>> Stacktrace
>>
>> Traceback (most recent call last):
>>   File "/usr/lib64/python2.7/unittest/case.py", line 367, in run
>> testMethod()
>>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
>> self.test(*self.arg)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in 
>> wrapped_test
>> return test()
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in 
>> wrapper
>> return func(get_test_prefix(), *args, **kwargs)
>>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in 
>> wrapper
>> return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
>>   File 
>> "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py",
>>  line 78, in add_vm_blank
>> api.vms.add(vm_params)
>>   File 
>> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 
>> 35701, in add
>> headers={"Correlation-Id":correlation_id, "Expect":expect}
>>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", 
>> line 79, in add
>> return self.request('POST', url, body, headers, cls=cls)
>>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", 
>> line 122, in request
>> persistent_auth=self.__persistent_auth
>>   File 
>> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
>>  line 79, in do_request
>> persistent_auth)
>>   File 
>> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
>>  line 162, in __do_request
>> raise errors.RequestError(response_code, response_reason, response_body)
>> RequestError:
>> status: 400
>> reason: Bad Request
>> detail: Cannot add VM: Storage Domain cannot be accessed.
>> -Please check that at least one Host is operational and Data Center state is 
>> up.
>>
>>
>> On Mon, Sep 19, 2016 at 8:22 PM,  wrote:
>>
>>> See 
>>>
>>> ___
>>> Infra mailing list
>>> Infra@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>
>>>
>>>
>>
>>
>> --
>> Eyal Edri
>> Associate Manager
>> RHV DevOps
>> EMEA ENG Virtualization R
>> Red Hat Israel
>>
>> phone: +972-9-7692018
>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>
>


-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra


Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288

2016-09-20 Thread Simone Tiraboschi
On Mon, Sep 19, 2016 at 7:49 PM, Eyal Edri  wrote:

> Is this the usual Sanlock issue?
>

Not that sure, however build 289 come back to stable by itself.



>
> Error Message
>
> status: 400
> reason: Bad Request
> detail: Cannot add VM: Storage Domain cannot be accessed.
> -Please check that at least one Host is operational and Data Center state is 
> up.
>
> Stacktrace
>
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/unittest/case.py", line 367, in run
> testMethod()
>   File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
> self.test(*self.arg)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 120, in 
> wrapped_test
> return test()
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 52, in 
> wrapper
> return func(get_test_prefix(), *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 61, in 
> wrapper
> return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
>   File 
> "/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py",
>  line 78, in add_vm_blank
> api.vms.add(vm_params)
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", 
> line 35701, in add
> headers={"Correlation-Id":correlation_id, "Expect":expect}
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", 
> line 79, in add
> return self.request('POST', url, body, headers, cls=cls)
>   File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", 
> line 122, in request
> persistent_auth=self.__persistent_auth
>   File 
> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
>  line 79, in do_request
> persistent_auth)
>   File 
> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
>  line 162, in __do_request
> raise errors.RequestError(response_code, response_reason, response_body)
> RequestError:
> status: 400
> reason: Bad Request
> detail: Cannot add VM: Storage Domain cannot be accessed.
> -Please check that at least one Host is operational and Data Center state is 
> up.
>
>
> On Mon, Sep 19, 2016 at 8:22 PM,  wrote:
>
>> See 
>>
>> ___
>> Infra mailing list
>> Infra@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>>
>
>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra


Re: Jenkins build became unstable: ovirt_4.0_he-system-tests #288

2016-09-19 Thread Eyal Edri
Is this the usual Sanlock issue?

Error Message

status: 400
reason: Bad Request
detail: Cannot add VM: Storage Domain cannot be accessed.
-Please check that at least one Host is operational and Data Center state is up.

Stacktrace

Traceback (most recent call last):
  File "/usr/lib64/python2.7/unittest/case.py", line 367, in run
testMethod()
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
120, in wrapped_test
return test()
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
52, in wrapper
return func(get_test_prefix(), *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line
61, in wrapper
return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs)
  File 
"/home/jenkins/workspace/ovirt_4.0_he-system-tests/ovirt-system-tests/he_basic_suite_4.0/test-scenarios/004_basic_sanity.py",
line 78, in add_vm_blank
api.vms.add(vm_params)
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py",
line 35701, in add
headers={"Correlation-Id":correlation_id, "Expect":expect}
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",
line 79, in add
return self.request('POST', url, body, headers, cls=cls)
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py",
line 122, in request
persistent_auth=self.__persistent_auth
  File 
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
line 79, in do_request
persistent_auth)
  File 
"/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py",
line 162, in __do_request
raise errors.RequestError(response_code, response_reason, response_body)
RequestError:
status: 400
reason: Bad Request
detail: Cannot add VM: Storage Domain cannot be accessed.
-Please check that at least one Host is operational and Data Center state is up.


On Mon, Sep 19, 2016 at 8:22 PM,  wrote:

> See 
>
> ___
> Infra mailing list
> Infra@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/infra
>
>
>


-- 
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R
Red Hat Israel

phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra