Re: [ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 (ovirt-engine) ] [ 15-11-2018 ] [ check_snapshot_with_memory ]
this has happened in the not so distant past:
[ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 (imgbased) ] [ 09-08-2018
] [ 004_basic_sanity.check_snapshot_with_memory ]
On Fri, Nov 16, 2018 at 3:54 PM Dafna Ron wrote:
> Other projects are passing so it cannot be ost
> we have ovirt-engine 4.2 that passed though:
> https://gerrit.ovirt.org/#/c/95447/
>
> logging a ticket to look into these failures.
>
> On Fri, Nov 16, 2018 at 1:30 PM Dafna Ron wrote:
>
>> I am seeing a second project to fail on this:
>>
>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-queue-tester/3495/
>>
>> I am not seeing any changes in OST merged in the last 2 days that could
>> have caused this and I am also not seeing any infrastructure commonality
>> (not running on same hosts).
>>
>> However, something has changed to cause the iscsi domain to be filled and
>> fail on run vm.
>>
>> Milan, can you please double check that this change is not causing any
>> leftovers in the iscsi domain?
>> https://gerrit.ovirt.org/#/c/95333/
>>
>> Thanks,
>> Dafna
>>
>>
>>
>> On Fri, Nov 16, 2018 at 11:32 AM Greg Sheremeta
>> wrote:
>>
>>> It's definitely not that patch - the patch is UI / GWT only and isn't
>>> tested outside of the selenium tests in 008. Also, this is a backport and
>>> we didn't see any issues on master.
>>>
>>> Greg
>>>
>>> On Fri, Nov 16, 2018, 6:24 AM Dafna Ron >>
Hi,
we have a failure on basic suite for test check_snapshot_with. _memory.
I am actually not seeing any reason the patch would cause that specific
issue but its consistently failing on this change.
I am seeing a vm memory saved to the iscsi domain but aside from that
nothing is failing before tto cause a cleanup issue or create low space on
the storage).
can some one please take a look to see if anything in the patch is
causing this?
Link and headline of suspected patches:
https://gerrit.ovirt.org/#/c/95436/1 -
webadmin: network operation in progress - sync host interfaces
Link to Job:
http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/
Link to all logs:
https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-4.2/post-004_basic_sanity.py/
(Relevant) error snippet from the log:
2018-11-15 08:19:33,409-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method:
runVdsCommand, params: [IsVmDuringInitiating
,
IsVmDuringInitiatingVDSCommandParameters:{vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
timeElapsed: 1ms
2018-11-15 08:19:33,430-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] EVENT_ID:
USER_FAILED_RUN_VM(54), Failed to run VM
vm0 due to a failed validation: [Cannot run VM. Low disk space on
Storage Domain iscsi.] (User: admin@internal-authz).
2018-11-15 08:19:33,430-05 WARN
[org.ovirt.engine.core.bll.RunVmCommand] (default task-5)
[dce28fd0-bebc-4aba-80d5-ffc081b1f591] Validation of action 'RunVm' failed
for user admin@internal-authz. Reasons: VAR__
ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName
iscsi
2018-11-15 08:19:33,431-05 INFO
[org.ovirt.engine.core.bll.RunVmCommand] (default task-5)
[dce28fd0-bebc-4aba-80d5-ffc081b1f591] Lock freed to object
'EngineLock:{exclusiveLocks='[9144eb88-8f0a-4e77-9d70-3b761e
48ecb4=VM]', sharedLocks=''}'
2018-11-15 08:19:33,438-05 DEBUG
[org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
(default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runAction,
params: [RunVm, RunVmParams:{comm
andId='69628467-56d3-4d93-aed1-a2807b7dbc45', user='null',
commandType='Unknown', vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
timeElapsed: 115ms
2018-11-15 08:19:33,441-05 ERROR
[org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
task-5) [] Operation Failed: [Cannot run VM. Low disk space on Storage
Domain iscsi.]
2018-11-15 08:19:34,044-05 DEBUG
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [] START,
GetAllVmStatsVDSCommand(HostName = lago-basic-su
ite-4-2-host-0,
VdsIdVDSCommandParametersBase:{hostId='063dfef9-bc9d-44c3-8ba3-8142e3eb129c'}),
log id: 65651e2b
2018-11-15 08:19:34,045-05 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [] SEND
destination:jms.topic.vdsm_requests
reply-to:jms.topic.vdsm_responses
content-length:103
Re: [ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 (ovirt-engine) ] [ 15-11-2018 ] [ check_snapshot_with_memory ]
Other projects are passing so it cannot be ost
we have ovirt-engine 4.2 that passed though:
https://gerrit.ovirt.org/#/c/95447/
logging a ticket to look into these failures.
On Fri, Nov 16, 2018 at 1:30 PM Dafna Ron wrote:
> I am seeing a second project to fail on this:
>
> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-queue-tester/3495/
>
> I am not seeing any changes in OST merged in the last 2 days that could
> have caused this and I am also not seeing any infrastructure commonality
> (not running on same hosts).
>
> However, something has changed to cause the iscsi domain to be filled and
> fail on run vm.
>
> Milan, can you please double check that this change is not causing any
> leftovers in the iscsi domain?
> https://gerrit.ovirt.org/#/c/95333/
>
> Thanks,
> Dafna
>
>
>
> On Fri, Nov 16, 2018 at 11:32 AM Greg Sheremeta
> wrote:
>
>> It's definitely not that patch - the patch is UI / GWT only and isn't
>> tested outside of the selenium tests in 008. Also, this is a backport and
>> we didn't see any issues on master.
>>
>> Greg
>>
>> On Fri, Nov 16, 2018, 6:24 AM Dafna Ron >
>>> Hi,
>>> we have a failure on basic suite for test check_snapshot_with. _memory.
>>>
>>> I am actually not seeing any reason the patch would cause that specific
>>> issue but its consistently failing on this change.
>>> I am seeing a vm memory saved to the iscsi domain but aside from that
>>> nothing is failing before tto cause a cleanup issue or create low space on
>>> the storage).
>>> can some one please take a look to see if anything in the patch is
>>> causing this?
>>>
>>>
>>> Link and headline of suspected patches:
>>>
>>> https://gerrit.ovirt.org/#/c/95436/1 -
>>> webadmin: network operation in progress - sync host interfaces
>>>
>>> Link to Job:
>>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/
>>>
>>> Link to all logs:
>>>
>>>
>>> https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-4.2/post-004_basic_sanity.py/
>>>
>>> (Relevant) error snippet from the log:
>>>
>>>
>>>
>>> 2018-11-15 08:19:33,409-05 DEBUG
>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
>>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method:
>>> runVdsCommand, params: [IsVmDuringInitiating
>>> ,
>>> IsVmDuringInitiatingVDSCommandParameters:{vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
>>> timeElapsed: 1ms
>>> 2018-11-15 08:19:33,430-05 ERROR
>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] EVENT_ID:
>>> USER_FAILED_RUN_VM(54), Failed to run VM
>>> vm0 due to a failed validation: [Cannot run VM. Low disk space on
>>> Storage Domain iscsi.] (User: admin@internal-authz).
>>> 2018-11-15 08:19:33,430-05 WARN
>>> [org.ovirt.engine.core.bll.RunVmCommand] (default task-5)
>>> [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Validation of action 'RunVm' failed
>>> for user admin@internal-authz. Reasons: VAR__
>>> ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName
>>> iscsi
>>> 2018-11-15 08:19:33,431-05 INFO
>>> [org.ovirt.engine.core.bll.RunVmCommand] (default task-5)
>>> [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Lock freed to object
>>> 'EngineLock:{exclusiveLocks='[9144eb88-8f0a-4e77-9d70-3b761e
>>> 48ecb4=VM]', sharedLocks=''}'
>>> 2018-11-15 08:19:33,438-05 DEBUG
>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
>>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runAction,
>>> params: [RunVm, RunVmParams:{comm
>>> andId='69628467-56d3-4d93-aed1-a2807b7dbc45', user='null',
>>> commandType='Unknown', vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
>>> timeElapsed: 115ms
>>> 2018-11-15 08:19:33,441-05 ERROR
>>> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
>>> task-5) [] Operation Failed: [Cannot run VM. Low disk space on Storage
>>> Domain iscsi.]
>>> 2018-11-15 08:19:34,044-05 DEBUG
>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
>>> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] START,
>>> GetAllVmStatsVDSCommand(HostName = lago-basic-su
>>> ite-4-2-host-0,
>>> VdsIdVDSCommandParametersBase:{hostId='063dfef9-bc9d-44c3-8ba3-8142e3eb129c'}),
>>> log id: 65651e2b
>>> 2018-11-15 08:19:34,045-05 DEBUG
>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
>>> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] SEND
>>> destination:jms.topic.vdsm_requests
>>> reply-to:jms.topic.vdsm_responses
>>> content-length:103
>>>
>>>
>>>
>>>
>>> ___
>>> Devel mailing list -- de...@ovirt.org
>>> To unsubscribe send an email to devel-le...@ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>>
Re: [ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 (ovirt-engine) ] [ 15-11-2018 ] [ check_snapshot_with_memory ]
I am seeing a second project to fail on this:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-queue-tester/3495/
I am not seeing any changes in OST merged in the last 2 days that could
have caused this and I am also not seeing any infrastructure commonality
(not running on same hosts).
However, something has changed to cause the iscsi domain to be filled and
fail on run vm.
Milan, can you please double check that this change is not causing any
leftovers in the iscsi domain?
https://gerrit.ovirt.org/#/c/95333/
Thanks,
Dafna
On Fri, Nov 16, 2018 at 11:32 AM Greg Sheremeta wrote:
> It's definitely not that patch - the patch is UI / GWT only and isn't
> tested outside of the selenium tests in 008. Also, this is a backport and
> we didn't see any issues on master.
>
> Greg
>
> On Fri, Nov 16, 2018, 6:24 AM Dafna Ron
>> Hi,
>> we have a failure on basic suite for test check_snapshot_with. _memory.
>>
>> I am actually not seeing any reason the patch would cause that specific
>> issue but its consistently failing on this change.
>> I am seeing a vm memory saved to the iscsi domain but aside from that
>> nothing is failing before tto cause a cleanup issue or create low space on
>> the storage).
>> can some one please take a look to see if anything in the patch is
>> causing this?
>>
>>
>> Link and headline of suspected patches:
>>
>> https://gerrit.ovirt.org/#/c/95436/1 -
>> webadmin: network operation in progress - sync host interfaces
>>
>> Link to Job:
>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/
>>
>> Link to all logs:
>>
>>
>> https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-4.2/post-004_basic_sanity.py/
>>
>> (Relevant) error snippet from the log:
>>
>>
>>
>> 2018-11-15 08:19:33,409-05 DEBUG
>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method:
>> runVdsCommand, params: [IsVmDuringInitiating
>> ,
>> IsVmDuringInitiatingVDSCommandParameters:{vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
>> timeElapsed: 1ms
>> 2018-11-15 08:19:33,430-05 ERROR
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] EVENT_ID:
>> USER_FAILED_RUN_VM(54), Failed to run VM
>> vm0 due to a failed validation: [Cannot run VM. Low disk space on Storage
>> Domain iscsi.] (User: admin@internal-authz).
>> 2018-11-15 08:19:33,430-05 WARN [org.ovirt.engine.core.bll.RunVmCommand]
>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Validation of
>> action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__
>> ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName
>> iscsi
>> 2018-11-15 08:19:33,431-05 INFO [org.ovirt.engine.core.bll.RunVmCommand]
>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Lock freed to
>> object 'EngineLock:{exclusiveLocks='[9144eb88-8f0a-4e77-9d70-3b761e
>> 48ecb4=VM]', sharedLocks=''}'
>> 2018-11-15 08:19:33,438-05 DEBUG
>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runAction,
>> params: [RunVm, RunVmParams:{comm
>> andId='69628467-56d3-4d93-aed1-a2807b7dbc45', user='null',
>> commandType='Unknown', vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
>> timeElapsed: 115ms
>> 2018-11-15 08:19:33,441-05 ERROR
>> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
>> task-5) [] Operation Failed: [Cannot run VM. Low disk space on Storage
>> Domain iscsi.]
>> 2018-11-15 08:19:34,044-05 DEBUG
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] START,
>> GetAllVmStatsVDSCommand(HostName = lago-basic-su
>> ite-4-2-host-0,
>> VdsIdVDSCommandParametersBase:{hostId='063dfef9-bc9d-44c3-8ba3-8142e3eb129c'}),
>> log id: 65651e2b
>> 2018-11-15 08:19:34,045-05 DEBUG
>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
>> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] SEND
>> destination:jms.topic.vdsm_requests
>> reply-to:jms.topic.vdsm_responses
>> content-length:103
>>
>>
>>
>>
>> ___
>> Devel mailing list -- de...@ovirt.org
>> To unsubscribe send an email to devel-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/ZIG4VQKIFFARDTDJ6B2JROENN2Z44B4I/
>>
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List
Re: [ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 (ovirt-engine) ] [ 15-11-2018 ] [ check_snapshot_with_memory ]
It's definitely not that patch - the patch is UI / GWT only and isn't
tested outside of the selenium tests in 008. Also, this is a backport and
we didn't see any issues on master.
Greg
On Fri, Nov 16, 2018, 6:24 AM Dafna Ron Hi,
> we have a failure on basic suite for test check_snapshot_with. _memory.
>
> I am actually not seeing any reason the patch would cause that specific
> issue but its consistently failing on this change.
> I am seeing a vm memory saved to the iscsi domain but aside from that
> nothing is failing before tto cause a cleanup issue or create low space on
> the storage).
> can some one please take a look to see if anything in the patch is causing
> this?
>
>
> Link and headline of suspected patches:
>
> https://gerrit.ovirt.org/#/c/95436/1 -
> webadmin: network operation in progress - sync host interfaces
>
> Link to Job:
> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/
>
> Link to all logs:
>
>
> https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-4.2/post-004_basic_sanity.py/
>
> (Relevant) error snippet from the log:
>
>
>
> 2018-11-15 08:19:33,409-05 DEBUG
> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method:
> runVdsCommand, params: [IsVmDuringInitiating
> ,
> IsVmDuringInitiatingVDSCommandParameters:{vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
> timeElapsed: 1ms
> 2018-11-15 08:19:33,430-05 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] EVENT_ID:
> USER_FAILED_RUN_VM(54), Failed to run VM
> vm0 due to a failed validation: [Cannot run VM. Low disk space on Storage
> Domain iscsi.] (User: admin@internal-authz).
> 2018-11-15 08:19:33,430-05 WARN [org.ovirt.engine.core.bll.RunVmCommand]
> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Validation of
> action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__
> ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName
> iscsi
> 2018-11-15 08:19:33,431-05 INFO [org.ovirt.engine.core.bll.RunVmCommand]
> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Lock freed to
> object 'EngineLock:{exclusiveLocks='[9144eb88-8f0a-4e77-9d70-3b761e
> 48ecb4=VM]', sharedLocks=''}'
> 2018-11-15 08:19:33,438-05 DEBUG
> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor]
> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runAction,
> params: [RunVm, RunVmParams:{comm
> andId='69628467-56d3-4d93-aed1-a2807b7dbc45', user='null',
> commandType='Unknown', vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}],
> timeElapsed: 115ms
> 2018-11-15 08:19:33,441-05 ERROR
> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
> task-5) [] Operation Failed: [Cannot run VM. Low disk space on Storage
> Domain iscsi.]
> 2018-11-15 08:19:34,044-05 DEBUG
> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] START,
> GetAllVmStatsVDSCommand(HostName = lago-basic-su
> ite-4-2-host-0,
> VdsIdVDSCommandParametersBase:{hostId='063dfef9-bc9d-44c3-8ba3-8142e3eb129c'}),
> log id: 65651e2b
> 2018-11-15 08:19:34,045-05 DEBUG
> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message]
> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] SEND
> destination:jms.topic.vdsm_requests
> reply-to:jms.topic.vdsm_responses
> content-length:103
>
>
>
>
> ___
> Devel mailing list -- de...@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/de...@ovirt.org/message/ZIG4VQKIFFARDTDJ6B2JROENN2Z44B4I/
>
___
Infra mailing list -- infra@ovirt.org
To unsubscribe send an email to infra-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/A62A2KACAGE5I7Y5MN5JPTKKI644YEJM/
