Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!

2018-03-19 Thread Enrico Becchetti 

Il 16/03/2018 15:48, Alex Crow ha scritto:

On 16/03/18 13:46, Nicolas Ecarnot wrote:

Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :



Den 16 mars 2018 12:26 skrev Enrico Becchetti 
:


   Dear All,
    Does someone had seen that error ?


Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has 
insufficient workload to trigger such event).

And in every case, there was no actual lack of space.


    Enrico Becchetti Servizio di Calcolo e Reti
I think I remember something to do with thin provisioning and not 
being able to grow fast enough, so out of space. Are the VM's disk 
thick or thin?


All our storage domains are thin-prov. and served by iSCSI 
(Equallogic PS6xxx and 4xxx).


Enrico, do you know if a bug has been filed about this?

Did the VM remain paused? In my experience the VM just gets 
temporarily paused while the storage is expanded. RH confirmed to me 
in a ticket that this is expected behaviour.


If you need high write performance your VM disks should always be 
preallocated. We only use Thin Provision for VMs where we know that 
disk writes are low (eg network services, CPU-bound apps, etc).



Thanks a lot !!!
Best Regards
Enrico


Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute 
advice.
The information provided is correct to our knowledge & belief and must 
not
be used as a substitute for obtaining tax, regulatory, investment, 
legal or

any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 
7608 5300.

(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



--
___

Enrico BecchettiServizio di Calcolo e Reti

Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
Via Pascoli,c/o Dipartimento di Fisica  06123 Perugia (ITALY)
Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it
__

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!

2018-03-18 Thread Yaniv Kaul 
On Fri, Mar 16, 2018 at 1:25 PM, Enrico Becchetti <
enrico.becche...@pg.infn.it> wrote:

>   Dear All,
> Does someone had seen that error ? When I run this command from my virtual
> machine:
>
> # time dd if=/dev/zero of=enrico.dd bs=4k count=1000
>

I don't think it's a very interesting test case for IO performance, but in
any case, it may cause the VM to try to write faster than its thin
provisioned disk can be extended.
A simple workaround would be to change in VDSM the threshold of when it
gets extended and by how much.
For example:
[irs]
volume_utilization_percent = 15
volume_utilization_chunk_mb = 4048

Y.


> VM was paused due to kind a storage error/problem. Strange message
> because tell about "no storage space error" but ovirt puts virtual machine
> in
> a paused state.
>
> Inside events  from  ovirt web interface  I see this:
>
> "VM has been paused due to lack of storage space"
>
> but no ERROR found in /var/log/vdsm.log.
>
> My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before
> now
> I haven't see any other problem during the normal functioning of the vm ,
> it's seem
> that this error occurs only when there is massive I/O.
>
> Any ideas ?
> Thanks a lot.
> Best Regards
> Enrico
>
>
> --
> ___
>
> Enrico BecchettiServizio di Calcolo e Reti
>
> Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
> Via Pascoli,c/o Dipartimento di Fisica  06123 Perugia (ITALY)
> Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it
> __
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!

2018-03-16 Thread Nicolas Ecarnot 

Le 16/03/2018 à 15:48, Alex Crow a écrit :

On 16/03/18 13:46, Nicolas Ecarnot wrote:

Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :



Den 16 mars 2018 12:26 skrev Enrico Becchetti 
:


   Dear All,
    Does someone had seen that error ?


Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has 
insufficient workload to trigger such event).

And in every case, there was no actual lack of space.


    Enrico Becchetti Servizio di Calcolo e Reti
I think I remember something to do with thin provisioning and not 
being able to grow fast enough, so out of space. Are the VM's disk 
thick or thin?


All our storage domains are thin-prov. and served by iSCSI (Equallogic 
PS6xxx and 4xxx).


Enrico, do you know if a bug has been filed about this?

Did the VM remain paused? In my experience the VM just gets temporarily 
paused while the storage is expanded. RH confirmed to me in a ticket 
that this is expected behaviour.


AFAIR, most of them went back up and running by themselves (we had to 
manually some of them from times to times).

The storage side weakness is an interesting trail to follow.
We also experienced this behavior when migrating lots of VMs at once, 
yet using a dedicated storage network.


Being on this mailing list since long, I remember we already discussed 
several times about how some users feel how oVirt can appear sensitive 
to storage latencies. On my side, the site where most of our workload 
resides is still in 3.6, so I can not yet witness the efforts oVirt devs 
have made to cope with this in 4.2 but I'm sure they did.


--
Nicolas ECARNOT
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!

2018-03-16 Thread Alex Crow 

On 16/03/18 13:46, Nicolas Ecarnot wrote:

Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :



Den 16 mars 2018 12:26 skrev Enrico Becchetti 
:


   Dear All,
    Does someone had seen that error ?


Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has 
insufficient workload to trigger such event).

And in every case, there was no actual lack of space.


    Enrico Becchetti Servizio di Calcolo e Reti
I think I remember something to do with thin provisioning and not 
being able to grow fast enough, so out of space. Are the VM's disk 
thick or thin?


All our storage domains are thin-prov. and served by iSCSI (Equallogic 
PS6xxx and 4xxx).


Enrico, do you know if a bug has been filed about this?

Did the VM remain paused? In my experience the VM just gets temporarily 
paused while the storage is expanded. RH confirmed to me in a ticket 
that this is expected behaviour.


If you need high write performance your VM disks should always be 
preallocated. We only use Thin Provision for VMs where we know that disk 
writes are low (eg network services, CPU-bound apps, etc).


Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!

2018-03-16 Thread Nicolas Ecarnot 

Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :



Den 16 mars 2018 12:26 skrev Enrico Becchetti :

   Dear All,
Does someone had seen that error ?


Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has 
insufficient workload to trigger such event).

And in every case, there was no actual lack of space.


Enrico Becchetti Servizio di Calcolo e Reti
I think I remember something to do with thin provisioning and not being 
able to grow fast enough, so out of space. Are the VM's disk thick or thin?


All our storage domains are thin-prov. and served by iSCSI (Equallogic 
PS6xxx and 4xxx).


Enrico, do you know if a bug has been filed about this?

--
Nicolas ECARNOT
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!

2018-03-16 Thread Enrico Becchetti 
yes ... it's a thin provisioning , in fact with preallocated disk type I 
haven't any problem.

Thanks you so much
Best Regards
Enrico

Il 16/03/2018 13:28, Karli Sjöberg ha scritto:



Den 16 mars 2018 12:26 skrev Enrico Becchetti 
:


  Dear All,
Does someone had seen that error ? When I run this command from my
virtual machine:

# time dd if=/dev/zero of=enrico.dd bs=4k count=1000

VM was paused due to kind a storage error/problem. Strange message
because tell about "no storage space error" but ovirt puts virtual
machine in
a paused state.

Inside events  from  ovirt web interface  I see this:

"VM has been paused due to lack of storage space"

but no ERROR found in /var/log/vdsm.log.

My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and
before now
I haven't see any other problem during the normal functioning of
the vm
, it's seem
that this error occurs only when there is massive I/O.

Any ideas ?
Thanks a lot.
Best Regards
Enrico


-- 
___


Enrico Becchetti Servizio di Calcolo e Reti

Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY)
Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it
__

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


I think I remember something to do with thin provisioning and not 
being able to grow fast enough, so out of space. Are the VM's disk 
thick or thin?


/K
  Dear All,
Does someone had seen that error ? When I run this command from my
virtual machine:

# time dd if=/dev/zero of=enrico.dd bs=4k count=1000

VM was paused due to kind a storage error/problem. Strange message
because tell about "no storage space error" but ovirt puts virtual
machine in
a paused state.

Inside events  from  ovirt web interface  I see this:

"VM has been paused due to lack of storage space"

but no ERROR found in /var/log/vdsm.log.

My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and
before now
I haven't see any other problem during the normal functioning of the vm
, it's seem
that this error occurs only when there is massive I/O.

Any ideas ?
Thanks a lot.
Best Regards
Enrico

--
___

Enrico Becchetti Servizio di Calcolo e Reti

Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY)
Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it
__

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



--
___

Enrico BecchettiServizio di Calcolo e Reti

Istituto Nazionale di Fisica Nucleare - Sezione di Perugia
Via Pascoli,c/o Dipartimento di Fisica  06123 Perugia (ITALY)
Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it
__

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread Nir Soffer 
On Thu, Apr 14, 2016 at 1:23 PM,   wrote:
> Hi Nir,
>
> El 2016-04-14 11:02, Nir Soffer escribió:
>>
>> On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland 
>> wrote:
>>>
>>> Nir,
>>> See attached the repoplot output.
>>
>>
>> So we have about one concurrent lvm command without any disk operations,
>> and
>> everything seems snappy.
>>
>> Nicolás, maybe this storage or the host is overloaded by the vms? Are your
>> vms
>> doing lot of io?
>>
>
> Not that I know, actually it should have been a "calm" time slot as far as
> IOs go, nor the storage was overloaded at that time. If I'm not mistaken, on
> the repoplot report I see there are two LVM operations at a time, maybe that
> has something to do with it?

The operation that took about 50 seconds started in the same time that another
operation started, but it does not explain why several other lvm comands took
about 15 seconds each.

> (although as you say, the lvextend is just a
> metadata change...)
>
>
>> lvextend operation should be very fast operation, this is just a
>> metadata change,
>> allocating couple of extents to that lv.
>>
>> Zdenek, how do you suggest to debug slow lvm commands?
>>
>> See the attached pdf, lvm commands took 15-50 seconds.
>>
>>>
>>> On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer  wrote:


 On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland 
 wrote:
 > From the log, we can see that the lvextend command took 18 sec, which
 > is
 > quite long.

 Fred, can you run repoplot on this log file? it will may explain why
 this
 lvm
 call took 18 seconds.

 Nir

 >
 > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
 > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
 > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
 > devices {
 > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
 > write_cache_state=0 disable_after_error_count=3 filter = [
 > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'',
 > '\''r|.*|'\''
 > ] }
 > global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1
 > use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } '
 > --autobackup
 > n --size 6016m
 >
 >
 > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
 > (cwd None)
 > 
 > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
 > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:  = '
 > WARNING: lvmetad is running but disabled. Restart lvmetad before
 > enabling
 > it!\n  WARNING: This metadata update is NOT backed up\n';  = 0
 >
 >
 > The watermark can be configured by the following value:
 >
 > 'volume_utilization_percent', '50',
 > 'Together with volume_utilization_chunk_mb, set the minimal free '
 > 'space before a thin provisioned block volume is extended. Use '
 > 'lower values to extend earlier.')
 >
 > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
 >  wrote:
 >>
 >>
 >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote:
 >> >
 >> > Ok, that makes sense, thanks for the insight both Alex and Fred.
 >> > I'm
 >> > attaching the VDSM log of the SPM node at the time of the pause. I
 >> > couldn't
 >> > find anything that would clearly identify the problem, but maybe
 >> > you'll be
 >> > able to.
 >>
 >> In extreme conditions it will happen. When your storage is slow to
 >> respond
 >> to extension request, and when your write rate is very high then it
 >> may
 >> happen, as it is happening to you, that you run out space sooner than
 >> the
 >> extension finishes. You can change the watermark value I guess(right,
 >> Fred?), but better would be to plan a bit more ahead and either use
 >> preallocated or create thin and then allocate expected size in
 >> advance
 >> before the operation causing it (typically it only happens during
 >> untarring
 >> gigabytes of data, or huge database dump/restore)
 >> Even then, the VM should always be automatially resumed once the disk
 >> space is allocated
 >>
 >> Thanks,
 >> michal
 >>
 >> >
 >> > Thanks.
 >> >
 >> > Regards.
 >> >
 >> > El 2016-04-13 13:09, Fred Rolland escribió:
 >> >> Hi,
 >> >> Yes, just as Alex explained, if the disk has been created as thin
 >> >> provisioning, the vdsm will extends once a watermark is reached.
 >> >> Usually it should not get to the state the Vm is paused.
 >> >> From the log, you can see that the request for extension has been
 >> >> sent
 >> >> before the VM got to the No Space Error.
 >> >> Later, we can see the VM resuming.
 >> >> INFO::2016-04-13
 >> >>

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread nicolas 

Hi Nir,

El 2016-04-14 11:02, Nir Soffer escribió:
On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland  
wrote:

Nir,
See attached the repoplot output.


So we have about one concurrent lvm command without any disk 
operations, and

everything seems snappy.

Nicolás, maybe this storage or the host is overloaded by the vms? Are 
your vms

doing lot of io?



Not that I know, actually it should have been a "calm" time slot as far 
as IOs go, nor the storage was overloaded at that time. If I'm not 
mistaken, on the repoplot report I see there are two LVM operations at a 
time, maybe that has something to do with it? (although as you say, the 
lvextend is just a metadata change...)



lvextend operation should be very fast operation, this is just a
metadata change,
allocating couple of extents to that lv.

Zdenek, how do you suggest to debug slow lvm commands?

See the attached pdf, lvm commands took 15-50 seconds.



On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer  
wrote:


On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland 
wrote:
> From the log, we can see that the lvextend command took 18 sec, which is
> quite long.

Fred, can you run repoplot on this log file? it will may explain why 
this

lvm
call took 18 seconds.

Nir

>
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
> devices {
> preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
> write_cache_state=0 disable_after_error_count=3 filter = [
> '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\''
> ] }
> global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1
> use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } '
> --autobackup
> n --size 6016m
>
> 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
> (cwd None)
> 
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:  = '
> WARNING: lvmetad is running but disabled. Restart lvmetad before
> enabling
> it!\n  WARNING: This metadata update is NOT backed up\n';  = 0
>
>
> The watermark can be configured by the following value:
>
> 'volume_utilization_percent', '50',
> 'Together with volume_utilization_chunk_mb, set the minimal free '
> 'space before a thin provisioned block volume is extended. Use '
> 'lower values to extend earlier.')
>
> On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
>  wrote:
>>
>>
>> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote:
>> >
>> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
>> > attaching the VDSM log of the SPM node at the time of the pause. I
>> > couldn't
>> > find anything that would clearly identify the problem, but maybe
>> > you'll be
>> > able to.
>>
>> In extreme conditions it will happen. When your storage is slow to
>> respond
>> to extension request, and when your write rate is very high then it may
>> happen, as it is happening to you, that you run out space sooner than
>> the
>> extension finishes. You can change the watermark value I guess(right,
>> Fred?), but better would be to plan a bit more ahead and either use
>> preallocated or create thin and then allocate expected size in advance
>> before the operation causing it (typically it only happens during
>> untarring
>> gigabytes of data, or huge database dump/restore)
>> Even then, the VM should always be automatially resumed once the disk
>> space is allocated
>>
>> Thanks,
>> michal
>>
>> >
>> > Thanks.
>> >
>> > Regards.
>> >
>> > El 2016-04-13 13:09, Fred Rolland escribió:
>> >> Hi,
>> >> Yes, just as Alex explained, if the disk has been created as thin
>> >> provisioning, the vdsm will extends once a watermark is reached.
>> >> Usually it should not get to the state the Vm is paused.
>> >> From the log, you can see that the request for extension has been
>> >> sent
>> >> before the VM got to the No Space Error.
>> >> Later, we can see the VM resuming.
>> >> INFO::2016-04-13
>> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension
>> >> for
>> >> volume
>> >> 
>> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
>> >> virtio-disk0 error enospc
>> >> 
>> >> INFO::2016-04-13
>> >> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
>> >> Note that the extension is done on the SPM host, so it would be
>> >> interesting to see the vdsm log from the host that was in SPM role
>> >> at
>> >> this timeframe.
>> >> Regards,
>> >> Fred
>> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow 
>> >> wrote:
>> >>> Hi,
>> >>> If you have

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread Nir Soffer 
On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland  wrote:
> Nir,
> See attached the repoplot output.

So we have about one concurrent lvm command without any disk operations, and
everything seems snappy.

Nicolás, maybe this storage or the host is overloaded by the vms? Are your vms
doing lot of io?

lvextend operation should be very fast operation, this is just a
metadata change,
allocating couple of extents to that lv.

Zdenek, how do you suggest to debug slow lvm commands?

See the attached pdf, lvm commands took 15-50 seconds.

>
> On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer  wrote:
>>
>> On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland 
>> wrote:
>> > From the log, we can see that the lvextend command took 18 sec, which is
>> > quite long.
>>
>> Fred, can you run repoplot on this log file? it will may explain why this
>> lvm
>> call took 18 seconds.
>>
>> Nir
>>
>> >
>> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
>> > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
>> > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
>> > devices {
>> > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
>> > write_cache_state=0 disable_after_error_count=3 filter = [
>> > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\''
>> > ] }
>> > global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1
>> > use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } '
>> > --autobackup
>> > n --size 6016m
>> >
>> > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
>> > (cwd None)
>> > 
>> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
>> > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:  = '
>> > WARNING: lvmetad is running but disabled. Restart lvmetad before
>> > enabling
>> > it!\n  WARNING: This metadata update is NOT backed up\n';  = 0
>> >
>> >
>> > The watermark can be configured by the following value:
>> >
>> > 'volume_utilization_percent', '50',
>> > 'Together with volume_utilization_chunk_mb, set the minimal free '
>> > 'space before a thin provisioned block volume is extended. Use '
>> > 'lower values to extend earlier.')
>> >
>> > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
>> >  wrote:
>> >>
>> >>
>> >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote:
>> >> >
>> >> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
>> >> > attaching the VDSM log of the SPM node at the time of the pause. I
>> >> > couldn't
>> >> > find anything that would clearly identify the problem, but maybe
>> >> > you'll be
>> >> > able to.
>> >>
>> >> In extreme conditions it will happen. When your storage is slow to
>> >> respond
>> >> to extension request, and when your write rate is very high then it may
>> >> happen, as it is happening to you, that you run out space sooner than
>> >> the
>> >> extension finishes. You can change the watermark value I guess(right,
>> >> Fred?), but better would be to plan a bit more ahead and either use
>> >> preallocated or create thin and then allocate expected size in advance
>> >> before the operation causing it (typically it only happens during
>> >> untarring
>> >> gigabytes of data, or huge database dump/restore)
>> >> Even then, the VM should always be automatially resumed once the disk
>> >> space is allocated
>> >>
>> >> Thanks,
>> >> michal
>> >>
>> >> >
>> >> > Thanks.
>> >> >
>> >> > Regards.
>> >> >
>> >> > El 2016-04-13 13:09, Fred Rolland escribió:
>> >> >> Hi,
>> >> >> Yes, just as Alex explained, if the disk has been created as thin
>> >> >> provisioning, the vdsm will extends once a watermark is reached.
>> >> >> Usually it should not get to the state the Vm is paused.
>> >> >> From the log, you can see that the request for extension has been
>> >> >> sent
>> >> >> before the VM got to the No Space Error.
>> >> >> Later, we can see the VM resuming.
>> >> >> INFO::2016-04-13
>> >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension
>> >> >> for
>> >> >> volume
>> >> >> 
>> >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
>> >> >> virtio-disk0 error enospc
>> >> >> 
>> >> >> INFO::2016-04-13
>> >> >> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
>> >> >> Note that the extension is done on the SPM host, so it would be
>> >> >> interesting to see the vdsm log from the host that was in SPM role
>> >> >> at
>> >> >> this timeframe.
>> >> >> Regards,
>> >> >> Fred
>> >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow 
>> >> >> wrote:
>> >> >>> Hi,
>> >> >>> If you have set up VM disks as Thin Provisioned, the VM has to
>> >> >>>

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread Fred Rolland 
Nir,
See attached the repoplot output.

On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer  wrote:

> On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland 
> wrote:
> > From the log, we can see that the lvextend command took 18 sec, which is
> > quite long.
>
> Fred, can you run repoplot on this log file? it will may explain why this
> lvm
> call took 18 seconds.
>
> Nir
>
> >
> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config '
> devices {
> > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3 filter = [
> > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\''
> ] }
> > global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1
> > use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } '
> --autobackup
> > n --size 6016m
> > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
> > (cwd None)
> > 
> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:  = '
> > WARNING: lvmetad is running but disabled. Restart lvmetad before enabling
> > it!\n  WARNING: This metadata update is NOT backed up\n';  = 0
> >
> >
> > The watermark can be configured by the following value:
> >
> > 'volume_utilization_percent', '50',
> > 'Together with volume_utilization_chunk_mb, set the minimal free '
> > 'space before a thin provisioned block volume is extended. Use '
> > 'lower values to extend earlier.')
> >
> > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
> >  wrote:
> >>
> >>
> >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote:
> >> >
> >> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
> >> > attaching the VDSM log of the SPM node at the time of the pause. I
> couldn't
> >> > find anything that would clearly identify the problem, but maybe
> you'll be
> >> > able to.
> >>
> >> In extreme conditions it will happen. When your storage is slow to
> respond
> >> to extension request, and when your write rate is very high then it may
> >> happen, as it is happening to you, that you run out space sooner than
> the
> >> extension finishes. You can change the watermark value I guess(right,
> >> Fred?), but better would be to plan a bit more ahead and either use
> >> preallocated or create thin and then allocate expected size in advance
> >> before the operation causing it (typically it only happens during
> untarring
> >> gigabytes of data, or huge database dump/restore)
> >> Even then, the VM should always be automatially resumed once the disk
> >> space is allocated
> >>
> >> Thanks,
> >> michal
> >>
> >> >
> >> > Thanks.
> >> >
> >> > Regards.
> >> >
> >> > El 2016-04-13 13:09, Fred Rolland escribió:
> >> >> Hi,
> >> >> Yes, just as Alex explained, if the disk has been created as thin
> >> >> provisioning, the vdsm will extends once a watermark is reached.
> >> >> Usually it should not get to the state the Vm is paused.
> >> >> From the log, you can see that the request for extension has been
> sent
> >> >> before the VM got to the No Space Error.
> >> >> Later, we can see the VM resuming.
> >> >> INFO::2016-04-13
> >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
> >> >> volume
> >> >> 
> >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
> >> >> virtio-disk0 error enospc
> >> >> 
> >> >> INFO::2016-04-13
> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
> >> >> Note that the extension is done on the SPM host, so it would be
> >> >> interesting to see the vdsm log from the host that was in SPM role at
> >> >> this timeframe.
> >> >> Regards,
> >> >> Fred
> >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow 
> >> >> wrote:
> >> >>> Hi,
> >> >>> If you have set up VM disks as Thin Provisioned, the VM has to
> >> >>> pause when the disk image needs to expand. You won't see this on VMs
> >> >>> with preallocated storage.
> >> >>> It's not the SAN that's running out of space, it's the VM image
> >> >>> needing to be expanded incrementally each time.
> >> >>> Cheers
> >> >>> Alex
> >> >>> On 13/04/16 12:04, nico...@devels.es wrote:
> >> >>> Hi Fred,
> >> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host
> >> >>> where this machine has been running. Should you need any further
> >> >>> info, don't hesitate to ask.
> >> >>> Thanks.
> >> >>> Regards.
> >> >>> El 2016-04-13 11:54, Fred Rolland escribió:
> >> >>> Hi,
> >> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
> >> >>>

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread Nir Soffer 
On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland  wrote:
> From the log, we can see that the lvextend command took 18 sec, which is
> quite long.

Fred, can you run repoplot on this log file? it will may explain why this lvm
call took 18 seconds.

Nir

>
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
> --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' devices {
> preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
> write_cache_state=0 disable_after_error_count=3 filter = [
> '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\'' ] }
> global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1
> use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --autobackup
> n --size 6016m
> 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
> (cwd None)
> 
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
> 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:  = '
> WARNING: lvmetad is running but disabled. Restart lvmetad before enabling
> it!\n  WARNING: This metadata update is NOT backed up\n';  = 0
>
>
> The watermark can be configured by the following value:
>
> 'volume_utilization_percent', '50',
> 'Together with volume_utilization_chunk_mb, set the minimal free '
> 'space before a thin provisioned block volume is extended. Use '
> 'lower values to extend earlier.')
>
> On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek
>  wrote:
>>
>>
>> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote:
>> >
>> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
>> > attaching the VDSM log of the SPM node at the time of the pause. I couldn't
>> > find anything that would clearly identify the problem, but maybe you'll be
>> > able to.
>>
>> In extreme conditions it will happen. When your storage is slow to respond
>> to extension request, and when your write rate is very high then it may
>> happen, as it is happening to you, that you run out space sooner than the
>> extension finishes. You can change the watermark value I guess(right,
>> Fred?), but better would be to plan a bit more ahead and either use
>> preallocated or create thin and then allocate expected size in advance
>> before the operation causing it (typically it only happens during untarring
>> gigabytes of data, or huge database dump/restore)
>> Even then, the VM should always be automatially resumed once the disk
>> space is allocated
>>
>> Thanks,
>> michal
>>
>> >
>> > Thanks.
>> >
>> > Regards.
>> >
>> > El 2016-04-13 13:09, Fred Rolland escribió:
>> >> Hi,
>> >> Yes, just as Alex explained, if the disk has been created as thin
>> >> provisioning, the vdsm will extends once a watermark is reached.
>> >> Usually it should not get to the state the Vm is paused.
>> >> From the log, you can see that the request for extension has been sent
>> >> before the VM got to the No Space Error.
>> >> Later, we can see the VM resuming.
>> >> INFO::2016-04-13
>> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
>> >> volume
>> >> 
>> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
>> >> virtio-disk0 error enospc
>> >> 
>> >> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
>> >> Note that the extension is done on the SPM host, so it would be
>> >> interesting to see the vdsm log from the host that was in SPM role at
>> >> this timeframe.
>> >> Regards,
>> >> Fred
>> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow 
>> >> wrote:
>> >>> Hi,
>> >>> If you have set up VM disks as Thin Provisioned, the VM has to
>> >>> pause when the disk image needs to expand. You won't see this on VMs
>> >>> with preallocated storage.
>> >>> It's not the SAN that's running out of space, it's the VM image
>> >>> needing to be expanded incrementally each time.
>> >>> Cheers
>> >>> Alex
>> >>> On 13/04/16 12:04, nico...@devels.es wrote:
>> >>> Hi Fred,
>> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host
>> >>> where this machine has been running. Should you need any further
>> >>> info, don't hesitate to ask.
>> >>> Thanks.
>> >>> Regards.
>> >>> El 2016-04-13 11:54, Fred Rolland escribió:
>> >>> Hi,
>> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
>> >>> Can you provide the vdsm logs from the host where this VM runs ?
>> >>> Thanks,
>> >>> Freddy
>> >>> On Wed, Apr 13, 2016 at 1:02 PM,  wrote:
>> >>> Hi,
>> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
>> >>> events like these:
>> >>> 2016-04-13 10:52:30,735 INFO
>> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread Fred Rolland 
>From the log, we can see that the lvextend command took 18 sec, which is
quite long.

60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset
--cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' devices
{ preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3 filter = [
'\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\'' ]
}  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1
use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } '
--autobackup n --size 6016m
5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0
(cwd None)

60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13
10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS:  = '
WARNING: lvmetad is running but disabled. Restart lvmetad before enabling
it!\n  WARNING: This metadata update is NOT backed up\n';  = 0


The watermark can be configured by the following value:

'volume_utilization_percent', '50',
'Together with volume_utilization_chunk_mb, set the minimal free '
'space before a thin provisioned block volume is extended. Use '
'lower values to extend earlier.')

On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote:
> >
> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm
> attaching the VDSM log of the SPM node at the time of the pause. I couldn't
> find anything that would clearly identify the problem, but maybe you'll be
> able to.
>
> In extreme conditions it will happen. When your storage is slow to respond
> to extension request, and when your write rate is very high then it may
> happen, as it is happening to you, that you run out space sooner than the
> extension finishes. You can change the watermark value I guess(right,
> Fred?), but better would be to plan a bit more ahead and either use
> preallocated or create thin and then allocate expected size in advance
> before the operation causing it (typically it only happens during untarring
> gigabytes of data, or huge database dump/restore)
> Even then, the VM should always be automatially resumed once the disk
> space is allocated
>
> Thanks,
> michal
>
> >
> > Thanks.
> >
> > Regards.
> >
> > El 2016-04-13 13:09, Fred Rolland escribió:
> >> Hi,
> >> Yes, just as Alex explained, if the disk has been created as thin
> >> provisioning, the vdsm will extends once a watermark is reached.
> >> Usually it should not get to the state the Vm is paused.
> >> From the log, you can see that the request for extension has been sent
> >> before the VM got to the No Space Error.
> >> Later, we can see the VM resuming.
> >> INFO::2016-04-13
> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
> >> volume
> >> 
> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
> >> virtio-disk0 error enospc
> >> 
> >> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
> >> Note that the extension is done on the SPM host, so it would be
> >> interesting to see the vdsm log from the host that was in SPM role at
> >> this timeframe.
> >> Regards,
> >> Fred
> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow 
> >> wrote:
> >>> Hi,
> >>> If you have set up VM disks as Thin Provisioned, the VM has to
> >>> pause when the disk image needs to expand. You won't see this on VMs
> >>> with preallocated storage.
> >>> It's not the SAN that's running out of space, it's the VM image
> >>> needing to be expanded incrementally each time.
> >>> Cheers
> >>> Alex
> >>> On 13/04/16 12:04, nico...@devels.es wrote:
> >>> Hi Fred,
> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host
> >>> where this machine has been running. Should you need any further
> >>> info, don't hesitate to ask.
> >>> Thanks.
> >>> Regards.
> >>> El 2016-04-13 11:54, Fred Rolland escribió:
> >>> Hi,
> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)
> >>> Can you provide the vdsm logs from the host where this VM runs ?
> >>> Thanks,
> >>> Freddy
> >>> On Wed, Apr 13, 2016 at 1:02 PM,  wrote:
> >>> Hi,
> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
> >>> events like these:
> >>> 2016-04-13 10:52:30,735 INFO
> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
> >>> from
> >>> 'Up' --> 'Paused'
> >>> 2016-04-13 10:52:30,815 INFO
> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> >>> (DefaultQuartzScheduler_Worker-86)

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread Michal Skrivanek 

> On 14 Apr 2016, at 09:57, nico...@devels.es wrote:
> 
> Ok, that makes sense, thanks for the insight both Alex and Fred. I'm 
> attaching the VDSM log of the SPM node at the time of the pause. I couldn't 
> find anything that would clearly identify the problem, but maybe you'll be 
> able to.

In extreme conditions it will happen. When your storage is slow to respond to 
extension request, and when your write rate is very high then it may happen, as 
it is happening to you, that you run out space sooner than the extension 
finishes. You can change the watermark value I guess(right, Fred?), but better 
would be to plan a bit more ahead and either use preallocated or create thin 
and then allocate expected size in advance before the operation causing it 
(typically it only happens during untarring gigabytes of data, or huge database 
dump/restore)
Even then, the VM should always be automatially resumed once the disk space is 
allocated

Thanks,
michal

> 
> Thanks.
> 
> Regards.
> 
> El 2016-04-13 13:09, Fred Rolland escribió:
>> Hi,
>> Yes, just as Alex explained, if the disk has been created as thin
>> provisioning, the vdsm will extends once a watermark is reached.
>> Usually it should not get to the state the Vm is paused.
>> From the log, you can see that the request for extension has been sent
>> before the VM got to the No Space Error.
>> Later, we can see the VM resuming.
>> INFO::2016-04-13
>> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
>> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
>> volume
>> 
>> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
>> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
>> virtio-disk0 error enospc
>> 
>> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
>> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume
>> Note that the extension is done on the SPM host, so it would be
>> interesting to see the vdsm log from the host that was in SPM role at
>> this timeframe.
>> Regards,
>> Fred
>> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow 
>> wrote:
>>> Hi,
>>> If you have set up VM disks as Thin Provisioned, the VM has to
>>> pause when the disk image needs to expand. You won't see this on VMs
>>> with preallocated storage.
>>> It's not the SAN that's running out of space, it's the VM image
>>> needing to be expanded incrementally each time.
>>> Cheers
>>> Alex
>>> On 13/04/16 12:04, nico...@devels.es wrote:
>>> Hi Fred,
>>> This is an iSCSI storage. I'm attaching the VDSM logs from the host
>>> where this machine has been running. Should you need any further
>>> info, don't hesitate to ask.
>>> Thanks.
>>> Regards.
>>> El 2016-04-13 11:54, Fred Rolland escribió:
>>> Hi,
>>> What kind of storage do you have ? (ISCSI,FC,NFS...)
>>> Can you provide the vdsm logs from the host where this VM runs ?
>>> Thanks,
>>> Freddy
>>> On Wed, Apr 13, 2016 at 1:02 PM,  wrote:
>>> Hi,
>>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
>>> events like these:
>>> 2016-04-13 10:52:30,735 INFO 
>>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
>>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>>> from
>>> 'Up' --> 'Paused'
>>> 2016-04-13 10:52:30,815 INFO 
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>>> [1] [1]
>>> has been paused.
>>> 2016-04-13 10:52:30,898 ERROR
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
>>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>>> [1] [1]
>>> has been paused due to no Storage space error.
>>> 2016-04-13 10:52:52,320 WARN 
>>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
>>> (org.ovirt.thread.pool-8-thread-38) [] domain
>>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
>>> 'host6.domain.com [2] [2]'
>>> 2016-04-13 10:52:55,183 INFO 
>>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
>>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
>>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
>>> from
>>> 'Paused' --> 'Up'
>>> 2016-04-13 10:52:55,318 INFO 
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
>>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
>>> [1] [1]
>>> has recovered from paused back to up.
>>> The storage domain is far from being full, though (400+ G available
>>> right now). Could this be related to this other issue [1]? If not,
>>> how could I debug what's going on?
>>> Thanks.
>>>  [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
>>> [3]
>>> [3]
>>>

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-14 Thread nicolas 
Ok, that makes sense, thanks for the insight both Alex and Fred. I'm 
attaching the VDSM log of the SPM node at the time of the pause. I 
couldn't find anything that would clearly identify the problem, but 
maybe you'll be able to.


Thanks.

Regards.

El 2016-04-13 13:09, Fred Rolland escribió:

Hi,

Yes, just as Alex explained, if the disk has been created as thin
provisioning, the vdsm will extends once a watermark is reached.
Usually it should not get to the state the Vm is paused.

From the log, you can see that the request for extension has been sent
before the VM got to the No Space Error.
Later, we can see the VM resuming.

INFO::2016-04-13
10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for
volume

INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
virtio-disk0 error enospc

INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume

Note that the extension is done on the SPM host, so it would be
interesting to see the vdsm log from the host that was in SPM role at
this timeframe.

Regards,

Fred

On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow 
wrote:


Hi,

If you have set up VM disks as Thin Provisioned, the VM has to
pause when the disk image needs to expand. You won't see this on VMs
with preallocated storage.

It's not the SAN that's running out of space, it's the VM image
needing to be expanded incrementally each time.

Cheers

Alex

On 13/04/16 12:04, nico...@devels.es wrote:
Hi Fred,

This is an iSCSI storage. I'm attaching the VDSM logs from the host
where this machine has been running. Should you need any further
info, don't hesitate to ask.

Thanks.

Regards.

El 2016-04-13 11:54, Fred Rolland escribió:
Hi,

What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?

Thanks,

Freddy

On Wed, Apr 13, 2016 at 1:02 PM,  wrote:

Hi,

We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
events like these:

2016-04-13 10:52:30,735 INFO 
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
from
'Up' --> 'Paused'
2016-04-13 10:52:30,815 INFO 



[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]


(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,

Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
[1] [1]
has been paused.
2016-04-13 10:52:30,898 ERROR



[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]


(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,

Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
[1] [1]
has been paused due to no Storage space error.
2016-04-13 10:52:52,320 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) [] domain
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
'host6.domain.com [2] [2]'
2016-04-13 10:52:55,183 INFO 
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved
from
'Paused' --> 'Up'
2016-04-13 10:52:55,318 INFO 



[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]


(DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,

Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com
[1] [1]
has recovered from paused back to up.

The storage domain is far from being full, though (400+ G available

right now). Could this be related to this other issue [1]? If not,
how could I debug what's going on?

Thanks.

 [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
[3]
[3]
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4] [4]

Links:
--
[1] http://vm.domain.com [1]
[2] http://host6.domain.com [2]
[3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [3]
[4] http://lists.ovirt.org/mailman/listinfo/users [4]


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4]

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute
advice.
The information provided is correct to our knowledge & belief and must
not
be used as a substitute for obtaining tax, regulatory, investment,
legal or
any other appropriate advice.

"Transact" is operated by Integrated

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-13 Thread Fred Rolland 
Hi,

Yes, just as Alex explained, if the disk has been created as thin
provisioning, the vdsm will extends once a watermark is reached.
Usually it should not get to the state the Vm is paused.

>From the log, you can see that the request for extension has been sent
before the VM got to the No Space Error.
Later, we can see the VM resuming.

INFO::2016-04-13 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for volume

INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device
virtio-disk0 error enospc

INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)
vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume


Note that the extension is done on the SPM host, so it would be interesting
to see the vdsm log from the host that was in SPM role at this timeframe.

Regards,

Fred


On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow  wrote:

> Hi,
>
> If you have set up VM disks as Thin Provisioned, the VM has to pause when
> the disk image needs to expand. You won't see this on VMs with preallocated
> storage.
>
> It's not the SAN that's running out of space, it's the VM image needing to
> be expanded incrementally each time.
>
> Cheers
>
> Alex
>
>
> On 13/04/16 12:04, nico...@devels.es wrote:
>
> Hi Fred,
>
> This is an iSCSI storage. I'm attaching the VDSM logs from the host where
> this machine has been running. Should you need any further info, don't
> hesitate to ask.
>
> Thanks.
>
> Regards.
>
> El 2016-04-13 11:54, Fred Rolland escribió:
>
> Hi,
>
> What kind of storage do you have ? (ISCSI,FC,NFS...)
> Can you provide the vdsm logs from the host where this VM runs ?
>
> Thanks,
>
> Freddy
>
> On Wed, Apr 13, 2016 at 1:02 PM,  
> wrote:
>
> Hi,
>
> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
> events like these:
>
> 2016-04-13 10:52:30,735 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
> 'Up' --> 'Paused'
> 2016-04-13 10:52:30,815 INFO
>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has been paused.
> 2016-04-13 10:52:30,898 ERROR
>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has been paused due to no Storage space error.
> 2016-04-13 10:52:52,320 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-38) [] domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
> 'host6.domain.com [2]'
> 2016-04-13 10:52:55,183 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
> 'Paused' --> 'Up'
> 2016-04-13 10:52:55,318 INFO
>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has recovered from paused back to up.
>
> The storage domain is far from being full, though (400+ G available
> right now). Could this be related to this other issue [1]? If not,
> how could I debug what's going on?
>
> Thanks.
>
>  [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [3]
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users [4]
>
>
>
>
> Links:
> --
> [1] http://vm.domain.com
> [2] http://host6.domain.com
> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [4] http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> ___
> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>
>
>
> --
> This message is intended only for the addressee and may contain
> confidential information. Unless you are that person, you may not
> disclose its contents or use it in any way and are requested to delete
> the message along with any attachments and notify us immediately.
> This email is not intended to, nor should it be taken to, constitute advice.
> The information provided is correct to our knowledge & belief and must not
> be used as a substitute for obtaining tax, regulatory, investment, legal or
> any other appropriate advice.
>
> "Transact" is operated by Integrated Financial Arrangements Ltd.
> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
> (Registered office: as

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-13 Thread Colin Coe 
Ahh, we've seen this as well in RHEV and have wondered what was going on.
A better message would be good.

On Wed, Apr 13, 2016 at 7:43 PM, Alex Crow  wrote:

> Hi,
>
> If you have set up VM disks as Thin Provisioned, the VM has to pause when
> the disk image needs to expand. You won't see this on VMs with preallocated
> storage.
>
> It's not the SAN that's running out of space, it's the VM image needing to
> be expanded incrementally each time.
>
> Cheers
>
> Alex
>
>
> On 13/04/16 12:04, nico...@devels.es wrote:
>
> Hi Fred,
>
> This is an iSCSI storage. I'm attaching the VDSM logs from the host where
> this machine has been running. Should you need any further info, don't
> hesitate to ask.
>
> Thanks.
>
> Regards.
>
> El 2016-04-13 11:54, Fred Rolland escribió:
>
> Hi,
>
> What kind of storage do you have ? (ISCSI,FC,NFS...)
> Can you provide the vdsm logs from the host where this VM runs ?
>
> Thanks,
>
> Freddy
>
> On Wed, Apr 13, 2016 at 1:02 PM,  
> wrote:
>
> Hi,
>
> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
> events like these:
>
> 2016-04-13 10:52:30,735 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
> 'Up' --> 'Paused'
> 2016-04-13 10:52:30,815 INFO
>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has been paused.
> 2016-04-13 10:52:30,898 ERROR
>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has been paused due to no Storage space error.
> 2016-04-13 10:52:52,320 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-38) [] domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
> 'host6.domain.com [2]'
> 2016-04-13 10:52:55,183 INFO
> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
> 'Paused' --> 'Up'
> 2016-04-13 10:52:55,318 INFO
>
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
> has recovered from paused back to up.
>
> The storage domain is far from being full, though (400+ G available
> right now). Could this be related to this other issue [1]? If not,
> how could I debug what's going on?
>
> Thanks.
>
>  [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [3]
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users [4]
>
>
>
>
> Links:
> --
> [1] http://vm.domain.com
> [2] http://host6.domain.com
> [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
> [4] http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> ___
> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>
>
>
> --
> This message is intended only for the addressee and may contain
> confidential information. Unless you are that person, you may not
> disclose its contents or use it in any way and are requested to delete
> the message along with any attachments and notify us immediately.
> This email is not intended to, nor should it be taken to, constitute advice.
> The information provided is correct to our knowledge & belief and must not
> be used as a substitute for obtaining tax, regulatory, investment, legal or
> any other appropriate advice.
>
> "Transact" is operated by Integrated Financial Arrangements Ltd.
> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
> (Registered office: as above; Registered in England and Wales under
> number: 3727592). Authorised and regulated by the Financial Conduct
> Authority (entered on the Financial Services Register; no. 190856).
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-13 Thread Alex Crow 

Hi,

If you have set up VM disks as Thin Provisioned, the VM has to pause 
when the disk image needs to expand. You won't see this on VMs with 
preallocated storage.


It's not the SAN that's running out of space, it's the VM image needing 
to be expanded incrementally each time.


Cheers

Alex

On 13/04/16 12:04, nico...@devels.es wrote:

Hi Fred,

This is an iSCSI storage. I'm attaching the VDSM logs from the host 
where this machine has been running. Should you need any further info, 
don't hesitate to ask.


Thanks.

Regards.

El 2016-04-13 11:54, Fred Rolland escribió:

Hi,

What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?

Thanks,

Freddy

On Wed, Apr 13, 2016 at 1:02 PM,  wrote:


Hi,

We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
events like these:

2016-04-13 10:52:30,735 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Up' --> 'Paused'
2016-04-13 10:52:30,815 INFO


[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]

(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused.
2016-04-13 10:52:30,898 ERROR


[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]

(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused due to no Storage space error.
2016-04-13 10:52:52,320 WARN
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) [] domain
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
'host6.domain.com [2]'
2016-04-13 10:52:55,183 INFO
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Paused' --> 'Up'
2016-04-13 10:52:55,318 INFO


[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]

(DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has recovered from paused back to up.

The storage domain is far from being full, though (400+ G available
right now). Could this be related to this other issue [1]? If not,
how could I debug what's going on?

Thanks.

 [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
[3]
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4]




Links:
--
[1] http://vm.domain.com
[2] http://host6.domain.com
[3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
[4] http://lists.ovirt.org/mailman/listinfo/users



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-13 Thread nicolas 

Hi Fred,

This is an iSCSI storage. I'm attaching the VDSM logs from the host 
where this machine has been running. Should you need any further info, 
don't hesitate to ask.


Thanks.

Regards.

El 2016-04-13 11:54, Fred Rolland escribió:

Hi,

What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?

Thanks,

Freddy

On Wed, Apr 13, 2016 at 1:02 PM,  wrote:


Hi,

We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of
events like these:

2016-04-13 10:52:30,735 INFO 
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-86) [60dea18f] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Up' --> 'Paused'
2016-04-13 10:52:30,815 INFO 


[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]

(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused.
2016-04-13 10:52:30,898 ERROR


[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]

(DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has been paused due to no Storage space error.
2016-04-13 10:52:52,320 WARN 
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-8-thread-38) [] domain
'5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
'host6.domain.com [2]'
2016-04-13 10:52:55,183 INFO 
[org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from
'Paused' --> 'Up'
2016-04-13 10:52:55,318 INFO 


[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]

(DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1]
has recovered from paused back to up.

The storage domain is far from being full, though (400+ G available
right now). Could this be related to this other issue [1]? If not,
how could I debug what's going on?

Thanks.

 [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
[3]
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users [4]




Links:
--
[1] http://vm.domain.com
[2] http://host6.domain.com
[3] https://www.mail-archive.com/users@ovirt.org/msg32079.html
[4] http://lists.ovirt.org/mailman/listinfo/users

vdsm.log.gz
Description: GNU Zip compressed data
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] VM has been paused due to no Storage space error.

2016-04-13 Thread Fred Rolland 
Hi,

What kind of storage do you have ? (ISCSI,FC,NFS...)
Can you provide the vdsm logs from the host where this VM runs ?

Thanks,

Freddy

On Wed, Apr 13, 2016 at 1:02 PM,  wrote:

> Hi,
>
> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events like
> these:
>
> 2016-04-13 10:52:30,735 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Up' -->
> 'Paused'
> 2016-04-13 10:52:30,815 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call
> Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been
> paused.
> 2016-04-13 10:52:30,898 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call
> Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been
> paused due to no Storage space error.
> 2016-04-13 10:52:52,320 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-38) [] domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: '
> host6.domain.com'
> 2016-04-13 10:52:55,183 INFO  [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM
> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Paused'
> --> 'Up'
> 2016-04-13 10:52:55,318 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call
> Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has recovered
> from paused back to up.
>
> The storage domain is far from being full, though (400+ G available right
> now). Could this be related to this other issue [1]? If not, how could I
> debug what's going on?
>
> Thanks.
>
>  [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users