[ovirt-users] VM has been paused due to no Storage space error.
Hello, running ovirt Version 4.5.4-1.el8 on Centos 8, randomly we have this error: VM has been paused due to no Storage space error. We have plenty of space on the iSCSI storage. This is a preallocated disk, VirtIO-SCSi. No user interaction. It happens, so far, with 3 VM, Windows and Ubuntu. This service was stopped: dnf-makecache.service This is what I found on the engine log: 2024-08-19 01:04:35,522+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-25) [eb7e5f1] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'Up' --> 'Paused' 2024-08-19 01:04:35,665+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-25) [eb7e5f1] EVENT_ID: VM_PAUSED_ENOSPC(138), VM Bravo has been paused due to no Storage space error. 2024-08-19 09:26:35,855+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-29) [72482216] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'Paused' --> 'Down' 2024-08-19 09:26:48,114+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [72482216] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'WaitForLaunch' --> 'PoweringUp' 2024-08-19 09:27:50,062+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-6) [] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'PoweringUp' --> 'Up' 2024-08-19 09:29:25,145+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [72482216] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'Up' --> 'Paused' 2024-08-19 09:29:25,273+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-15) [72482216] EVENT_ID: VM_PAUSED_ENOSPC(138), VM Bravo has been paused due to no Storage space error. 2024-08-19 09:37:26,128+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [6d88f065] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'Paused' --> 'Down' 2024-08-19 09:41:43,300+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [6d88f065] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'WaitForLaunch' --> 'PoweringUp' 2024-08-19 09:42:14,882+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-23) [6d88f065] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'PoweringUp' --> 'Up' 2024-08-19 09:42:59,792+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [6d88f065] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'Up' --> 'Paused' 2024-08-19 09:42:59,894+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-15) [6d88f065] EVENT_ID: VM_PAUSED_ENOSPC(138), VM Bravo has been paused due to no Storage space error. 2024-08-19 09:45:30,334+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [6b3d8ee] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'Paused' --> 'Down' 2024-08-19 09:47:51,068+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [6b3d8ee] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'WaitForLaunch' --> 'PoweringUp' 2024-08-19 09:48:50,710+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'PoweringUp' --> 'Up' 2024-08-19 10:06:38,810+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [1dd98021] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'PoweringDown' --> 'Down' 2024-08-19 10:08:11,606+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [1dd98021] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'WaitForLaunch' --> 'PoweringUp' 2024-08-19 10:09:12,507+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'PoweringUp' --> 'Up' 2024-08-19 10:21:13,835+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [63fa2421] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'Up' --> 'Down' 2024-08-19 10:25:19,302+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-15) [63fa2421] VM 'ccc65521-934d-4f77-adf3-9f9eeb83a4f8'(Bravo) moved from 'WaitForLaunch' --> 'PoweringUp' 2024-08-19 10:26:05,456+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-3) [63fa2421] VM 'ccc65521-934d-4f
[ovirt-users] VM has been paused due to no Storage space error on ovirt 4.5
Hi there, last days we are facing issues with paused VMs (in past it was for few second to resize lv device), but now it doesn't resume. we migrated to 4.5.2 cluster, this never happened before with the same storage. there is almost notning in engine log 2022-09-06 09:47:11,160+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [51eb7178] VM 'cfff0648-6502-4977-95a8-c6f95c723f6d'(cm1.util.prod.hq.slde v.cz) moved from 'Up' --> 'Paused' 2022-09-06 09:47:11,264+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED(1,025), VM cm1.util.prod.hq. sldev.cz has been paused. 2022-09-06 09:47:11,271+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED_ENOSPC(138), VM cm1.util.pro d.hq.sldev.cz has been paused due to no Storage space error. but there are erros with LVM in vdsmlog. (attached) ovirt 4.5.2 OS - ovirt-node-ng. Thank you for any hint. Jirka 2022-09-06 09:47:00,702+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,60738 (api:48) 2022-09-06 09:47:00,715+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,60738 (api:54) 2022-09-06 09:47:00,872+0200 INFO (monitor/cf819cb) [storage.storagedomaincache] Removing domain cf819cb9-a51f-4b7a-baf6-a2a472aae6da from storage domain cache (sdc:211) 2022-09-06 09:47:01,301+0200 WARN (monitor/cf819cb) [storage.lvm] Command ['/sbin/lvm', 'vgs', '--devices', '/dev/mapper/36589cfc004b7c76d143cdc41ac27,/dev/mapper/36589cfc00512813b031ea0c556e6,/dev/mapper/36589cfc00667f45684cee766d4aa', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3hints="none" obtain_device_list_from_udev=0 } global { prioritise_write_locks=1 wait_for_locks=1 use_lvmpolld=1 } backup { retain_min=50 retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name', 'cf819cb9-a51f-4b7a-baf6-a2a472aae6da'] succeeded with warnings: [' Error reading device /dev/mapper/36589cfc004b7c76d143cdc41ac27 at 10115372744704 length 512.', ' Failed to read metadata area header on /dev/mapper/36589cfc004b7c76d143cdc41ac27 at 10115372744704', ' WARNING: bad metadata header on /dev/mapper/36589cfc004b7c76d143cdc41ac27 at 10115372744704.', ' WARNING: scanning /dev/mapper/36589cfc004b7c76d143cdc41ac27 mda2 failed to read metadata summary.', ' WARNING: repair VG metadata on /dev/mapper/36589cfc004b7c76d143cdc41ac27 with vgck --updatemetadata.', ' WARNING: Device /dev/mapper/36589cfc004b7c76d143cdc41ac27 has size of 15032385536 sectors which is smaller than corresponding PV size of 19756323200 sectors. Was device resized?', ' WARNING: One or more devices used as PVs in VG cf819cb9-a51f-4b7a-baf6-a2a472aae6da have changed sizes.'] (lvm:358) 2022-09-06 09:47:01,458+0200 WARN (monitor/cf819cb) [storage.lvm] Command ['/sbin/lvm', 'vgck', '--devices', '/dev/mapper/36589cfc004b7c76d143cdc41ac27', '--config', 'devices { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3hints="none" obtain_device_list_from_udev=0 } global { prioritise_write_locks=1 wait_for_locks=1 use_lvmpolld=1 } backup { retain_min=50 retain_days=0 }', 'cf819cb9-a51f-4b7a-baf6-a2a472aae6da'] succeeded with warnings: [' Error reading device /dev/mapper/36589cfc004b7c76d143cdc41ac27 at 10115372744704 length 512.', ' Failed to read metadata area header on /dev/mapper/36589cfc004b7c76d143cdc41ac27 at 10115372744704', ' WARNING: bad metadata header on /dev/mapper/36589cfc004b7c76d143cdc41ac27 at 10115372744704.', ' WARNING: scanning /dev/mapper/36589cfc004b7c76d143cdc41ac27 mda2 failed to read metadata summary.', ' WARNING: repair VG metadata on /dev/mapper/36589cfc004b7c76d143cdc41ac27 with vgck --updatemetadata.', ' WARNING: Device /dev/mapper/36589cfc004b7c76d143cdc41ac27 has size of 15032385536 sectors which is smaller than corresponding PV size of 19756323200 sectors. Was device resized?', ' WARNING: One or more devices used as PVs in VG cf819cb9-a51f-4b7a-baf6-a2a472aae6da have changed sizes.'] (lvm:358) 2022-09-06 09:47:02,407+0200 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,39940, vmId=0c418190-42da-42fe-a992-dbb83a364b93 (api:48) 2022-09-06 09:47:02,408+0200 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': '0c418190-42da-42fe-a992-dbb83a364b93'} (api:129) 2022-09-06 09:47:02,408+0200 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message':
Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
Il 16/03/2018 15:48, Alex Crow ha scritto: On 16/03/18 13:46, Nicolas Ecarnot wrote: Le 16/03/2018 à 13:28, Karli Sjöberg a écrit : Den 16 mars 2018 12:26 skrev Enrico Becchetti : Dear All, Does someone had seen that error ? Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space. Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin? All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx). Enrico, do you know if a bug has been filed about this? Did the VM remain paused? In my experience the VM just gets temporarily paused while the storage is expanded. RH confirmed to me in a ticket that this is expected behaviour. If you need high write performance your VM disks should always be preallocated. We only use Thin Provision for VMs where we know that disk writes are low (eg network services, CPU-bound apps, etc). Thanks a lot !!! Best Regards Enrico Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- ___ Enrico BecchettiServizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it __ ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
On Fri, Mar 16, 2018 at 1:25 PM, Enrico Becchetti < enrico.becche...@pg.infn.it> wrote: > Dear All, > Does someone had seen that error ? When I run this command from my virtual > machine: > > # time dd if=/dev/zero of=enrico.dd bs=4k count=1000 > I don't think it's a very interesting test case for IO performance, but in any case, it may cause the VM to try to write faster than its thin provisioned disk can be extended. A simple workaround would be to change in VDSM the threshold of when it gets extended and by how much. For example: [irs] volume_utilization_percent = 15 volume_utilization_chunk_mb = 4048 Y. > VM was paused due to kind a storage error/problem. Strange message > because tell about "no storage space error" but ovirt puts virtual machine > in > a paused state. > > Inside events from ovirt web interface I see this: > > "VM has been paused due to lack of storage space" > > but no ERROR found in /var/log/vdsm.log. > > My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before > now > I haven't see any other problem during the normal functioning of the vm , > it's seem > that this error occurs only when there is massive I/O. > > Any ideas ? > Thanks a lot. > Best Regards > Enrico > > > -- > ___ > > Enrico BecchettiServizio di Calcolo e Reti > > Istituto Nazionale di Fisica Nucleare - Sezione di Perugia > Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) > Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it > __ > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
Le 16/03/2018 à 15:48, Alex Crow a écrit : On 16/03/18 13:46, Nicolas Ecarnot wrote: Le 16/03/2018 à 13:28, Karli Sjöberg a écrit : Den 16 mars 2018 12:26 skrev Enrico Becchetti : Dear All, Does someone had seen that error ? Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space. Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin? All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx). Enrico, do you know if a bug has been filed about this? Did the VM remain paused? In my experience the VM just gets temporarily paused while the storage is expanded. RH confirmed to me in a ticket that this is expected behaviour. AFAIR, most of them went back up and running by themselves (we had to manually some of them from times to times). The storage side weakness is an interesting trail to follow. We also experienced this behavior when migrating lots of VMs at once, yet using a dedicated storage network. Being on this mailing list since long, I remember we already discussed several times about how some users feel how oVirt can appear sensitive to storage latencies. On my side, the site where most of our workload resides is still in 3.6, so I can not yet witness the efforts oVirt devs have made to cope with this in 4.2 but I'm sure they did. -- Nicolas ECARNOT ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
On 16/03/18 13:46, Nicolas Ecarnot wrote: Le 16/03/2018 à 13:28, Karli Sjöberg a écrit : Den 16 mars 2018 12:26 skrev Enrico Becchetti : Dear All, Does someone had seen that error ? Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space. Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin? All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx). Enrico, do you know if a bug has been filed about this? Did the VM remain paused? In my experience the VM just gets temporarily paused while the storage is expanded. RH confirmed to me in a ticket that this is expected behaviour. If you need high write performance your VM disks should always be preallocated. We only use Thin Provision for VMs where we know that disk writes are low (eg network services, CPU-bound apps, etc). Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
Le 16/03/2018 à 13:28, Karli Sjöberg a écrit : Den 16 mars 2018 12:26 skrev Enrico Becchetti : Dear All, Does someone had seen that error ? Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space. Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin? All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx). Enrico, do you know if a bug has been filed about this? -- Nicolas ECARNOT ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
yes ... it's a thin provisioning , in fact with preallocated disk type I haven't any problem. Thanks you so much Best Regards Enrico Il 16/03/2018 13:28, Karli Sjöberg ha scritto: Den 16 mars 2018 12:26 skrev Enrico Becchetti : Dear All, Does someone had seen that error ? When I run this command from my virtual machine: # time dd if=/dev/zero of=enrico.dd bs=4k count=1000 VM was paused due to kind a storage error/problem. Strange message because tell about "no storage space error" but ovirt puts virtual machine in a paused state. Inside events from ovirt web interface I see this: "VM has been paused due to lack of storage space" but no ERROR found in /var/log/vdsm.log. My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before now I haven't see any other problem during the normal functioning of the vm , it's seem that this error occurs only when there is massive I/O. Any ideas ? Thanks a lot. Best Regards Enrico -- ___ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it __ ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin? /K Dear All, Does someone had seen that error ? When I run this command from my virtual machine: # time dd if=/dev/zero of=enrico.dd bs=4k count=1000 VM was paused due to kind a storage error/problem. Strange message because tell about "no storage space error" but ovirt puts virtual machine in a paused state. Inside events from ovirt web interface I see this: "VM has been paused due to lack of storage space" but no ERROR found in /var/log/vdsm.log. My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before now I haven't see any other problem during the normal functioning of the vm , it's seem that this error occurs only when there is massive I/O. Any ideas ? Thanks a lot. Best Regards Enrico -- ___ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it __ ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- ___ Enrico BecchettiServizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it __ ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
Den 16 mars 2018 12:26 skrev Enrico Becchetti : Dear All,Does someone had seen that error ? When I run this command from my virtual machine:# time dd if=/dev/zero of=enrico.dd bs=4k count=1000VM was paused due to kind a storage error/problem. Strange messagebecause tell about "no storage space error" but ovirt puts virtual machine ina paused state.Inside events from ovirt web interface I see this:"VM has been paused due to lack of storage space"but no ERROR found in /var/log/vdsm.log.My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before nowI haven't see any other problem during the normal functioning of the vm , it's seemthat this error occurs only when there is massive I/O.Any ideas ?Thanks a lot.Best RegardsEnrico-- ___Enrico BecchettiServizio di Calcolo e RetiIstituto Nazionale di Fisica Nucleare - Sezione di PerugiaVia Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY)Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it_Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/usersI think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin?/K Dear All,Does someone had seen that error ? When I run this command from my virtual machine:# time dd if=/dev/zero of=enrico.dd bs=4k count=1000VM was paused due to kind a storage error/problem. Strange messagebecause tell about "no storage space error" but ovirt puts virtual machine ina paused state.Inside events from ovirt web interface I see this:"VM has been paused due to lack of storage space"but no ERROR found in /var/log/vdsm.log.My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before nowI haven't see any other problem during the normal functioning of the vm , it's seemthat this error occurs only when there is massive I/O.Any ideas ?Thanks a lot.Best RegardsEnrico-- ___Enrico BecchettiServizio di Calcolo e RetiIstituto Nazionale di Fisica Nucleare - Sezione di PerugiaVia Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY)Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it_Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
Dear All, Does someone had seen that error ? When I run this command from my virtual machine: # time dd if=/dev/zero of=enrico.dd bs=4k count=1000 VM was paused due to kind a storage error/problem. Strange message because tell about "no storage space error" but ovirt puts virtual machine in a paused state. Inside events from ovirt web interface I see this: "VM has been paused due to lack of storage space" but no ERROR found in /var/log/vdsm.log. My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before now I haven't see any other problem during the normal functioning of the vm , it's seem that this error occurs only when there is massive I/O. Any ideas ? Thanks a lot. Best Regards Enrico -- ___ Enrico BecchettiServizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it __ ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to no Storage space error.
On Thu, Apr 14, 2016 at 1:23 PM, wrote: > Hi Nir, > > El 2016-04-14 11:02, Nir Soffer escribió: >> >> On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland >> wrote: >>> >>> Nir, >>> See attached the repoplot output. >> >> >> So we have about one concurrent lvm command without any disk operations, >> and >> everything seems snappy. >> >> Nicolás, maybe this storage or the host is overloaded by the vms? Are your >> vms >> doing lot of io? >> > > Not that I know, actually it should have been a "calm" time slot as far as > IOs go, nor the storage was overloaded at that time. If I'm not mistaken, on > the repoplot report I see there are two LVM operations at a time, maybe that > has something to do with it? The operation that took about 50 seconds started in the same time that another operation started, but it does not explain why several other lvm comands took about 15 seconds each. > (although as you say, the lvextend is just a > metadata change...) > > >> lvextend operation should be very fast operation, this is just a >> metadata change, >> allocating couple of extents to that lv. >> >> Zdenek, how do you suggest to debug slow lvm commands? >> >> See the attached pdf, lvm commands took 15-50 seconds. >> >>> >>> On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer wrote: On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland wrote: > From the log, we can see that the lvextend command took 18 sec, which > is > quite long. Fred, can you run repoplot on this log file? it will may explain why this lvm call took 18 seconds. Nir > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' > devices { > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 > write_cache_state=0 disable_after_error_count=3 filter = [ > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', > '\''r|.*|'\'' > ] } > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' > --autobackup > n --size 6016m > > > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0 > (cwd None) > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: = ' > WARNING: lvmetad is running but disabled. Restart lvmetad before > enabling > it!\n WARNING: This metadata update is NOT backed up\n'; = 0 > > > The watermark can be configured by the following value: > > 'volume_utilization_percent', '50', > 'Together with volume_utilization_chunk_mb, set the minimal free ' > 'space before a thin provisioned block volume is extended. Use ' > 'lower values to extend earlier.') > > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek > wrote: >> >> >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote: >> > >> > Ok, that makes sense, thanks for the insight both Alex and Fred. >> > I'm >> > attaching the VDSM log of the SPM node at the time of the pause. I >> > couldn't >> > find anything that would clearly identify the problem, but maybe >> > you'll be >> > able to. >> >> In extreme conditions it will happen. When your storage is slow to >> respond >> to extension request, and when your write rate is very high then it >> may >> happen, as it is happening to you, that you run out space sooner than >> the >> extension finishes. You can change the watermark value I guess(right, >> Fred?), but better would be to plan a bit more ahead and either use >> preallocated or create thin and then allocate expected size in >> advance >> before the operation causing it (typically it only happens during >> untarring >> gigabytes of data, or huge database dump/restore) >> Even then, the VM should always be automatially resumed once the disk >> space is allocated >> >> Thanks, >> michal >> >> > >> > Thanks. >> > >> > Regards. >> > >> > El 2016-04-13 13:09, Fred Rolland escribió: >> >> Hi, >> >> Yes, just as Alex explained, if the disk has been created as thin >> >> provisioning, the vdsm will extends once a watermark is reached. >> >> Usually it should not get to the state the Vm is paused. >> >> From the log, you can see that the request for extension has been >> >> sent >> >> before the VM got to the No Space Error. >> >> Later, we can see the VM resuming. >> >> INFO::2016-04-13 >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requ
Re: [ovirt-users] VM has been paused due to no Storage space error.
Hi Nir, El 2016-04-14 11:02, Nir Soffer escribió: On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland wrote: Nir, See attached the repoplot output. So we have about one concurrent lvm command without any disk operations, and everything seems snappy. Nicolás, maybe this storage or the host is overloaded by the vms? Are your vms doing lot of io? Not that I know, actually it should have been a "calm" time slot as far as IOs go, nor the storage was overloaded at that time. If I'm not mistaken, on the repoplot report I see there are two LVM operations at a time, maybe that has something to do with it? (although as you say, the lvextend is just a metadata change...) lvextend operation should be very fast operation, this is just a metadata change, allocating couple of extents to that lv. Zdenek, how do you suggest to debug slow lvm commands? See the attached pdf, lvm commands took 15-50 seconds. On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer wrote: On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland wrote: > From the log, we can see that the lvextend command took 18 sec, which is > quite long. Fred, can you run repoplot on this log file? it will may explain why this lvm call took 18 seconds. Nir > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' > devices { > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 > write_cache_state=0 disable_after_error_count=3 filter = [ > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\'' > ] } > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' > --autobackup > n --size 6016m > > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0 > (cwd None) > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: = ' > WARNING: lvmetad is running but disabled. Restart lvmetad before > enabling > it!\n WARNING: This metadata update is NOT backed up\n'; = 0 > > > The watermark can be configured by the following value: > > 'volume_utilization_percent', '50', > 'Together with volume_utilization_chunk_mb, set the minimal free ' > 'space before a thin provisioned block volume is extended. Use ' > 'lower values to extend earlier.') > > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek > wrote: >> >> >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote: >> > >> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm >> > attaching the VDSM log of the SPM node at the time of the pause. I >> > couldn't >> > find anything that would clearly identify the problem, but maybe >> > you'll be >> > able to. >> >> In extreme conditions it will happen. When your storage is slow to >> respond >> to extension request, and when your write rate is very high then it may >> happen, as it is happening to you, that you run out space sooner than >> the >> extension finishes. You can change the watermark value I guess(right, >> Fred?), but better would be to plan a bit more ahead and either use >> preallocated or create thin and then allocate expected size in advance >> before the operation causing it (typically it only happens during >> untarring >> gigabytes of data, or huge database dump/restore) >> Even then, the VM should always be automatially resumed once the disk >> space is allocated >> >> Thanks, >> michal >> >> > >> > Thanks. >> > >> > Regards. >> > >> > El 2016-04-13 13:09, Fred Rolland escribió: >> >> Hi, >> >> Yes, just as Alex explained, if the disk has been created as thin >> >> provisioning, the vdsm will extends once a watermark is reached. >> >> Usually it should not get to the state the Vm is paused. >> >> From the log, you can see that the request for extension has been >> >> sent >> >> before the VM got to the No Space Error. >> >> Later, we can see the VM resuming. >> >> INFO::2016-04-13 >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension >> >> for >> >> volume >> >> >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device >> >> virtio-disk0 error enospc >> >> >> >> INFO::2016-04-13 >> >> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume >> >> Note that the extension is done on the SPM host, so it would be >> >> interesting to see the vdsm log from the host that was in SPM role >> >> at >> >> this timeframe. >> >> Regards, >> >> Fred >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow >> >> wrote: >> >>> Hi, >> >>> If you have set up VM disks as Thin Provisioned, the VM has to >> >>> pause when the disk image needs to expand. You won't see thi
Re: [ovirt-users] VM has been paused due to no Storage space error.
On Thu, Apr 14, 2016 at 12:38 PM, Fred Rolland wrote: > Nir, > See attached the repoplot output. So we have about one concurrent lvm command without any disk operations, and everything seems snappy. Nicolás, maybe this storage or the host is overloaded by the vms? Are your vms doing lot of io? lvextend operation should be very fast operation, this is just a metadata change, allocating couple of extents to that lv. Zdenek, how do you suggest to debug slow lvm commands? See the attached pdf, lvm commands took 15-50 seconds. > > On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer wrote: >> >> On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland >> wrote: >> > From the log, we can see that the lvextend command took 18 sec, which is >> > quite long. >> >> Fred, can you run repoplot on this log file? it will may explain why this >> lvm >> call took 18 seconds. >> >> Nir >> >> > >> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 >> > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset >> > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' >> > devices { >> > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 >> > write_cache_state=0 disable_after_error_count=3 filter = [ >> > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\'' >> > ] } >> > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 >> > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' >> > --autobackup >> > n --size 6016m >> > >> > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0 >> > (cwd None) >> > >> > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 >> > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: = ' >> > WARNING: lvmetad is running but disabled. Restart lvmetad before >> > enabling >> > it!\n WARNING: This metadata update is NOT backed up\n'; = 0 >> > >> > >> > The watermark can be configured by the following value: >> > >> > 'volume_utilization_percent', '50', >> > 'Together with volume_utilization_chunk_mb, set the minimal free ' >> > 'space before a thin provisioned block volume is extended. Use ' >> > 'lower values to extend earlier.') >> > >> > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek >> > wrote: >> >> >> >> >> >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote: >> >> > >> >> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm >> >> > attaching the VDSM log of the SPM node at the time of the pause. I >> >> > couldn't >> >> > find anything that would clearly identify the problem, but maybe >> >> > you'll be >> >> > able to. >> >> >> >> In extreme conditions it will happen. When your storage is slow to >> >> respond >> >> to extension request, and when your write rate is very high then it may >> >> happen, as it is happening to you, that you run out space sooner than >> >> the >> >> extension finishes. You can change the watermark value I guess(right, >> >> Fred?), but better would be to plan a bit more ahead and either use >> >> preallocated or create thin and then allocate expected size in advance >> >> before the operation causing it (typically it only happens during >> >> untarring >> >> gigabytes of data, or huge database dump/restore) >> >> Even then, the VM should always be automatially resumed once the disk >> >> space is allocated >> >> >> >> Thanks, >> >> michal >> >> >> >> > >> >> > Thanks. >> >> > >> >> > Regards. >> >> > >> >> > El 2016-04-13 13:09, Fred Rolland escribió: >> >> >> Hi, >> >> >> Yes, just as Alex explained, if the disk has been created as thin >> >> >> provisioning, the vdsm will extends once a watermark is reached. >> >> >> Usually it should not get to the state the Vm is paused. >> >> >> From the log, you can see that the request for extension has been >> >> >> sent >> >> >> before the VM got to the No Space Error. >> >> >> Later, we can see the VM resuming. >> >> >> INFO::2016-04-13 >> >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) >> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension >> >> >> for >> >> >> volume >> >> >> >> >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) >> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device >> >> >> virtio-disk0 error enospc >> >> >> >> >> >> INFO::2016-04-13 >> >> >> 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) >> >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume >> >> >> Note that the extension is done on the SPM host, so it would be >> >> >> interesting to see the vdsm log from the host that was in SPM role >> >> >> at >> >> >> this timeframe. >> >> >> Regards, >> >> >> Fred >> >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow >> >> >> wrote: >> >> >>> Hi, >> >> >>> If you have set up VM disks as Thin Provisioned, the VM has to >> >> >>> pause when the disk image needs to expand. You won't see this on >> >> >>> VMs >> >> >>> with preallocated storage. >> >
Re: [ovirt-users] VM has been paused due to no Storage space error.
Nir, See attached the repoplot output. On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer wrote: > On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland > wrote: > > From the log, we can see that the lvextend command took 18 sec, which is > > quite long. > > Fred, can you run repoplot on this log file? it will may explain why this > lvm > call took 18 seconds. > > Nir > > > > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset > > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' > devices { > > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 > > write_cache_state=0 disable_after_error_count=3 filter = [ > > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\'' > ] } > > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 > > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' > --autobackup > > n --size 6016m > > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0 > > (cwd None) > > > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: = ' > > WARNING: lvmetad is running but disabled. Restart lvmetad before enabling > > it!\n WARNING: This metadata update is NOT backed up\n'; = 0 > > > > > > The watermark can be configured by the following value: > > > > 'volume_utilization_percent', '50', > > 'Together with volume_utilization_chunk_mb, set the minimal free ' > > 'space before a thin provisioned block volume is extended. Use ' > > 'lower values to extend earlier.') > > > > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek > > wrote: > >> > >> > >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote: > >> > > >> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm > >> > attaching the VDSM log of the SPM node at the time of the pause. I > couldn't > >> > find anything that would clearly identify the problem, but maybe > you'll be > >> > able to. > >> > >> In extreme conditions it will happen. When your storage is slow to > respond > >> to extension request, and when your write rate is very high then it may > >> happen, as it is happening to you, that you run out space sooner than > the > >> extension finishes. You can change the watermark value I guess(right, > >> Fred?), but better would be to plan a bit more ahead and either use > >> preallocated or create thin and then allocate expected size in advance > >> before the operation causing it (typically it only happens during > untarring > >> gigabytes of data, or huge database dump/restore) > >> Even then, the VM should always be automatially resumed once the disk > >> space is allocated > >> > >> Thanks, > >> michal > >> > >> > > >> > Thanks. > >> > > >> > Regards. > >> > > >> > El 2016-04-13 13:09, Fred Rolland escribió: > >> >> Hi, > >> >> Yes, just as Alex explained, if the disk has been created as thin > >> >> provisioning, the vdsm will extends once a watermark is reached. > >> >> Usually it should not get to the state the Vm is paused. > >> >> From the log, you can see that the request for extension has been > sent > >> >> before the VM got to the No Space Error. > >> >> Later, we can see the VM resuming. > >> >> INFO::2016-04-13 > >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) > >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for > >> >> volume > >> >> > >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) > >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device > >> >> virtio-disk0 error enospc > >> >> > >> >> INFO::2016-04-13 > 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) > >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume > >> >> Note that the extension is done on the SPM host, so it would be > >> >> interesting to see the vdsm log from the host that was in SPM role at > >> >> this timeframe. > >> >> Regards, > >> >> Fred > >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow > >> >> wrote: > >> >>> Hi, > >> >>> If you have set up VM disks as Thin Provisioned, the VM has to > >> >>> pause when the disk image needs to expand. You won't see this on VMs > >> >>> with preallocated storage. > >> >>> It's not the SAN that's running out of space, it's the VM image > >> >>> needing to be expanded incrementally each time. > >> >>> Cheers > >> >>> Alex > >> >>> On 13/04/16 12:04, nico...@devels.es wrote: > >> >>> Hi Fred, > >> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host > >> >>> where this machine has been running. Should you need any further > >> >>> info, don't hesitate to ask. > >> >>> Thanks. > >> >>> Regards. > >> >>> El 2016-04-13 11:54, Fred Rolland escribió: > >> >>> Hi, > >> >>> What kind of storage do you have ? (ISCSI,FC,NFS...) > >> >>> Can you provide the vdsm logs from the host where this VM runs ? > >> >>> Thanks, > >> >>> Fred
Re: [ovirt-users] VM has been paused due to no Storage space error.
On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland wrote: > From the log, we can see that the lvextend command took 18 sec, which is > quite long. Fred, can you run repoplot on this log file? it will may explain why this lvm call took 18 seconds. Nir > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset > --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' devices { > preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 > write_cache_state=0 disable_after_error_count=3 filter = [ > '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\'' ] } > global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 > use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --autobackup > n --size 6016m > 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0 > (cwd None) > > 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 > 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: = ' > WARNING: lvmetad is running but disabled. Restart lvmetad before enabling > it!\n WARNING: This metadata update is NOT backed up\n'; = 0 > > > The watermark can be configured by the following value: > > 'volume_utilization_percent', '50', > 'Together with volume_utilization_chunk_mb, set the minimal free ' > 'space before a thin provisioned block volume is extended. Use ' > 'lower values to extend earlier.') > > On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek > wrote: >> >> >> > On 14 Apr 2016, at 09:57, nico...@devels.es wrote: >> > >> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm >> > attaching the VDSM log of the SPM node at the time of the pause. I couldn't >> > find anything that would clearly identify the problem, but maybe you'll be >> > able to. >> >> In extreme conditions it will happen. When your storage is slow to respond >> to extension request, and when your write rate is very high then it may >> happen, as it is happening to you, that you run out space sooner than the >> extension finishes. You can change the watermark value I guess(right, >> Fred?), but better would be to plan a bit more ahead and either use >> preallocated or create thin and then allocate expected size in advance >> before the operation causing it (typically it only happens during untarring >> gigabytes of data, or huge database dump/restore) >> Even then, the VM should always be automatially resumed once the disk >> space is allocated >> >> Thanks, >> michal >> >> > >> > Thanks. >> > >> > Regards. >> > >> > El 2016-04-13 13:09, Fred Rolland escribió: >> >> Hi, >> >> Yes, just as Alex explained, if the disk has been created as thin >> >> provisioning, the vdsm will extends once a watermark is reached. >> >> Usually it should not get to the state the Vm is paused. >> >> From the log, you can see that the request for extension has been sent >> >> before the VM got to the No Space Error. >> >> Later, we can see the VM resuming. >> >> INFO::2016-04-13 >> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for >> >> volume >> >> >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device >> >> virtio-disk0 error enospc >> >> >> >> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) >> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume >> >> Note that the extension is done on the SPM host, so it would be >> >> interesting to see the vdsm log from the host that was in SPM role at >> >> this timeframe. >> >> Regards, >> >> Fred >> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow >> >> wrote: >> >>> Hi, >> >>> If you have set up VM disks as Thin Provisioned, the VM has to >> >>> pause when the disk image needs to expand. You won't see this on VMs >> >>> with preallocated storage. >> >>> It's not the SAN that's running out of space, it's the VM image >> >>> needing to be expanded incrementally each time. >> >>> Cheers >> >>> Alex >> >>> On 13/04/16 12:04, nico...@devels.es wrote: >> >>> Hi Fred, >> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host >> >>> where this machine has been running. Should you need any further >> >>> info, don't hesitate to ask. >> >>> Thanks. >> >>> Regards. >> >>> El 2016-04-13 11:54, Fred Rolland escribió: >> >>> Hi, >> >>> What kind of storage do you have ? (ISCSI,FC,NFS...) >> >>> Can you provide the vdsm logs from the host where this VM runs ? >> >>> Thanks, >> >>> Freddy >> >>> On Wed, Apr 13, 2016 at 1:02 PM, wrote: >> >>> Hi, >> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of >> >>> events like these: >> >>> 2016-04-13 10:52:30,735 INFO >> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer] >> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM >> >>> 'f9cd282e-110a-4896-98d3-6d3206
Re: [ovirt-users] VM has been paused due to no Storage space error.
>From the log, we can see that the lvextend command took 18 sec, which is quite long. 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ '\''a|/dev/mapper/36000eb3a4f1acbc20043|'\'', '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --autobackup n --size 6016m 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0 (cwd None) 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n WARNING: This metadata update is NOT backed up\n'; = 0 The watermark can be configured by the following value: 'volume_utilization_percent', '50', 'Together with volume_utilization_chunk_mb, set the minimal free ' 'space before a thin provisioned block volume is extended. Use ' 'lower values to extend earlier.') On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek < michal.skriva...@redhat.com> wrote: > > > On 14 Apr 2016, at 09:57, nico...@devels.es wrote: > > > > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm > attaching the VDSM log of the SPM node at the time of the pause. I couldn't > find anything that would clearly identify the problem, but maybe you'll be > able to. > > In extreme conditions it will happen. When your storage is slow to respond > to extension request, and when your write rate is very high then it may > happen, as it is happening to you, that you run out space sooner than the > extension finishes. You can change the watermark value I guess(right, > Fred?), but better would be to plan a bit more ahead and either use > preallocated or create thin and then allocate expected size in advance > before the operation causing it (typically it only happens during untarring > gigabytes of data, or huge database dump/restore) > Even then, the VM should always be automatially resumed once the disk > space is allocated > > Thanks, > michal > > > > > Thanks. > > > > Regards. > > > > El 2016-04-13 13:09, Fred Rolland escribió: > >> Hi, > >> Yes, just as Alex explained, if the disk has been created as thin > >> provisioning, the vdsm will extends once a watermark is reached. > >> Usually it should not get to the state the Vm is paused. > >> From the log, you can see that the request for extension has been sent > >> before the VM got to the No Space Error. > >> Later, we can see the VM resuming. > >> INFO::2016-04-13 > >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) > >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for > >> volume > >> > >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) > >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device > >> virtio-disk0 error enospc > >> > >> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) > >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume > >> Note that the extension is done on the SPM host, so it would be > >> interesting to see the vdsm log from the host that was in SPM role at > >> this timeframe. > >> Regards, > >> Fred > >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow > >> wrote: > >>> Hi, > >>> If you have set up VM disks as Thin Provisioned, the VM has to > >>> pause when the disk image needs to expand. You won't see this on VMs > >>> with preallocated storage. > >>> It's not the SAN that's running out of space, it's the VM image > >>> needing to be expanded incrementally each time. > >>> Cheers > >>> Alex > >>> On 13/04/16 12:04, nico...@devels.es wrote: > >>> Hi Fred, > >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host > >>> where this machine has been running. Should you need any further > >>> info, don't hesitate to ask. > >>> Thanks. > >>> Regards. > >>> El 2016-04-13 11:54, Fred Rolland escribió: > >>> Hi, > >>> What kind of storage do you have ? (ISCSI,FC,NFS...) > >>> Can you provide the vdsm logs from the host where this VM runs ? > >>> Thanks, > >>> Freddy > >>> On Wed, Apr 13, 2016 at 1:02 PM, wrote: > >>> Hi, > >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of > >>> events like these: > >>> 2016-04-13 10:52:30,735 INFO > >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer] > >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM > >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved > >>> from > >>> 'Up' --> 'Paused' > >>> 2016-04-13 10:52:30,815 INFO > >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, > >>> Call Stack:
Re: [ovirt-users] VM has been paused due to no Storage space error.
> On 14 Apr 2016, at 09:57, nico...@devels.es wrote: > > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm > attaching the VDSM log of the SPM node at the time of the pause. I couldn't > find anything that would clearly identify the problem, but maybe you'll be > able to. In extreme conditions it will happen. When your storage is slow to respond to extension request, and when your write rate is very high then it may happen, as it is happening to you, that you run out space sooner than the extension finishes. You can change the watermark value I guess(right, Fred?), but better would be to plan a bit more ahead and either use preallocated or create thin and then allocate expected size in advance before the operation causing it (typically it only happens during untarring gigabytes of data, or huge database dump/restore) Even then, the VM should always be automatially resumed once the disk space is allocated Thanks, michal > > Thanks. > > Regards. > > El 2016-04-13 13:09, Fred Rolland escribió: >> Hi, >> Yes, just as Alex explained, if the disk has been created as thin >> provisioning, the vdsm will extends once a watermark is reached. >> Usually it should not get to the state the Vm is paused. >> From the log, you can see that the request for extension has been sent >> before the VM got to the No Space Error. >> Later, we can see the VM resuming. >> INFO::2016-04-13 >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for >> volume >> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device >> virtio-disk0 error enospc >> >> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume >> Note that the extension is done on the SPM host, so it would be >> interesting to see the vdsm log from the host that was in SPM role at >> this timeframe. >> Regards, >> Fred >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow >> wrote: >>> Hi, >>> If you have set up VM disks as Thin Provisioned, the VM has to >>> pause when the disk image needs to expand. You won't see this on VMs >>> with preallocated storage. >>> It's not the SAN that's running out of space, it's the VM image >>> needing to be expanded incrementally each time. >>> Cheers >>> Alex >>> On 13/04/16 12:04, nico...@devels.es wrote: >>> Hi Fred, >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host >>> where this machine has been running. Should you need any further >>> info, don't hesitate to ask. >>> Thanks. >>> Regards. >>> El 2016-04-13 11:54, Fred Rolland escribió: >>> Hi, >>> What kind of storage do you have ? (ISCSI,FC,NFS...) >>> Can you provide the vdsm logs from the host where this VM runs ? >>> Thanks, >>> Freddy >>> On Wed, Apr 13, 2016 at 1:02 PM, wrote: >>> Hi, >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of >>> events like these: >>> 2016-04-13 10:52:30,735 INFO >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer] >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved >>> from >>> 'Up' --> 'Paused' >>> 2016-04-13 10:52:30,815 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com >>> [1] [1] >>> has been paused. >>> 2016-04-13 10:52:30,898 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com >>> [1] [1] >>> has been paused due to no Storage space error. >>> 2016-04-13 10:52:52,320 WARN >>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] >>> (org.ovirt.thread.pool-8-thread-38) [] domain >>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: >>> 'host6.domain.com [2] [2]' >>> 2016-04-13 10:52:55,183 INFO >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer] >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved >>> from >>> 'Paused' --> 'Up' >>> 2016-04-13 10:52:55,318 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, >>> Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com >>> [1] [1] >>> has recovered from paused back to up. >>> The storage domain is far from being full, though (400+ G available >>> right now). Could this be related to this other issue [1]? If not, >>> how could I debug what's going on? >>> Thanks. >>> [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html >>> [3] >>> [3] >>> ___
Re: [ovirt-users] VM has been paused due to no Storage space error.
Ok, that makes sense, thanks for the insight both Alex and Fred. I'm attaching the VDSM log of the SPM node at the time of the pause. I couldn't find anything that would clearly identify the problem, but maybe you'll be able to. Thanks. Regards. El 2016-04-13 13:09, Fred Rolland escribió: Hi, Yes, just as Alex explained, if the disk has been created as thin provisioning, the vdsm will extends once a watermark is reached. Usually it should not get to the state the Vm is paused. From the log, you can see that the request for extension has been sent before the VM got to the No Space Error. Later, we can see the VM resuming. INFO::2016-04-13 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for volume INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device virtio-disk0 error enospc INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume Note that the extension is done on the SPM host, so it would be interesting to see the vdsm log from the host that was in SPM role at this timeframe. Regards, Fred On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow wrote: Hi, If you have set up VM disks as Thin Provisioned, the VM has to pause when the disk image needs to expand. You won't see this on VMs with preallocated storage. It's not the SAN that's running out of space, it's the VM image needing to be expanded incrementally each time. Cheers Alex On 13/04/16 12:04, nico...@devels.es wrote: Hi Fred, This is an iSCSI storage. I'm attaching the VDSM logs from the host where this machine has been running. Should you need any further info, don't hesitate to ask. Thanks. Regards. El 2016-04-13 11:54, Fred Rolland escribió: Hi, What kind of storage do you have ? (ISCSI,FC,NFS...) Can you provide the vdsm logs from the host where this VM runs ? Thanks, Freddy On Wed, Apr 13, 2016 at 1:02 PM, wrote: Hi, We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events like these: 2016-04-13 10:52:30,735 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-86) [60dea18f] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved from 'Up' --> 'Paused' 2016-04-13 10:52:30,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] [1] has been paused. 2016-04-13 10:52:30,898 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] [1] has been paused due to no Storage space error. 2016-04-13 10:52:52,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [] domain '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: 'host6.domain.com [2] [2]' 2016-04-13 10:52:55,183 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1] [1]) moved from 'Paused' --> 'Up' 2016-04-13 10:52:55,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] [1] has recovered from paused back to up. The storage domain is far from being full, though (400+ G available right now). Could this be related to this other issue [1]? If not, how could I debug what's going on? Thanks. [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html [3] [3] ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users [4] [4] Links: -- [1] http://vm.domain.com [1] [2] http://host6.domain.com [2] [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [3] [4] http://lists.ovirt.org/mailman/listinfo/users [4] ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users [4] -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, Lon
Re: [ovirt-users] VM has been paused due to no Storage space error.
Hi, Yes, just as Alex explained, if the disk has been created as thin provisioning, the vdsm will extends once a watermark is reached. Usually it should not get to the state the Vm is paused. >From the log, you can see that the request for extension has been sent before the VM got to the No Space Error. Later, we can see the VM resuming. INFO::2016-04-13 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded) vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for volume INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError) vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device virtio-disk0 error enospc INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus) vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume Note that the extension is done on the SPM host, so it would be interesting to see the vdsm log from the host that was in SPM role at this timeframe. Regards, Fred On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow wrote: > Hi, > > If you have set up VM disks as Thin Provisioned, the VM has to pause when > the disk image needs to expand. You won't see this on VMs with preallocated > storage. > > It's not the SAN that's running out of space, it's the VM image needing to > be expanded incrementally each time. > > Cheers > > Alex > > > On 13/04/16 12:04, nico...@devels.es wrote: > > Hi Fred, > > This is an iSCSI storage. I'm attaching the VDSM logs from the host where > this machine has been running. Should you need any further info, don't > hesitate to ask. > > Thanks. > > Regards. > > El 2016-04-13 11:54, Fred Rolland escribió: > > Hi, > > What kind of storage do you have ? (ISCSI,FC,NFS...) > Can you provide the vdsm logs from the host where this VM runs ? > > Thanks, > > Freddy > > On Wed, Apr 13, 2016 at 1:02 PM, > wrote: > > Hi, > > We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of > events like these: > > 2016-04-13 10:52:30,735 INFO > [org.ovirt.engine.core.vdsbroker.VmAnalyzer] > (DefaultQuartzScheduler_Worker-86) [60dea18f] VM > 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from > 'Up' --> 'Paused' > 2016-04-13 10:52:30,815 INFO > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, > Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] > has been paused. > 2016-04-13 10:52:30,898 ERROR > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, > Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] > has been paused due to no Storage space error. > 2016-04-13 10:52:52,320 WARN > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] > (org.ovirt.thread.pool-8-thread-38) [] domain > '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: > 'host6.domain.com [2]' > 2016-04-13 10:52:55,183 INFO > [org.ovirt.engine.core.vdsbroker.VmAnalyzer] > (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM > 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from > 'Paused' --> 'Up' > 2016-04-13 10:52:55,318 INFO > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, > Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] > has recovered from paused back to up. > > The storage domain is far from being full, though (400+ G available > right now). Could this be related to this other issue [1]? If not, > how could I debug what's going on? > > Thanks. > > [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html > [3] > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users [4] > > > > > Links: > -- > [1] http://vm.domain.com > [2] http://host6.domain.com > [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html > [4] http://lists.ovirt.org/mailman/listinfo/users > > > > ___ > Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users > > > > -- > This message is intended only for the addressee and may contain > confidential information. Unless you are that person, you may not > disclose its contents or use it in any way and are requested to delete > the message along with any attachments and notify us immediately. > This email is not intended to, nor should it be taken to, constitute advice. > The information provided is correct to our knowledge & belief and must not > be used as a substitute for obtaining tax, regulatory, investment, legal or > any other appropriate advice. > > "Transact" is operated by Integrated Financial Arrangements Ltd. > 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. > (Registered office: as above; Registered in England and Wales under > number: 3727592). Au
Re: [ovirt-users] VM has been paused due to no Storage space error.
Ahh, we've seen this as well in RHEV and have wondered what was going on. A better message would be good. On Wed, Apr 13, 2016 at 7:43 PM, Alex Crow wrote: > Hi, > > If you have set up VM disks as Thin Provisioned, the VM has to pause when > the disk image needs to expand. You won't see this on VMs with preallocated > storage. > > It's not the SAN that's running out of space, it's the VM image needing to > be expanded incrementally each time. > > Cheers > > Alex > > > On 13/04/16 12:04, nico...@devels.es wrote: > > Hi Fred, > > This is an iSCSI storage. I'm attaching the VDSM logs from the host where > this machine has been running. Should you need any further info, don't > hesitate to ask. > > Thanks. > > Regards. > > El 2016-04-13 11:54, Fred Rolland escribió: > > Hi, > > What kind of storage do you have ? (ISCSI,FC,NFS...) > Can you provide the vdsm logs from the host where this VM runs ? > > Thanks, > > Freddy > > On Wed, Apr 13, 2016 at 1:02 PM, > wrote: > > Hi, > > We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of > events like these: > > 2016-04-13 10:52:30,735 INFO > [org.ovirt.engine.core.vdsbroker.VmAnalyzer] > (DefaultQuartzScheduler_Worker-86) [60dea18f] VM > 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from > 'Up' --> 'Paused' > 2016-04-13 10:52:30,815 INFO > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, > Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] > has been paused. > 2016-04-13 10:52:30,898 ERROR > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, > Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] > has been paused due to no Storage space error. > 2016-04-13 10:52:52,320 WARN > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] > (org.ovirt.thread.pool-8-thread-38) [] domain > '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: > 'host6.domain.com [2]' > 2016-04-13 10:52:55,183 INFO > [org.ovirt.engine.core.vdsbroker.VmAnalyzer] > (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM > 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from > 'Paused' --> 'Up' > 2016-04-13 10:52:55,318 INFO > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, > Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] > has recovered from paused back to up. > > The storage domain is far from being full, though (400+ G available > right now). Could this be related to this other issue [1]? If not, > how could I debug what's going on? > > Thanks. > > [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html > [3] > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users [4] > > > > > Links: > -- > [1] http://vm.domain.com > [2] http://host6.domain.com > [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html > [4] http://lists.ovirt.org/mailman/listinfo/users > > > > ___ > Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users > > > > -- > This message is intended only for the addressee and may contain > confidential information. Unless you are that person, you may not > disclose its contents or use it in any way and are requested to delete > the message along with any attachments and notify us immediately. > This email is not intended to, nor should it be taken to, constitute advice. > The information provided is correct to our knowledge & belief and must not > be used as a substitute for obtaining tax, regulatory, investment, legal or > any other appropriate advice. > > "Transact" is operated by Integrated Financial Arrangements Ltd. > 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. > (Registered office: as above; Registered in England and Wales under > number: 3727592). Authorised and regulated by the Financial Conduct > Authority (entered on the Financial Services Register; no. 190856). > > > > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to no Storage space error.
Hi, If you have set up VM disks as Thin Provisioned, the VM has to pause when the disk image needs to expand. You won't see this on VMs with preallocated storage. It's not the SAN that's running out of space, it's the VM image needing to be expanded incrementally each time. Cheers Alex On 13/04/16 12:04, nico...@devels.es wrote: Hi Fred, This is an iSCSI storage. I'm attaching the VDSM logs from the host where this machine has been running. Should you need any further info, don't hesitate to ask. Thanks. Regards. El 2016-04-13 11:54, Fred Rolland escribió: Hi, What kind of storage do you have ? (ISCSI,FC,NFS...) Can you provide the vdsm logs from the host where this VM runs ? Thanks, Freddy On Wed, Apr 13, 2016 at 1:02 PM, wrote: Hi, We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events like these: 2016-04-13 10:52:30,735 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-86) [60dea18f] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from 'Up' --> 'Paused' 2016-04-13 10:52:30,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] has been paused. 2016-04-13 10:52:30,898 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] has been paused due to no Storage space error. 2016-04-13 10:52:52,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [] domain '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: 'host6.domain.com [2]' 2016-04-13 10:52:55,183 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from 'Paused' --> 'Up' 2016-04-13 10:52:55,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] has recovered from paused back to up. The storage domain is far from being full, though (400+ G available right now). Could this be related to this other issue [1]? If not, how could I debug what's going on? Thanks. [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html [3] ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users [4] Links: -- [1] http://vm.domain.com [2] http://host6.domain.com [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [4] http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to no Storage space error.
Hi Fred, This is an iSCSI storage. I'm attaching the VDSM logs from the host where this machine has been running. Should you need any further info, don't hesitate to ask. Thanks. Regards. El 2016-04-13 11:54, Fred Rolland escribió: Hi, What kind of storage do you have ? (ISCSI,FC,NFS...) Can you provide the vdsm logs from the host where this VM runs ? Thanks, Freddy On Wed, Apr 13, 2016 at 1:02 PM, wrote: Hi, We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events like these: 2016-04-13 10:52:30,735 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-86) [60dea18f] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from 'Up' --> 'Paused' 2016-04-13 10:52:30,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] has been paused. 2016-04-13 10:52:30,898 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] has been paused due to no Storage space error. 2016-04-13 10:52:52,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [] domain '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: 'host6.domain.com [2]' 2016-04-13 10:52:55,183 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com [1]) moved from 'Paused' --> 'Up' 2016-04-13 10:52:55,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com [1] has recovered from paused back to up. The storage domain is far from being full, though (400+ G available right now). Could this be related to this other issue [1]? If not, how could I debug what's going on? Thanks. [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html [3] ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users [4] Links: -- [1] http://vm.domain.com [2] http://host6.domain.com [3] https://www.mail-archive.com/users@ovirt.org/msg32079.html [4] http://lists.ovirt.org/mailman/listinfo/users vdsm.log.gz Description: GNU Zip compressed data ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM has been paused due to no Storage space error.
Hi, What kind of storage do you have ? (ISCSI,FC,NFS...) Can you provide the vdsm logs from the host where this VM runs ? Thanks, Freddy On Wed, Apr 13, 2016 at 1:02 PM, wrote: > Hi, > > We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events like > these: > > 2016-04-13 10:52:30,735 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] > (DefaultQuartzScheduler_Worker-86) [60dea18f] VM > 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Up' --> > 'Paused' > 2016-04-13 10:52:30,815 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call > Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been > paused. > 2016-04-13 10:52:30,898 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call > Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been > paused due to no Storage space error. > 2016-04-13 10:52:52,320 WARN > [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] > (org.ovirt.thread.pool-8-thread-38) [] domain > '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: ' > host6.domain.com' > 2016-04-13 10:52:55,183 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] > (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM > 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Paused' > --> 'Up' > 2016-04-13 10:52:55,318 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call > Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has recovered > from paused back to up. > > The storage domain is far from being full, though (400+ G available right > now). Could this be related to this other issue [1]? If not, how could I > debug what's going on? > > Thanks. > > [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] VM has been paused due to no Storage space error.
Hi, We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of events like these: 2016-04-13 10:52:30,735 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-86) [60dea18f] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Up' --> 'Paused' 2016-04-13 10:52:30,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been paused. 2016-04-13 10:52:30,898 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has been paused due to no Storage space error. 2016-04-13 10:52:52,320 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-8-thread-38) [] domain '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds: 'host6.domain.com' 2016-04-13 10:52:55,183 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM 'f9cd282e-110a-4896-98d3-6d320662744d'(vm.domain.com) moved from 'Paused' --> 'Up' 2016-04-13 10:52:55,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM vm.domain.com has recovered from paused back to up. The storage domain is far from being full, though (400+ G available right now). Could this be related to this other issue [1]? If not, how could I debug what's going on? Thanks. [1]: https://www.mail-archive.com/users@ovirt.org/msg32079.html ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users