Based on this (I'm no developer...) https://github.com/apache/cloudstack/blob/4.11/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L69
...it seems like CloudStack could not fetch free mem from libvirt - i.e. I would check libvirt logs (you can also try to spin a VM manually via virsh or Virtual Machine Managed) - or simply restart libvirt daemon. Could you make a conclusion if this error always happens on some particular host(s) - libvirt is known to get stuck on very rare occasions - so a restart would help. BTW - you seems to be reaching 1.000.000 VMs created in total ??? If so - that's a nice number :) Cheers Andrija On Thu, 25 Jul 2019 at 17:10, Dmitry Berezhnoy <d.berezh...@emzior.ru> wrote: > *4.11.2.0 > > чт, 25 июл. 2019 г. в 18:09, Dmitry Berezhnoy <d.berezh...@emzior.ru>: > > > First log messages from 10.11.2.0 > > > > чт, 25 июл. 2019 г. в 17:56, Andrija Panic <andrija.pa...@gmail.com>: > > > >> FYI,4.10 is so broken that it's not considered an official releases > (check > >> yourself on the GitHub) - please upgrade to 4.11.3 if possible. > >> > >> Will check logs later today. > >> > >> Andrija > >> > >> On Thu, 25 Jul 2019 at 16:48, Dmitry Berezhnoy <d.berezh...@emzior.ru> > >> wrote: > >> > >> > Log from that CloudStack station really huge, so I download log from > >> > another CloudStack(version 4.10.0) with the same errors. > logid:f0c4abda > >> for > >> > example. > >> > > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl] > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7 > >> > FirstFitRoutingAllocator) (logid:f0c4abda) Free RAM: 88188272640 , > >> > Requested RAM: 402653184 > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl] > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7 > >> > FirstFitRoutingAllocator) (logid:f0c4abda) Host has enough CPU and RAM > >> > available > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl] > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7 > >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc CPU from > >> host: > >> > 1, used: 48556, reserved: 0, actual total: 144000, total with > >> > overprovisioning: 144000; requested > cpu:384,alloc_from_last_host?:false > >> > ,considerReservedCapacity?: true > >> > 2019-07-25 15:57:58,895 DEBUG [c.c.c.CapacityManagerImpl] > >> > (API-Job-Executor-149:ctx-d621076d job-6654658 ctx-d5e8eff7 > >> > FirstFitRoutingAllocator) (logid:f0c4abda) STATS: Can alloc MEM from > >> host: > >> > 1, used: 45902462976, reserved: 0, total: 134090735616; requested mem: > >> > 402653184,alloc_from_last_host?:false ,considerReservedCapacity?: true > >> > ... > >> > 2019-07-25 15:58:07,660 INFO [c.c.v.VirtualMachineManagerImpl] > >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 > ctx-fe95df77) > >> > (logid:f0c4abda) Unable to start VM on Host[-1-Routing] due to failed > to > >> > get free memory > >> > ... > >> > 2019-07-25 15:58:14,421 ERROR [c.c.v.VmWorkJobHandlerProxy] > >> > (Work-Job-Executor-27:ctx-bce5a678 job-6654658/job-6654659 > ctx-fe95df77) > >> > (logid:f0c4abda) Invocation exception, caused by: > >> > com.cloud.exception.InsufficientServerCapacityException: Unable to > >> create a > >> > deployment for VM[User|i-2-977270-VM]Scope=interface > >> > com.cloud.dc.DataCenter; id=3 > >> > > >> > management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ > >> > > >> > чт, 25 июл. 2019 г. в 16:51, Andrija Panic <andrija.pa...@gmail.com>: > >> > > >> > > Can you share the whole log file (pastebin.org please, or similar) > ? > >> > > > >> > > On Wed, 24 Jul 2019 at 17:34, Dmitry Berezhnoy < > d.berezh...@emzior.ru > >> > > >> > > wrote: > >> > > > >> > > > Hello, > >> > > > > >> > > > Asynchronous creating VM bring to "Unable to start VM on > >> > Host[-1-Routing] > >> > > > due to failed to get free memory". After that I see ERROR: > >> > > > InsufficientServerCapacityException. > >> > > > In previous messages I see large amount of free resources: > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl] > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 > >> > ctx-52b48ca6) > >> > > > (logid:b6e33d80) Current Used CPU: 115708 , Free CPU:76356 > >> ,Requested > >> > > CPU: > >> > > > 512 > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl] > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 > >> > ctx-52b48ca6) > >> > > > (logid:b6e33d80) Current Used RAM: 122003914752 , Free > >> RAM:145667670016 > >> > > > ,Requested RAM: 536870912 > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl] > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 > >> > ctx-52b48ca6) > >> > > > (logid:b6e33d80) CPU STATS after allocation: for host: 1, old > used: > >> > > 115708, > >> > > > old reserved: 1536, actual total: 193600, total with > >> overprovisioning: > >> > > > 193600; new used:116220, reserved:1536; requested > >> > > > cpu:512,alloc_from_last:false > >> > > > 2019-07-24 04:50:45,443 DEBUG [c.c.c.CapacityManagerImpl] > >> > > > (Work-Job-Executor-82:ctx-8d1155a3 job-2281443/job-2281455 > >> > ctx-52b48ca6) > >> > > > (logid:b6e33d80) RAM STATS after allocation: for host: 1, old > used: > >> > > > 122003914752, old reserved: 1610612736, total: 269282197504; new > >> used: > >> > > > 122540785664, reserved: 1610612736; requested mem: > >> > > > 536870912,alloc_from_last:false > >> > > > > >> > > > How it possible? > >> > > > > >> > > > Thanks in advance for the help, > >> > > > Dmitry. > >> > > > > >> > > > >> > > > >> > > -- > >> > > > >> > > Andrija Panić > >> > > > >> > > >> > >> > >> -- > >> > >> Andrija Panić > >> > > > -- Andrija Panić
