[ovirt-users] Re: VM migrations failing after upgrade 4.3.2 -> 4.3.3

2019-04-17 Thread Eduardo Mayoral 
You are right. All VMs on that cluster have empty values for "Memory",
"CPU" and "Network"

I tried restarting vdsmd on one of the hosts and now I see values for
"Memory", "CPU" and "Network" on the VMs running on that host.

Since I cannot set the hosts on maintenance beacuse I cannot migrate the
VMs inside I will have to restart vdsmd on the host and cross my fingers...

Thank you very much for your help!


On 17/4/19 11:16, Andrej Krejcir wrote:
> Hi,
>
> this is a bug:  https://bugzilla.redhat.com/show_bug.cgi?id=1696621
>
> It happens when a migrated VM has missing CPU load.
> Do you have some special VMs without updated CPU load, or VMs that
> were recently started? 
>
>
> Regards,
> Andrej
>
> On Wed, 17 Apr 2019 at 11:08, Eduardo Mayoral  > wrote:
>
>
> Hi,
>
>     After upgrade to 4.3.3 from 4.3.2, VM migrations are failing with
> "No available host to migrate VMs to"
>
>     Interestingly, this happens in one of our clusters, not on the
> other. Both on the same datacenter, both in 4.3 compatibility version.
>
>     This are the relevant lines from /var/log/ovirt-engine/engine.log
>
>
>
>
> 2019-04-17 08:58:41,816Z ERROR
> [org.ovirt.engine.core.bll.GetValidHostsForVmsQuery] (default task-7)
> [51191ffb-0ae1-4414-8916-9e5012d86289] Query
> 'GetValidHostsForVmsQuery'
> failed: null
> 2019-04-17 08:58:41,816Z ERROR
> [org.ovirt.engine.core.bll.GetValidHostsForVmsQuery] (default task-7)
> [51191ffb-0ae1-4414-8916-9e5012d86289] Exception:
> java.lang.NullPointerException
>     at
> 
> org.ovirt.engine.core.bll.scheduling.SchedulingManager.subtractRunningVmResources(SchedulingManager.java:923)
> [bll.jar:]
>     at
> 
> org.ovirt.engine.core.bll.scheduling.SchedulingManager.canSchedule(SchedulingManager.java:616)
> [bll.jar:]
>     at
> 
> org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.lambda$getValidHosts$0(GetValidHostsForVmsQuery.java:56)
> [bll.jar:]
>     at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> [rt.jar:1.8.0_201]
>     at
> 
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> [rt.jar:1.8.0_201]
>     at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> [rt.jar:1.8.0_201]
>     at
> 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> [rt.jar:1.8.0_201]
>     at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> [rt.jar:1.8.0_201]
>     at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> [rt.jar:1.8.0_201]
>     at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> [rt.jar:1.8.0_201]
>     at
> 
> org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.getValidHosts(GetValidHostsForVmsQuery.java:59)
> [bll.jar:]
>     at
> 
> org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.executeQueryCommand(GetValidHostsForVmsQuery.java:36)
> [bll.jar:]
>     at
> 
> org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106)
> [bll.jar:]
>     at
> org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
> [dal.jar:]
>     at
> 
> org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14)
> [bll.jar:]
>     at
> org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:521)
> [bll.jar:]
>     at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:490)
> [bll.jar:]
>     at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
> [:1.8.0_201]
>     at
> 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_201]
>     at java.lang.reflect.Method.invoke(Method.java:498)
> [rt.jar:1.8.0_201]
>     at
> 
> org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
>     at
> 
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
>     at
> 
> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
>     at
> 
> org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92)
> [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final]
>     at
> 
> org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107)
> [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
>     at
> 
> org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126)
> [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
>     at
> 
> org.ovirt.engine.core.common.di.intercept

[ovirt-users] Re: VM migrations failing after upgrade 4.3.2 -> 4.3.3

2019-04-17 Thread Andrej Krejcir 
Hi,

this is a bug:  https://bugzilla.redhat.com/show_bug.cgi?id=1696621

It happens when a migrated VM has missing CPU load.
Do you have some special VMs without updated CPU load, or VMs that were
recently started?


Regards,
Andrej

On Wed, 17 Apr 2019 at 11:08, Eduardo Mayoral  wrote:

>
> Hi,
>
> After upgrade to 4.3.3 from 4.3.2, VM migrations are failing with
> "No available host to migrate VMs to"
>
> Interestingly, this happens in one of our clusters, not on the
> other. Both on the same datacenter, both in 4.3 compatibility version.
>
> This are the relevant lines from /var/log/ovirt-engine/engine.log
>
>
>
>
> 2019-04-17 08:58:41,816Z ERROR
> [org.ovirt.engine.core.bll.GetValidHostsForVmsQuery] (default task-7)
> [51191ffb-0ae1-4414-8916-9e5012d86289] Query 'GetValidHostsForVmsQuery'
> failed: null
> 2019-04-17 08:58:41,816Z ERROR
> [org.ovirt.engine.core.bll.GetValidHostsForVmsQuery] (default task-7)
> [51191ffb-0ae1-4414-8916-9e5012d86289] Exception:
> java.lang.NullPointerException
> at
>
> org.ovirt.engine.core.bll.scheduling.SchedulingManager.subtractRunningVmResources(SchedulingManager.java:923)
> [bll.jar:]
> at
>
> org.ovirt.engine.core.bll.scheduling.SchedulingManager.canSchedule(SchedulingManager.java:616)
> [bll.jar:]
> at
>
> org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.lambda$getValidHosts$0(GetValidHostsForVmsQuery.java:56)
> [bll.jar:]
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> [rt.jar:1.8.0_201]
> at
>
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> [rt.jar:1.8.0_201]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> [rt.jar:1.8.0_201]
> at
>
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> [rt.jar:1.8.0_201]
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> [rt.jar:1.8.0_201]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> [rt.jar:1.8.0_201]
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> [rt.jar:1.8.0_201]
> at
>
> org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.getValidHosts(GetValidHostsForVmsQuery.java:59)
> [bll.jar:]
> at
>
> org.ovirt.engine.core.bll.GetValidHostsForVmsQuery.executeQueryCommand(GetValidHostsForVmsQuery.java:36)
> [bll.jar:]
> at
>
> org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106)
> [bll.jar:]
> at
> org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
> [dal.jar:]
> at
>
> org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14)
> [bll.jar:]
> at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:521)
> [bll.jar:]
> at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:490)
> [bll.jar:]
> at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
> [:1.8.0_201]
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_201]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
> at
>
> org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at
>
> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at
>
> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
> at
>
> org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92)
> [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final]
> at
>
> org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107)
> [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
> at
>
> org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126)
> [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
> at
>
> org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
> [common.jar:]
> at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
> [:1.8.0_201]
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_201]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]
> at
>
> org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
> [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
> at
>
> org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92)
> [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
> at
>
> org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124)
> [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]
> at
> o