> On 21 Dec 2016, at 14:56, Michal Skrivanek <michal.skriva...@redhat.com> > wrote: > > >> On 21 Dec 2016, at 12:19, Eyal Edri <ee...@redhat.com >> <mailto:ee...@redhat.com>> wrote: >> >> >> >> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra <vfeen...@redhat.com >> <mailto:vfeen...@redhat.com>> wrote: >> >>> On Dec 21, 2016, at 11:17 AM, Barak Korren <bkor...@redhat.com >>> <mailto:bkor...@redhat.com>> wrote: >>> >>> The test for running VMs had been failing since yesterday. >>> >>> The patch merged before the failures started was: >>> https://gerrit.ovirt.org/#/c/68826/ <https://gerrit.ovirt.org/#/c/68826/> >> >> >> >>> >>> The error we`re seeing is a time-out (after two minutes) while running >>> this API call: >>> >>> api.vms.get(VM0_NAME).status.state == ‘up' >> >> This is a REST API call, the patch above is Frontend. So this is unrelated. >> >> However on Host 0 I can see this: >> >> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] >> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') The vm start process failed >> (vm:615) >> Traceback (most recent call last): >> File "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm >> self._run() >> File "/usr/share/vdsm/virt/vm.py", line 1991, in _run >> self._connection.createXML(domxml, flags), >> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line >> 123, in wrapper >> ret = f(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper >> return func(inst, *args, **kwargs) >> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in >> createXML >> if ret is None:raise libvirtError('virDomainCreateXML() failed', >> conn=self) >> libvirtError: internal error: process exited while connecting to monitor: >> 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any >> NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >> 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus >> should be described in NUMA config >> 2016-12-20T21:54:43.101886Z qemu-kvm: -device >> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach >> usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high speed) > > it is likely related to the recent USB patches > investigating
hm, there are multiple problems (features/bugs depending on prefferred point of view:) but there is an easy “fix” taking care of this particular problem, so we can start with that and figure out the proper approach later arik will push that and merge it soon, likely today >> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] >> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: >> internal error: process exited while connecting to monitor: >> 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any >> NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >> 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus >> should be described in NUMA config >> 2016-12-20T21:54:43.101886Z qemu-kvm: -device >> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach >> usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high >> speed) (code=1) (vm:1197) >> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] >> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Stopping connection >> (guestagent:430) >> >> >> And on The engine loads of these: >> >> 2016-12-20 16:53:57,844-05 ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) >> [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command 'PollVDSCommand(HostName = >> lago-basic-suite-4-1-host0, VdsIdVDSCommandParametersBase:{runAsync='true', >> hostId='994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: >> VDSGenericException: VDSNetworkException: Timeout during rpc call >> 2016-12-20 16:53:57,849-05 DEBUG >> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp >> Reactor) [7971dfb4] MESSAGE >> content-length:80 >> destination:jms.topic.vdsm_responses >> content-type:application/json >> subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 >> >> {"jsonrpc": "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": >> true}� >> 2016-12-20 16:53:57,850-05 DEBUG >> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] >> Message received: {"jsonrpc": "2.0", "id": >> "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} >> 2016-12-20 16:53:57,850-05 ERROR >> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able >> to update response for "3c95fdb0-5b77-4927-9f6e-adc7395c122d" >> 2016-12-20 16:53:57,844-05 DEBUG >> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) >> [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: >> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >> VDSGenericException: VDSNetworkException: Timeout during rpc call >> at >> org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >> [vdsbroker.jar:] >> at >> org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:426) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:287) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1249) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1389) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2053) >> [bll.jar:] >> at >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) >> [utils.jar:] >> at >> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) >> [utils.jar:] >> at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) >> [bll.jar:] >> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) >> [bll.jar:] >> at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) >> [bll.jar:] >> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) >> [bll.jar:] >> at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) >> [:1.8.0_111] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> [rt.jar:1.8.0_111] >> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] >> at >> org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >> at >> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) >> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) >> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) >> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >> at >> org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) >> [bll.jar:] >> at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) >> [:1.8.0_111] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> [rt.jar:1.8.0_111] >> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] >> at >> org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >> at >> org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) >> [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] >> at >> org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) >> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) >> [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >> at >> org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) >> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >> at >> org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) >> at >> org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >> at >> org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >> at >> org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >> at >> org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >> at >> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >> at >> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >> at >> org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) >> at >> org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown >> Source) [common.jar:] >> at >> org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) >> at >> org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) >> at >> org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:170) >> at >> org.ovirt.engine.api.restapi.resource.BackendHostResource.setupNetworks(BackendHostResource.java:212) >> at org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) >> at >> org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.java:182) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> [rt.jar:1.8.0_111] >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> [rt.jar:1.8.0_111] >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> [rt.jar:1.8.0_111] >> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_111] >> at >> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) >> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >> [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] >> at >> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) >> at >> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >> at >> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >> at >> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) >> at >> io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) >> at >> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) >> at >> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) >> at >> io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) >> at >> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) >> at >> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:116) >> at >> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:71) >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) >> [aaa.jar:] >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) >> [aaa.jar:] >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) >> [aaa.jar:] >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) >> [aaa.jar:] >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) >> [aaa.jar:] >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) >> [aaa.jar:] >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >> at >> org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) >> at >> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >> at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >> at >> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) >> at >> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >> at >> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >> at >> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) >> at >> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) >> at >> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) >> at >> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >> at >> io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) >> at >> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) >> at >> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >> at >> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) >> at >> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >> at >> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) >> at >> io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) >> at >> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) >> at >> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >> at >> io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) >> at >> io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) >> at >> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >> at >> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >> at >> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >> at >> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >> at >> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) >> at >> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) >> at >> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) >> at >> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [rt.jar:1.8.0_111] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [rt.jar:1.8.0_111] >> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] >> >> 2016-12-20 16:53:57,859-05 ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-17) >> [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout waiting for VDSM response: >> Internal timeout occured >> >> This is definitely not related to the patch linked above. >> >> However I am also not quite sure what is the cause of this right now, it >> might be a breaking change in VDSM. >> >> >> The DC level is 4.0, is it possible that is the reason for the failure? >> >> >> >>> >>> Full test code can be seen here: >>> https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=basic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=refs/heads/master#l291 >>> >>> <https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=basic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=refs/heads/master#l291> >>> >>> Full test exception can be seen here: >>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testReport/junit/%28root%29/004_basic_sanity/vm_run/ >>> >>> <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testReport/junit/%28root%29/004_basic_sanity/vm_run/> >>> >>> Further logs can be seen in Jenkins: >>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/ >>> >>> <http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/> >>> >>> -- >>> Barak Korren >>> bkor...@redhat.com <mailto:bkor...@redhat.com> >>> RHCE, RHCi, RHV-DevOps Team >>> https://ifireball.wordpress.com/ <https://ifireball.wordpress.com/> >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org <mailto:Devel@ovirt.org> >>> http://lists.ovirt.org/mailman/listinfo/devel >>> <http://lists.ovirt.org/mailman/listinfo/devel> >> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org <mailto:Devel@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/devel >> <http://lists.ovirt.org/mailman/listinfo/devel> >> >> >> >> -- >> Eyal Edri >> Associate Manager >> RHV DevOps >> EMEA ENG Virtualization R&D >> Red Hat Israel >> >> phone: +972-9-7692018 >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org <mailto:Devel@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/devel >
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel