On Thu, Dec 22, 2016 at 4:17 PM, Fred Rolland <[email protected]> wrote:
> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ > 4.1/37/artifact/exported-artifacts/basic_suite_4.1.sh- > el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_ > basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ > ovirt-engine/engine.log > > The "Duplicate Key" does not appear in the last log. > > Once things go out of balance, they may take different directions. This run had no available hosts, so run vm failed earlier due to lack of hosts: 2016-12-22 08:43:38,463-05 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (org.ovirt.thread.pool-6-thread-37) [] Validation of action 'RunVmOnce' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_NO_HOSTS So the right thing should still be to get it back to stability and work our way from there. Git bisect may help. > On Thu, Dec 22, 2016 at 11:59 AM, Michal Skrivanek <[email protected]> > wrote: > >> >> On 22 Dec 2016, at 10:46, Michal Skrivanek <[email protected]> >> wrote: >> >> >> On 22 Dec 2016, at 08:57, Doron Fediuck <[email protected]> wrote: >> >> >> >> On Thu, Dec 22, 2016 at 9:48 AM, Michal Skrivanek <michal.skrivanek@re >> dhat.com> wrote: >> >>> >>> On 21 Dec 2016, at 20:52, Eyal Edri <[email protected]> wrote: >>> >>> Not as easy at it sounds, the current flow of ost is much more >>> complicated than just build artifacts. I'm not saving we won't do it, but >>> it won't be ready tomorrow and we designed lago exactly for being >>> independent of any ci system so anyone can run it on their laptop. >>> >>> >>> +1 >>> we just need to improve a little bit to make it possible for everyone. >>> Almost there. >>> >>> So maybe using Jenkins might be simpler but people shouldn't skip >>> verification just because such a job doesn't exist yet. >>> >>> On Dec 21, 2016 9:02 PM, "Oved Ourfali" <[email protected]> wrote: >>> >>>> Why not run it via Jenkins for patches? >>>> Like, if you add a comment saying "run: ost" it will run it? >>>> >>> >>> please don’t add more. Even the Rerun-hooks thing is hard to remember. >>> Even after years I always keep asking is it "Re-run hooks” or “Rerun-Hooks” >>> or “rerun hooks”, does case matter…..bleh >>> Either add the button or maybe add it as a link in the automated >>> comments (that would actually work good enough) >>> >>> Thanks, >>> michal >>> >>> It do it automatically based on another thing? >>>> >>>> On Dec 21, 2016 17:42, "Eyal Edri" <[email protected]> wrote: >>>> > >>>> > >>>> > >>>> > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek < >>>> [email protected]> wrote: >>>> >> >>>> >> >>>> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <[email protected]> wrote: >>>> >>> >>>> >>> >>>> >>> >>>> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek < >>>> [email protected]> wrote: >>>> >>>> >>>> >>>> >>>> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek < >>>> [email protected]> wrote: >>>> >>>>> >>>> >>>>> >>>> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <[email protected]> wrote: >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra < >>>> [email protected]> wrote: >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren <[email protected]> >>>> wrote: >>>> >>>>>>>> >>>> >>>>>>>> The test for running VMs had been failing since yesterday. >>>> >>>>>>>> >>>> >>>>>>>> The patch merged before the failures started was: >>>> >>>>>>>> https://gerrit.ovirt.org/#/c/68826/ >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> The error we`re seeing is a time-out (after two minutes) while >>>> running >>>> >>>>>>>> this API call: >>>> >>>>>>>> >>>> >>>>>>>> api.vms.get(VM0_NAME).status.state == ‘up' >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> This is a REST API call, the patch above is Frontend. So this >>>> is unrelated. >>>> >>>>>>> >>>> >>>>>>> However on Host 0 I can see this: >>>> >>>>>>> >>>> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] >>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') The vm start process >>>> failed (vm:615) Traceback (most recent call last): File >>>> "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm self._run() >>>> File "/usr/share/vdsm/virt/vm.py", line 1991, in _run >>>> self._connection.createXML(domxml, flags), File >>>> "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line >>>> 123, in wrapper ret = f(*args, **kwargs) File >>>> "/usr/lib/python2.7/site-packages/vdsm/utils.py", >>>> line 941, in wrapper return func(inst, *args, **kwargs) File >>>> "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in >>>> createXML if ret is None:raise libvirtError('virDomainCreateXML() >>>> failed', conn=self) libvirtError: internal error: process exited while >>>> connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: warning: >>>> CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >>>> 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to maxcpus >>>> should be described in NUMA config 2016-12-20T21:54:43.101886Z qemu-kvm: >>>> -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch >>>> trying to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", >>>> port "1" (high speed) >>>> >>>>> >>>> >>>>> >>>> >>>>> it is likely related to the recent USB patches >>>> >>>>> investigating >>>> >>>> >>>> >>>> >>>> >>>> hm, there are multiple problems (features/bugs depending on >>>> prefferred point of view:) >>>> >>>> but there is an easy “fix” taking care of this particular problem, >>>> so we can start with that and figure out the proper approach later >>>> >>>> arik will push that and merge it soon, likely today >>>> >>> >>>> >>> >>>> >>> Thanks - if there is a quicker way to resolve this by reverting, I >>>> think it's a better option. >>>> >> >>>> >> >>>> >> I really need to talk you out of this approach:-) >>>> >> It does sound tempting and logical, but with our development model >>>> of large patch series combined with late detection it really is quite >>>> risky. Here it wouldn’t help much…and figuring out the right revert patch >>>> is more complicated then fixing it. >>>> > >>>> > >>>> > Can we start asking developers run OST before they merge so it will >>>> be early detection and not late detection? >>>> > We have video sessions on how to use OST, so it shouldn't be any >>>> issues in running it on a patch. >>>> > >>>> >> >>>> >> I believe the best is to identify it early and notify the maintainer >>>> who merged that patch ASAP, as that person is in the best position to asses >>>> if revert is safe or if there is a simple follow up patch he can push right >>>> away >>>> >> >>>> >> We can surely improve on reporting, so Barak, how/why did you point >>>> to that particular patch in your email? It should start failing on >>>> 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 >>>> 01:56:07 2016 -0500) >>>> >> Even though it may be that it was hidden because >>>> of c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 >>>> 15:16:40 2016 -0500) >>>> >> >>>> >> (fix is ready, waiting on CI now) >>>> >> >>>> >> Thanks, >>>> >> michal >>>> >> >>>> >>> Y. >>>> >>> >>>> >>>> >>>> >>>> >>>> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] >>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Changed state to Down: >>>> internal error: process exited while connecting to monitor: >>>> 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present in any >>>> NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z >>>> qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA >>>> config 2016-12-20T21:54:43.101886Z qemu-kvm: -device >>>> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to >>>> attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" >>>> (high speed) (code=1) (vm:1197) 2016-12-20 16:54:43,550 INFO (vm/d299ab29) >>>> [virt.vm] (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Stopping >>>> connection (guestagent:430) >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> And on The engine loads of these: >>>> >>>>>>> >>>> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR >>>> >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command >>>> 'PollVDSCommand(HostName = lago-basic-suite-4-1-host0, >>>> VdsIdVDSCommandParametersBase:{runAsync='true', >>>> hostId='994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: >>>> VDSGenericException: VDSNetworkException: Timeout during rpc call >>>> 2016-12-20 16:53:57,849-05 DEBUG >>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] >>>> (SSL Stomp Reactor) [7971dfb4] MESSAGE content-length:80 >>>> destination:jms.topic.vdsm_responses content-type:application/json >>>> subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": "2.0", >>>> "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true}� >>>> 2016-12-20 16:53:57,850-05 DEBUG >>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] >>>> (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": >>>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 >>>> 16:53:57,850-05 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] >>>> (ResponseWorker) [] Not able to update response for >>>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 >>>> DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Exception: >>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>>> VDSGenericException: VDSNetworkException: Timeout during rpc call at >>>> org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >>>> [vdsbroker.jar:] at org.ovirt.engine.core.bll.netw >>>> ork.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) >>>> [bll.jar:] at org.ovirt.engine.core.bll.netw >>>> ork.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) >>>> [bll.jar:] at org.ovirt.engine.core.bll.netw >>>> ork.host.HostSetupNetworksCommand.invokeSetupNetworksCommand >>>> (HostSetupNetworksCommand.java:426) [bll.jar:] at >>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworksComm >>>> and.executeCommand(HostSetupNetworksCommand.java:287) [bll.jar:] at >>>> org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1249) >>>> [bll.jar:] at org.ovirt.engine.core.bll.Comm >>>> andBase.executeActionInTransactionScope(CommandBase.java:1389) >>>> [bll.jar:] at org.ovirt.engine.core.bll.Comm >>>> andBase.runInTransaction(CommandBase.java:2053) [bll.jar:] at >>>> org.ovirt.engine.core.utils.transaction.TransactionSuppor >>>> t.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at >>>> org.ovirt.engine.core.utils.transaction.TransactionSuppor >>>> t.executeInScope(TransactionSupport.java:103) [utils.jar:] at >>>> org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) >>>> [bll.jar:] at org.ovirt.engine.core.bll.Comm >>>> andBase.executeAction(CommandBase.java:395) [bll.jar:] at >>>> org.ovirt.engine.core.bll.executor.DefaultBackendActionExecu >>>> tor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at >>>> org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) >>>> [bll.jar:] at org.ovirt.engine.core.bll.Back >>>> end.runActionImpl(Backend.java:493) [bll.jar:] at >>>> org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) >>>> [bll.jar:] at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown >>>> Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAc >>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) >>>> [rt.jar:1.8.0_111] at org.jboss.as.ee.component.Mana >>>> gedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >>>> at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInte >>>> rception(Jsr299BindingsInterceptor.java:70) >>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInte >>>> rception(Jsr299BindingsInterceptor.java:80) >>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvoc >>>> ation(Jsr299BindingsInterceptor.java:93) >>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] >>>> at org.jboss.as.ee.component.interceptors.UserInterceptorFactor >>>> y$1.processInvocation(UserInterceptorFactory.java:63) at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>> at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerI >>>> nterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) >>>> [bll.jar:] at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown >>>> Source) [:1.8.0_111] at sun.reflect.DelegatingMethodAc >>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) >>>> [rt.jar:1.8.0_111] at org.jboss.as.ee.component.Mana >>>> gedReferenceLifecycleMethodInterceptor.processInvocation(Man >>>> agedReferenceLifecycleMethodInterceptor.java:89) at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTi >>>> meInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationIntercep >>>> tor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) >>>> [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at >>>> org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.p >>>> rocessInvocation(EjbRequestScopeActivationInterceptor.java:83) >>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.proc >>>> essInvocation(ConcurrentContextInterceptor.java:45) >>>> [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) >>>> at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>> at org.jboss.as.ee.component.interceptors.ComponentDispatcherIn >>>> terceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.component.singleton.SingletonComponentI >>>> nstanceAssociationInterceptor.processInvocation( >>>> SingletonComponentInstanceAssociationInterceptor.java:53) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.component.interceptors.CurrentInvocatio >>>> nContextInterceptor.processInvocation(CurrentInvoc >>>> ationContextInterceptor.java:41) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] >>>> at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInt >>>> erceptor.processInvocation(WaitTimeInterceptor.java:47) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.security.SecurityContextInterceptor.proces >>>> sInvocation(SecurityContextInterceptor.java:100) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.deployment.processors.StartupAwaitIntercep >>>> tor.processInvocation(StartupAwaitInterceptor.java:22) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.component.interceptors.ShutDownIntercep >>>> torFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ejb3.component.interceptors.LoggingIntercept >>>> or.processInvocation(LoggingInterceptor.java:67) >>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.as.ee.component.Name >>>> <http://org.jboss.as.ee.component.name/>spaceContextInterceptor.pro >>>> cessInvocation(NamespaceContextInterceptor.java:50) at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.invocation.ContextClassLoaderInterceptor.processIn >>>> vocation(ContextClassLoaderInterceptor.java:64) at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >>>> at org.wildfly.security.manager.WildFlySecurityManager.doChecke >>>> d(WildFlySecurityManager.java:636) at org.jboss.invocation.AccessChe >>>> ckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) >>>> at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >>>> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.proce >>>> ssInvocation(PrivilegedWithCombinerInterceptor.java:80) at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >>>> at >>>> org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >>>> at >>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> at >>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>> at >>>> org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) >>>> at >>>> org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown >>>> Source) [common.jar:] at org.ovirt.engine.api.restapi.r >>>> esource.BackendResource.doAction(BackendResource.java:250) at >>>> org.ovirt.engine.api.restapi.resource.BackendResource.perfor >>>> mAction(BackendResource.java:182) at org.ovirt.engine.api.restapi.r >>>> esource.BackendResource.performAction(BackendResource.java:170) at >>>> org.ovirt.engine.api.restapi.resource.BackendHostResource.se >>>> <http://esource.backendhostresource.se/>tupNetworks(BackendHostResource.java:212) >>>> at org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at >>>> org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.java:182) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> [rt.jar:1.8.0_111] at sun.reflect.NativeMethodAccess >>>> orImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_111] at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> [rt.jar:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) >>>> [rt.jar:1.8.0_111] at org.jboss.resteasy.core.Method >>>> InjectorImpl.invoke(MethodInjectorImpl.java:139) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget >>>> (ResourceMethodInvoker.java:295) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >>>> at >>>> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTarge >>>> tObject(ResourceLocatorInvoker.java:138) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >>>> at >>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTarge >>>> tObject(ResourceLocatorInvoker.java:133) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] >>>> at >>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDi >>>> spatcher.service(ServletContainerDispatcher.java:221) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatc >>>> her.service(HttpServletDispatcher.java:56) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatc >>>> her.service(HttpServletDispatcher.java:51) >>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>>> [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at >>>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) >>>> at >>>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >>>> at io.undertow.servlet.handlers.security.ServletSecurityRoleHan >>>> dler.handleRequest(ServletSecurityRoleHandler.java:62) at >>>> io.undertow.servlet.handlers.ServletDispatchingHandler.handl >>>> eRequest(ServletDispatchingHandler.java:36) at >>>> io.undertow.server.handlers.Pr <http://io.undertow.server.handlers.pr/> >>>> edicateHandler.handleRequest(PredicateHandler.java:43) at >>>> io.undertow.server.handlers.Pr <http://io.undertow.server.handlers.pr/> >>>> edicateHandler.handleRequest(PredicateHandler.java:43) at >>>> io.undertow.server.handlers.Pr <http://io.undertow.server.handlers.pr/> >>>> edicateHandler.handleRequest(PredicateHandler.java:43) at >>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchR >>>> equest(ServletInitialHandler.java:274) at >>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchT >>>> oPath(ServletInitialHandler.java:209) at io.undertow.servlet.spec.Reque >>>> stDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) at >>>> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSe >>>> tup(RequestDispatcherImpl.java:147) at io.undertow.servlet.spec.Reque >>>> stDispatcherImpl.forward(RequestDispatcherImpl.java:111) at >>>> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) >>>> at >>>> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) >>>> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.i >>>> nvocation.CurrentFilter.doFilter(CurrentFilter.java:116) at >>>> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:71) >>>> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt >>>> ers.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) >>>> [aaa.jar:] at io.undertow.servlet.core.Manag >>>> edFilter.doFilter(ManagedFilter.java:61) at >>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt >>>> ers.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] >>>> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt >>>> ers.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) >>>> [aaa.jar:] at io.undertow.servlet.core.Manag >>>> edFilter.doFilter(ManagedFilter.java:61) at >>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt >>>> ers.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) >>>> [aaa.jar:] at io.undertow.servlet.core.Manag >>>> edFilter.doFilter(ManagedFilter.java:61) at >>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt >>>> ers.SessionValidationFilter.do <http://ers.sessionvalidationfilter.do/> >>>> Filter(SessionValidationFilter.java:59) [aaa.jar:] at >>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.core.aaa.filt >>>> ers.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) >>>> [aaa.jar:] at io.undertow.servlet.core.Manag >>>> edFilter.doFilter(ManagedFilter.java:61) at >>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.s >>>> ecurity.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >>>> at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.d >>>> oFilter(CSRFProtectionFilter.java:102) at >>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at org.ovirt.engine.api.restapi.s >>>> ecurity.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) at >>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.d >>>> oFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.F >>>> ilterHandler.handleRequest(FilterHandler.java:84) at >>>> io.undertow.servlet.handlers.security.ServletSecurityRoleHan >>>> dler.handleRequest(ServletSecurityRoleHandler.java:62) at >>>> io.undertow.servlet.handlers.ServletDispatchingHandler.handl >>>> eRequest(ServletDispatchingHandler.java:36) at >>>> org.wildfly.extension.undertow.security.SecurityContextAssoc >>>> iationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >>>> at io.undertow.server.handlers.Pr >>>> <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(PredicateHandler.java:43) >>>> at io.undertow.servlet.handlers.security.SSLInformationAssociat >>>> ionHandler.handleRequest(SSLInformationAssociationHandler.java:131) at >>>> io.undertow.servlet.handlers.security.ServletAuthenticationC >>>> allHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at >>>> io.undertow.server.handlers.Pr <http://io.undertow.server.handlers.pr/> >>>> edicateHandler.handleRequest(PredicateHandler.java:43) at >>>> io.undertow.security.handlers.AuthenticationConstraintHandle >>>> r.handleRequest(AuthenticationConstraintHandler.java:53) at >>>> io.undertow.security.handlers.AbstractConfidentialityHandler >>>> .handleRequest(AbstractConfidentialityHandler.java:46) at >>>> io.undertow.servlet.handlers.security.ServletConfidentiality >>>> ConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >>>> at io.undertow.servlet.handlers.security.ServletSecurityConstra >>>> intHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at >>>> io.undertow.security.handlers.AuthenticationMechanismsHandle >>>> r.handleRequest(AuthenticationMechanismsHandler.java:60) at >>>> io.undertow.servlet.handlers.security.CachedAuthenticatedSes >>>> sionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >>>> at io.undertow.security.handlers.NotificationReceiverHandler.ha >>>> ndleRequest(NotificationReceiverHandler.java:50) at >>>> io.undertow.security.handlers.AbstractSecurityContextAssocia >>>> tionHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >>>> at io.undertow.server.handlers.Pr >>>> <http://io.undertow.server.handlers.pr/>edicateHandler.handleRequest(PredicateHandler.java:43) >>>> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHa >>>> ndler.handleRequest(JACCContextIdHandler.java:61) at >>>> io.undertow.server.handlers.Pr <http://io.undertow.server.handlers.pr/> >>>> edicateHandler.handleRequest(PredicateHandler.java:43) at >>>> io.undertow.server.handlers.Pr <http://io.undertow.server.handlers.pr/> >>>> edicateHandler.handleRequest(PredicateHandler.java:43) at >>>> io.undertow.servlet.handlers.ServletInitialHandler.handleFir >>>> stRequest(ServletInitialHandler.java:292) at >>>> io.undertow.servlet.handlers.ServletInitialHandler.access$10 >>>> 0(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.S >>>> ervletInitialHandler$2.call(ServletInitialHandler.java:138) at >>>> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >>>> at io.undertow.servlet.core.ServletRequestContextThreadSetupAct >>>> ion$1.call(ServletRequestContextThreadSetupAction.java:48) at >>>> io.undertow.servlet.core.ContextClassLoaderSetupAction$1.cal >>>> l(ContextClassLoaderSetupAction.java:43) at >>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.cal >>>> l(LegacyThreadSetupActionWrapper.java:44) at >>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.cal >>>> l(LegacyThreadSetupActionWrapper.java:44) at >>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.cal >>>> l(LegacyThreadSetupActionWrapper.java:44) at >>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.cal >>>> l(LegacyThreadSetupActionWrapper.java:44) at >>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchR >>>> equest(ServletInitialHandler.java:272) at >>>> io.undertow.servlet.handlers.ServletInitialHandler.access$00 >>>> 0(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.S >>>> ervletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >>>> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) >>>> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> [rt.jar:1.8.0_111] at java.util.concurrent.ThreadPoo >>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_111] >>>> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_111] 2016-12-20 >>>> 16:53:57,859-05 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout >>>> waiting for VDSM response: Internal timeout occured >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> This is definitely not related to the patch linked above. >>>> >>>>>>> >>>> >>>>>>> However I am also not quite sure what is the cause of this >>>> right now, it might be a breaking change in VDSM. >>>> >>>>>>> >>>> >>>>>> >>>> >>>>>> The DC level is 4.0, is it possible that is the reason for the >>>> failure? >>>> >>>>>> >>>> >>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> Full test code can be seen here: >>>> >>>>>>>> https://gerrit.ovirt.org/gitweb?p=ovirt-system-test >>>> s.git;a=blob;f=basic-suite-4.1/test-scenarios/004_basic_ >>>> sanity.py;hb=refs/heads/master#l291 >>>> >>>>>>>> >>>> >>>>>>>> Full test exception can be seen here: >>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experi >>>> mental_4.1/13/testReport/junit/%28root%29/004_basic_sanity/vm_run/ >>>> >>>>>>>> >>>> >>>>>>>> Further logs can be seen in Jenkins: >>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experi >>>> mental_4.1/13/artifact/exported-artifacts/basic_suite_4.1. >>>> sh-el7/exported-artifacts/test_logs/basic-suite-4.1/ >>>> post-004_basic_sanity.py/ >>>> >>>>>>>> >>>> >>>>>>>> -- >>>> >>>>>>>> Barak Korren >>>> >>>>>>>> [email protected] >>>> >>>>>>>> RHCE, RHCi, RHV-DevOps Team >>>> >>>>>>>> https://ifireball.wordpress.com/ >>>> >>>>>>>> _______________________________________________ >>>> >>>>>>>> Devel mailing list >>>> >>>>>>>> [email protected] >>>> >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> _______________________________________________ >>>> >>>>>>> Devel mailing list >>>> >>>>>>> [email protected] >>>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> -- >>>> >>>>>> Eyal Edri >>>> >>>>>> Associate Manager >>>> >>>>>> RHV DevOps >>>> >>>>>> EMEA ENG Virtualization R&D >>>> >>>>>> Red Hat Israel >>>> >>>>>> >>>> >>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>>>>> _______________________________________________ >>>> >>>>>> Devel mailing list >>>> >>>>>> [email protected] >>>> >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> >>>> Devel mailing list >>>> >>>> [email protected] >>>> >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>>> >>> >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Eyal Edri >>>> > Associate Manager >>>> > RHV DevOps >>>> > EMEA ENG Virtualization R&D >>>> > Red Hat Israel >>>> > >>>> > phone: +972-9-7692018 <+972%209-769-2018> >>>> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> > >>>> > _______________________________________________ >>>> > Devel mailing list >>>> > [email protected] >>>> > http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> [email protected] >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> Guys, >> 4.1 is broken for 11 hours now. >> It seems that the issues are accumulating; looking into >> http://jenkins.ovirt.org/job/test-repo_ovirt_experiment >> al_4.1/28/artifact/exported-artifacts/basic_suite_4.1.sh- >> el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_ >> basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ >> ovirt-engine/engine.log >> >> shows a duplicate entry in disk mapping: >> >> >> interesting error, Arik. Reruns should be added to lago >> The root cause is few lines above and the same as was fixed yesterday on >> master. It just needs a backport to 4.1 >> >> 2016-12-21 10:52:51,436-05 ERROR >> [org.ovirt.engine.core.bll.RunVmOnceCommand] >> (org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: >> java.lang.IllegalStateException: Duplicate key >> org.ovirt.engine.core.common.businessentities.storage.DiskVmElement@8f24f60b >> at >> java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) >> [rt.jar:1.8.0_111] >> at java.util.HashMap.merge(HashMap.java:1253) [rt.jar:1.8.0_111] >> at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) >> [rt.jar:1.8.0_111] >> at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) >> [rt.jar:1.8.0_111] >> at >> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) >> [rt.jar:1.8.0_111] >> at >> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) >> [rt.jar:1.8.0_111] >> at >> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) >> [rt.jar:1.8.0_111] >> at >> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) >> [rt.jar:1.8.0_111] >> at >> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) >> [rt.jar:1.8.0_111] >> at >> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) >> [rt.jar:1.8.0_111] >> at >> org.ovirt.engine.core.bll.storage.disk.DiskHandler.getDiskToDiskVmElementMap(DiskHandler.java:46) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.validator.RunVmValidator.validateDisksPassDiscard(RunVmValidator.java:318) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.validator.RunVmValidator.canRunVm(RunVmValidator.java:146) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:920) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:58) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:834) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:388) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.RunVmCommandBase.reexecuteCommand(RunVmCommandBase.java:163) >> [bll.jar:] >> at >> org.ovirt.engine.core.bll.RunVmCommandBase.rerun(RunVmCommandBase.java:138) >> [bll.jar:] >> at org.ovirt.engine.core.bll.RunVmCommand.rerun(RunVmCommand.java:327) >> [bll.jar:] >> >> >> But as I understood from Arik this is indeed a separate problem in rerun, >> likely caused by recent storage code. Adding Tal >> >> >> I suggest we consider reverting to the last good state, and work out the >> issues from there. >> >> >> >> _______________________________________________ >> Devel mailing list >> [email protected] >> http://lists.ovirt.org/mailman/listinfo/devel >> > > > _______________________________________________ > Devel mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/devel >
_______________________________________________ Devel mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/devel
