> On 22 Dec 2016, at 15:27, Fred Rolland <[email protected]> wrote: > > In any case the VM fails to run, can someone from virt team check the log ?
The answer is in previous emails;-) The re-run problem was triggered because of the failure and it is another interesting test to add, but currently when the original problem is addressed you won’t see it. the fix is merged on master, but not in 4.1 yet. Hmm..did we forgot to pick the stable branch maintainer?:-) It's https://gerrit.ovirt.org/#/c/68974/ Thanks, michal > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/37/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ovirt-engine/engine.log > > Here a snippet : > 2016-12-22 08:43:37,134-05 DEBUG > [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp > Reactor) [188ab77c] MESSAGE > content-length:743 > destination:jms.queue.events > content-type:application/json > subscription:db78fa50-cfb9-462e-860f-871ea935d645 > > {"params": {"notify_time": 4295938370, > "09f862a9-9531-4c4c-b846-0d95d374f29c": {"status": "Down", "timeOffset": "0", > "exitReason": 1, "exitMessage": "internal error: qemu unexpectedly closed the > monitor: 2016-12-22T13:43:35.791590Z qemu-kvm: warning: CPU(s) not present in > any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 > 15\n2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) up to maxcpus > should be described in NUMA config\n2016-12-22T13:43:35.840363Z qemu-kvm: > -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to > attach usb device \"QEMU USB CCID\" (full speed) to bus \"usb.0\", port \"1\" > (high speed)", "exitCode": 1}}, "jsonrpc": "2.0", "method": > "|virt|VM_status|09f862a9-9531-4c4c-b846-0d95d374f29c"}� > 2016-12-22 08:43:37,134-05 DEBUG > [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] > Message received: {"params": {"notify_time": 4295938370, > "09f862a9-9531-4c4c-b846-0d95d374f29c": {"status": "Down", "timeOffset": "0", > "exitReason": 1, "exitMessage": "internal error: qemu unexpectedly closed the > monitor: 2016-12-22T13:43:35.791590Z qemu-kvm: warning: CPU(s) not present in > any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 > 15\n2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) up to maxcpus > should be described in NUMA config\n2016-12-22T13:43:35.840363Z qemu-kvm: > -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to > attach usb device \"QEMU USB CCID\" (full speed) to bus \"usb.0\", port \"1\" > (high speed)", "exitCode": 1}}, "jsonrpc": "2.0", "method": > "|virt|VM_status|09f862a9-9531-4c4c-b846-0d95d374f29c"} > 2016-12-22 08:43:37,136-05 DEBUG > [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] > Event arrived from 192.168.201.4 containing > {"notify_time":4295938370,"09f862a9-9531-4c4c-b846-0d95d374f29c":{"status":"Down","timeOffset":"0","exitReason":1,"exitMessage":"internal > error: qemu unexpectedly closed the monitor: 2016-12-22T13:43:35.791590Z > qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 > 11 12 13 14 15\n2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) up > to maxcpus should be described in NUMA config\n2016-12-22T13:43:35.840363Z > qemu-kvm: -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch > trying to attach usb device \"QEMU USB CCID\" (full speed) to bus \"usb.0\", > port \"1\" (high speed)","exitCode":1}} > 2016-12-22 08:43:37,140-05 DEBUG > [org.ovirt.engine.core.vdsbroker.monitoring.EventVmStatsRefresher] > (ForkJoinPool-1-worker-4) [] processing event for host > lago-basic-suite-4-1-host0 data: > notify_time = 4295938370 > 09f862a9-9531-4c4c-b846-0d95d374f29c: > status = Down > timeOffset = 0 > exitReason = 1 > exitMessage = internal error: qemu unexpectedly closed the monitor: > 2016-12-22T13:43:35.791590Z qemu-kvm: warning: CPU(s) not present in any NUMA > nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 > 2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) up to maxcpus > should be described in NUMA config > 2016-12-22T13:43:35.840363Z qemu-kvm: -device > usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach > usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high speed) > exitCode = 1 > > > 2016-12-22 08:43:37,143-05 DEBUG > [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] > (ForkJoinPool-1-worker-4) [] graphics data missing in XML. > 2016-12-22 08:43:37,149-05 INFO > [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] > (ForkJoinPool-1-worker-4) [] VM '09f862a9-9531-4c4c-b846-0d95d374f29c' was > reported as Down on VDS > '6337f989-31cd-4462-b4f6-448237c05cab'(lago-basic-suite-4-1-host0) > 2016-12-22 0 > > > > > > On Thu, Dec 22, 2016 at 4:17 PM, Fred Rolland <[email protected]> wrote: > http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/37/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ovirt-engine/engine.log > > The "Duplicate Key" does not appear in the last log. > > On Thu, Dec 22, 2016 at 11:59 AM, Michal Skrivanek <[email protected]> > wrote: > >> On 22 Dec 2016, at 10:46, Michal Skrivanek <[email protected]> >> wrote: >> >>> >>> On 22 Dec 2016, at 08:57, Doron Fediuck <[email protected]> wrote: >>> >>> >>> >>> On Thu, Dec 22, 2016 at 9:48 AM, Michal Skrivanek >>> <[email protected]> wrote: >>> >>>> On 21 Dec 2016, at 20:52, Eyal Edri <[email protected]> wrote: >>>> >>>> Not as easy at it sounds, the current flow of ost is much more >>>> complicated than just build artifacts. I'm not saving we won't do it, but >>>> it won't be ready tomorrow and we designed lago exactly for being >>>> independent of any ci system so anyone can run it on their laptop. >>> >>> +1 >>> we just need to improve a little bit to make it possible for everyone. >>> Almost there. >>> >>>> So maybe using Jenkins might be simpler but people shouldn't skip >>>> verification just because such a job doesn't exist yet. >>>> >>>> On Dec 21, 2016 9:02 PM, "Oved Ourfali" <[email protected]> wrote: >>>> Why not run it via Jenkins for patches? >>>> Like, if you add a comment saying "run: ost" it will run it? >>>> >>> >>> please don’t add more. Even the Rerun-hooks thing is hard to remember. Even >>> after years I always keep asking is it "Re-run hooks” or “Rerun-Hooks” or >>> “rerun hooks”, does case matter…..bleh >>> Either add the button or maybe add it as a link in the automated comments >>> (that would actually work good enough) >>> >>> Thanks, >>> michal >>> >>>> It do it automatically based on another thing? >>>> >>>> On Dec 21, 2016 17:42, "Eyal Edri" <[email protected]> wrote: >>>> > >>>> > >>>> > >>>> > On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek >>>> > <[email protected]> wrote: >>>> >> >>>> >> >>>> >>> On 21 Dec 2016, at 16:25, Yaniv Kaul <[email protected]> wrote: >>>> >>> >>>> >>> >>>> >>> >>>> >>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek >>>> >>> <[email protected]> wrote: >>>> >>>> >>>> >>>> >>>> >>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek >>>> >>>>> <[email protected]> wrote: >>>> >>>>> >>>> >>>>> >>>> >>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <[email protected]> wrote: >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra >>>> >>>>>> <[email protected]> wrote: >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren <[email protected]> >>>> >>>>>>>> wrote: >>>> >>>>>>>> >>>> >>>>>>>> The test for running VMs had been failing since yesterday. >>>> >>>>>>>> >>>> >>>>>>>> The patch merged before the failures started was: >>>> >>>>>>>> https://gerrit.ovirt.org/#/c/68826/ >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> The error we`re seeing is a time-out (after two minutes) while >>>> >>>>>>>> running >>>> >>>>>>>> this API call: >>>> >>>>>>>> >>>> >>>>>>>> api.vms.get(VM0_NAME).status.state == ‘up' >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> This is a REST API call, the patch above is Frontend. So this is >>>> >>>>>>> unrelated. >>>> >>>>>>> >>>> >>>>>>> However on Host 0 I can see this: >>>> >>>>>>> >>>> >>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] >>>> >>>>>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') The vm start process >>>> >>>>>>> failed (vm:615) Traceback (most recent call last): File >>>> >>>>>>> "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm >>>> >>>>>>> self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run >>>> >>>>>>> self._connection.createXML(domxml, flags), File >>>> >>>>>>> "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line >>>> >>>>>>> 123, in wrapper ret = f(*args, **kwargs) File >>>> >>>>>>> "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in >>>> >>>>>>> wrapper return func(inst, *args, **kwargs) File >>>> >>>>>>> "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in >>>> >>>>>>> createXML if ret is None:raise libvirtError('virDomainCreateXML() >>>> >>>>>>> failed', conn=self) libvirtError: internal error: process exited >>>> >>>>>>> while connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: >>>> >>>>>>> warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 >>>> >>>>>>> 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: >>>> >>>>>>> All CPU(s) up to maxcpus should be described in NUMA config >>>> >>>>>>> 2016-12-20T21:54:43.101886Z qemu-kvm: -device >>>> >>>>>>> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying >>>> >>>>>>> to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", >>>> >>>>>>> port "1" (high speed) >>>> >>>>> >>>> >>>>> >>>> >>>>> it is likely related to the recent USB patches >>>> >>>>> investigating >>>> >>>> >>>> >>>> >>>> >>>> hm, there are multiple problems (features/bugs depending on >>>> >>>> prefferred point of view:) >>>> >>>> but there is an easy “fix” taking care of this particular problem, so >>>> >>>> we can start with that and figure out the proper approach later >>>> >>>> arik will push that and merge it soon, likely today >>>> >>> >>>> >>> >>>> >>> Thanks - if there is a quicker way to resolve this by reverting, I >>>> >>> think it's a better option. >>>> >> >>>> >> >>>> >> I really need to talk you out of this approach:-) >>>> >> It does sound tempting and logical, but with our development model of >>>> >> large patch series combined with late detection it really is quite >>>> >> risky. Here it wouldn’t help much…and figuring out the right revert >>>> >> patch is more complicated then fixing it. >>>> > >>>> > >>>> > Can we start asking developers run OST before they merge so it will be >>>> > early detection and not late detection? >>>> > We have video sessions on how to use OST, so it shouldn't be any issues >>>> > in running it on a patch. >>>> > >>>> >> >>>> >> I believe the best is to identify it early and notify the maintainer >>>> >> who merged that patch ASAP, as that person is in the best position to >>>> >> asses if revert is safe or if there is a simple follow up patch he can >>>> >> push right away >>>> >> >>>> >> We can surely improve on reporting, so Barak, how/why did you point to >>>> >> that particular patch in your email? It should start failing on >>>> >> 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 >>>> >> 01:56:07 2016 -0500) >>>> >> Even though it may be that it was hidden because of >>>> >> c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 >>>> >> 15:16:40 2016 -0500) >>>> >> >>>> >> (fix is ready, waiting on CI now) >>>> >> >>>> >> Thanks, >>>> >> michal >>>> >> >>>> >>> Y. >>>> >>> >>>> >>>> >>>> >>>> >>>> >>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] >>>> >>>>>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Changed state to >>>> >>>>>>> Down: internal error: process exited while connecting to monitor: >>>> >>>>>>> 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present >>>> >>>>>>> in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >>>> >>>>>>> 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to >>>> >>>>>>> maxcpus should be described in NUMA config >>>> >>>>>>> 2016-12-20T21:54:43.101886Z qemu-kvm: -device >>>> >>>>>>> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying >>>> >>>>>>> to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", >>>> >>>>>>> port "1" (high speed) (code=1) (vm:1197) 2016-12-20 16:54:43,550 >>>> >>>>>>> INFO (vm/d299ab29) [virt.vm] >>>> >>>>>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Stopping connection >>>> >>>>>>> (guestagent:430) >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> And on The engine loads of these: >>>> >>>>>>> >>>> >>>>>>> 2016-12-20 16:53:57,844-05 ERROR >>>> >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>> >>>>>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command >>>> >>>>>>> 'PollVDSCommand(HostName = lago-basic-suite-4-1-host0, >>>> >>>>>>> VdsIdVDSCommandParametersBase:{runAsync='true', >>>> >>>>>>> hostId='994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: >>>> >>>>>>> VDSGenericException: VDSNetworkException: Timeout during rpc call >>>> >>>>>>> 2016-12-20 16:53:57,849-05 DEBUG >>>> >>>>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL >>>> >>>>>>> Stomp Reactor) [7971dfb4] MESSAGE content-length:80 >>>> >>>>>>> destination:jms.topic.vdsm_responses content-type:application/json >>>> >>>>>>> subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": >>>> >>>>>>> "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": >>>> >>>>>>> true}� 2016-12-20 16:53:57,850-05 DEBUG >>>> >>>>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] >>>> >>>>>>> (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": >>>> >>>>>>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 >>>> >>>>>>> 16:53:57,850-05 ERROR >>>> >>>>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] >>>> >>>>>>> Not able to update response for >>>> >>>>>>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 >>>> >>>>>>> DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>> >>>>>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] >>>> >>>>>>> Exception: >>>> >>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>>> >>>>>>> VDSGenericException: VDSNetworkException: Timeout during rpc call >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >>>> >>>>>>> [vdsbroker.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:426) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:287) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1249) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1389) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2053) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) >>>> >>>>>>> [utils.jar:] at >>>> >>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) >>>> >>>>>>> [utils.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) >>>> >>>>>>> [:1.8.0_111] at >>>> >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> java.lang.reflect.Method.invoke(Method.java:498) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) >>>> >>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) >>>> >>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) >>>> >>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) >>>> >>>>>>> [bll.jar:] at >>>> >>>>>>> sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) >>>> >>>>>>> [:1.8.0_111] at >>>> >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> java.lang.reflect.Method.invoke(Method.java:498) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) >>>> >>>>>>> [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at >>>> >>>>>>> org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) >>>> >>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) >>>> >>>>>>> [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) >>>> >>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >>>> >>>>>>> at >>>> >>>>>>> org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>> >>>>>>> at >>>> >>>>>>> org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown >>>> >>>>>>> Source) [common.jar:] at >>>> >>>>>>> org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:170) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.resource.BackendHostResource.setupNetworks(BackendHostResource.java:212) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at >>>> >>>>>>> org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.java:182) >>>> >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> java.lang.reflect.Method.invoke(Method.java:498) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>>> >>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>> >>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>>> >>>>>>> [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at >>>> >>>>>>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:116) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:71) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) >>>> >>>>>>> [aaa.jar:] at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) >>>> >>>>>>> [aaa.jar:] at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) >>>> >>>>>>> [aaa.jar:] at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) >>>> >>>>>>> [aaa.jar:] at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) >>>> >>>>>>> [aaa.jar:] at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) >>>> >>>>>>> [aaa.jar:] at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>>>>> at >>>> >>>>>>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) >>>> >>>>>>> at >>>> >>>>>>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) >>>> >>>>>>> at >>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> >>>>>>> [rt.jar:1.8.0_111] at >>>> >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> >>>>>>> [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) >>>> >>>>>>> [rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR >>>> >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>> >>>>>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout >>>> >>>>>>> waiting for VDSM response: Internal timeout occured >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> This is definitely not related to the patch linked above. >>>> >>>>>>> >>>> >>>>>>> However I am also not quite sure what is the cause of this right >>>> >>>>>>> now, it might be a breaking change in VDSM. >>>> >>>>>>> >>>> >>>>>> >>>> >>>>>> The DC level is 4.0, is it possible that is the reason for the >>>> >>>>>> failure? >>>> >>>>>> >>>> >>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> Full test code can be seen here: >>>> >>>>>>>> https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=basic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=refs/heads/master#l291 >>>> >>>>>>>> >>>> >>>>>>>> Full test exception can be seen here: >>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testReport/junit/%28root%29/004_basic_sanity/vm_run/ >>>> >>>>>>>> >>>> >>>>>>>> Further logs can be seen in Jenkins: >>>> >>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/ >>>> >>>>>>>> >>>> >>>>>>>> -- >>>> >>>>>>>> Barak Korren >>>> >>>>>>>> [email protected] >>>> >>>>>>>> RHCE, RHCi, RHV-DevOps Team >>>> >>>>>>>> https://ifireball.wordpress.com/ >>>> >>>>>>>> _______________________________________________ >>>> >>>>>>>> Devel mailing list >>>> >>>>>>>> [email protected] >>>> >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> _______________________________________________ >>>> >>>>>>> Devel mailing list >>>> >>>>>>> [email protected] >>>> >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> -- >>>> >>>>>> Eyal Edri >>>> >>>>>> Associate Manager >>>> >>>>>> RHV DevOps >>>> >>>>>> EMEA ENG Virtualization R&D >>>> >>>>>> Red Hat Israel >>>> >>>>>> >>>> >>>>>> phone: +972-9-7692018 >>>> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>>>>> _______________________________________________ >>>> >>>>>> Devel mailing list >>>> >>>>>> [email protected] >>>> >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> >>>> Devel mailing list >>>> >>>> [email protected] >>>> >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>>> >>> >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Eyal Edri >>>> > Associate Manager >>>> > RHV DevOps >>>> > EMEA ENG Virtualization R&D >>>> > Red Hat Israel >>>> > >>>> > phone: +972-9-7692018 >>>> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> > >>>> > _______________________________________________ >>>> > Devel mailing list >>>> > [email protected] >>>> > http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> [email protected] >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >>> Guys, >>> 4.1 is broken for 11 hours now. >>> It seems that the issues are accumulating; looking into >>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ovirt-engine/engine.log >>> >>> shows a duplicate entry in disk mapping: >> >> interesting error, Arik. Reruns should be added to lago >> The root cause is few lines above and the same as was fixed yesterday on >> master. It just needs a backport to 4.1 >>> 2016-12-21 10:52:51,436-05 ERROR >>> [org.ovirt.engine.core.bll.RunVmOnceCommand] >>> (org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: >>> java.lang.IllegalStateException: Duplicate key >>> org.ovirt.engine.core.common.businessentities.storage.DiskVm >>> Element@8f24f60b >>> at >>> java.util.stream.Collectors.lambda$throwingMerger$0(Collecto >>> rs.java:133) [rt.jar:1.8.0_111] >>> at java.util.HashMap.merge(HashMa >>> p.java:1253) [rt.jar:1.8.0_111] >>> at >>> java.util.stream.Collectors.lambda$toMap$58(Collectors.java: >>> 1320) [rt.jar:1.8.0_111] >>> at java.util.stream.ReduceOps$3Re >>> ducingSink.accept(ReduceOps. >>> java:169) [rt.jar:1.8.0_111] >>> at java.util.ArrayList$ArrayListS >>> pliterator.forEachRemaining( >>> ArrayList.java:1374) [rt.jar:1.8.0_111] >>> at java.util.stream.AbstractPipel >>> ine.copyInto(AbstractPipeline. >>> java:481) [rt.jar:1.8.0_111] >>> at java.util.stream.AbstractPipel >>> ine.wrapAndCopyInto(AbstractPi >>> peline.java:471) [rt.jar:1.8.0_111] >>> at java.util.stream.ReduceOps$Red >>> uceOp.evaluateSequential(Reduc >>> eOps.java:708) [rt.jar:1.8.0_111] >>> at java.util.stream.AbstractPipel >>> ine.evaluate(AbstractPipeline. >>> java:234) [rt.jar:1.8.0_111] >>> at java.util.stream.ReferencePipe >>> line.collect(ReferencePipeline >>> .java:499) [rt.jar:1.8.0_111] >>> at org.ovirt.engine.core.bll.stor >>> age.disk.DiskHandler.getDiskToDiskVmElementMap(DiskHandler. >>> java:46) [bll.jar:] >>> at org.ovirt.engine.core.bll.vali >>> dator.RunVmValidator.validateDisksPassDiscard(RunVmValidator >>> .java:318) [bll.jar:] >>> at org.ovirt.engine.core.bll.vali >>> dator.RunVmValidator.canRunVm( >>> RunVmValidator.java:146) [bll.jar:] >>> at org.ovirt.engine.core.bll.RunV >>> mCommand.validate(RunVmCommand >>> .java:920) [bll.jar:] >>> at org.ovirt.engine.core.bll.RunV >>> mOnceCommand.validate(RunVmOnc >>> eCommand.java:58) [bll.jar:] >>> at org.ovirt.engine.core.bll.Comm >>> andBase.internalValidate(Comma >>> ndBase.java:834) [bll.jar:] >>> at org.ovirt.engine.core.bll.Comm >>> andBase.executeAction(CommandB >>> ase.java:388) [bll.jar:] >>> at org.ovirt.engine.core.bll.RunV >>> mCommandBase.reexecuteCommand( >>> RunVmCommandBase.java:163) [bll.jar:] >>> at org.ovirt.engine.core.bll.RunV >>> mCommandBase.rerun(RunVmComman >>> dBase.java:138) [bll.jar:] >>> at org.ovirt.engine.core.bll.RunV >>> mCommand.rerun(RunVmCommand.java:327) [bll.jar:] > > But as I understood from Arik this is indeed a separate problem in rerun, > likely caused by recent storage code. Adding Tal >>> >>> I suggest we consider reverting to the last good state, and work out the >>> issues from there. > > > _______________________________________________ > Devel mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/devel > > _______________________________________________ Devel mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/devel
