> On 22 Dec 2016, at 17:11, Michal Skrivanek <[email protected]> wrote: > > >> On 22 Dec 2016, at 15:27, Fred Rolland <[email protected]> wrote: >> >> In any case the VM fails to run, can someone from virt team check the log ? > > The answer is in previous emails;-) > The re-run problem was triggered because of the failure and it is another > interesting test to add, but currently when the original problem is addressed > you won’t see it. > > the fix is merged on master, but not in 4.1 yet. Hmm..did we forgot to pick > the stable branch maintainer?:-) > It's https://gerrit.ovirt.org/#/c/68974/
if it was just copied from 4.0 then 4.0 stable branch maintainer can take it in > > Thanks, > michal > >> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/37/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ovirt-engine/engine.log >> >> Here a snippet : >> 2016-12-22 08:43:37,134-05 DEBUG >> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp >> Reactor) [188ab77c] MESSAGE >> content-length:743 >> destination:jms.queue.events >> content-type:application/json >> subscription:db78fa50-cfb9-462e-860f-871ea935d645 >> >> {"params": {"notify_time": 4295938370, >> "09f862a9-9531-4c4c-b846-0d95d374f29c": {"status": "Down", "timeOffset": >> "0", "exitReason": 1, "exitMessage": "internal error: qemu unexpectedly >> closed the monitor: 2016-12-22T13:43:35.791590Z qemu-kvm: warning: CPU(s) >> not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 >> 15\n2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) up to maxcpus >> should be described in NUMA config\n2016-12-22T13:43:35.840363Z qemu-kvm: >> -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying >> to attach usb device \"QEMU USB CCID\" (full speed) to bus \"usb.0\", port >> \"1\" (high speed)", "exitCode": 1}}, "jsonrpc": "2.0", "method": >> "|virt|VM_status|09f862a9-9531-4c4c-b846-0d95d374f29c"}� >> 2016-12-22 08:43:37,134-05 DEBUG >> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] >> Message received: {"params": {"notify_time": 4295938370, >> "09f862a9-9531-4c4c-b846-0d95d374f29c": {"status": "Down", "timeOffset": >> "0", "exitReason": 1, "exitMessage": "internal error: qemu unexpectedly >> closed the monitor: 2016-12-22T13:43:35.791590Z qemu-kvm: warning: CPU(s) >> not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 >> 15\n2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) up to maxcpus >> should be described in NUMA config\n2016-12-22T13:43:35.840363Z qemu-kvm: >> -device usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying >> to attach usb device \"QEMU USB CCID\" (full speed) to bus \"usb.0\", port >> \"1\" (high speed)", "exitCode": 1}}, "jsonrpc": "2.0", "method": >> "|virt|VM_status|09f862a9-9531-4c4c-b846-0d95d374f29c"} >> 2016-12-22 08:43:37,136-05 DEBUG >> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] >> Event arrived from 192.168.201.4 containing >> {"notify_time":4295938370,"09f862a9-9531-4c4c-b846-0d95d374f29c":{"status":"Down","timeOffset":"0","exitReason":1,"exitMessage":"internal >> error: qemu unexpectedly closed the monitor: 2016-12-22T13:43:35.791590Z >> qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 >> 10 11 12 13 14 15\n2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) >> up to maxcpus should be described in NUMA >> config\n2016-12-22T13:43:35.840363Z qemu-kvm: -device >> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach >> usb device \"QEMU USB CCID\" (full speed) to bus \"usb.0\", port \"1\" (high >> speed)","exitCode":1}} >> 2016-12-22 08:43:37,140-05 DEBUG >> [org.ovirt.engine.core.vdsbroker.monitoring.EventVmStatsRefresher] >> (ForkJoinPool-1-worker-4) [] processing event for host >> lago-basic-suite-4-1-host0 data: >> notify_time = 4295938370 >> 09f862a9-9531-4c4c-b846-0d95d374f29c: >> status = Down >> timeOffset = 0 >> exitReason = 1 >> exitMessage = internal error: qemu unexpectedly closed the monitor: >> 2016-12-22T13:43:35.791590Z qemu-kvm: warning: CPU(s) not present in any >> NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >> 2016-12-22T13:43:35.791854Z qemu-kvm: warning: All CPU(s) up to maxcpus >> should be described in NUMA config >> 2016-12-22T13:43:35.840363Z qemu-kvm: -device >> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying to attach >> usb device "QEMU USB CCID" (full speed) to bus "usb.0", port "1" (high speed) >> exitCode = 1 >> >> >> 2016-12-22 08:43:37,143-05 DEBUG >> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] >> (ForkJoinPool-1-worker-4) [] graphics data missing in XML. >> 2016-12-22 08:43:37,149-05 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >> (ForkJoinPool-1-worker-4) [] VM '09f862a9-9531-4c4c-b846-0d95d374f29c' was >> reported as Down on VDS >> '6337f989-31cd-4462-b4f6-448237c05cab'(lago-basic-suite-4-1-host0) >> 2016-12-22 0 >> >> >> >> >> >> On Thu, Dec 22, 2016 at 4:17 PM, Fred Rolland <[email protected]> wrote: >> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/37/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ovirt-engine/engine.log >> >> The "Duplicate Key" does not appear in the last log. >> >> On Thu, Dec 22, 2016 at 11:59 AM, Michal Skrivanek <[email protected]> >> wrote: >> >>> On 22 Dec 2016, at 10:46, Michal Skrivanek <[email protected]> >>> wrote: >>> >>>> >>>> On 22 Dec 2016, at 08:57, Doron Fediuck <[email protected]> wrote: >>>> >>>> >>>> >>>> On Thu, Dec 22, 2016 at 9:48 AM, Michal Skrivanek >>>> <[email protected]> wrote: >>>> >>>>> On 21 Dec 2016, at 20:52, Eyal Edri <[email protected]> wrote: >>>>> >>>>> Not as easy at it sounds, the current flow of ost is much more >>>>> complicated than just build artifacts. I'm not saving we won't do it, >>>>> but it won't be ready tomorrow and we designed lago exactly for being >>>>> independent of any ci system so anyone can run it on their laptop. >>>> >>>> +1 >>>> we just need to improve a little bit to make it possible for everyone. >>>> Almost there. >>>> >>>>> So maybe using Jenkins might be simpler but people shouldn't skip >>>>> verification just because such a job doesn't exist yet. >>>>> >>>>> On Dec 21, 2016 9:02 PM, "Oved Ourfali" <[email protected]> wrote: >>>>> Why not run it via Jenkins for patches? >>>>> Like, if you add a comment saying "run: ost" it will run it? >>>>> >>>> >>>> please don’t add more. Even the Rerun-hooks thing is hard to remember. >>>> Even after years I always keep asking is it "Re-run hooks” or >>>> “Rerun-Hooks” or “rerun hooks”, does case matter…..bleh >>>> Either add the button or maybe add it as a link in the automated comments >>>> (that would actually work good enough) >>>> >>>> Thanks, >>>> michal >>>> >>>>> It do it automatically based on another thing? >>>>> >>>>> On Dec 21, 2016 17:42, "Eyal Edri" <[email protected]> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Dec 21, 2016 at 5:36 PM, Michal Skrivanek >>>>>> <[email protected]> wrote: >>>>>>> >>>>>>> >>>>>>>> On 21 Dec 2016, at 16:25, Yaniv Kaul <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Dec 21, 2016 at 5:19 PM, Michal Skrivanek >>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 21 Dec 2016, at 14:56, Michal Skrivanek >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 21 Dec 2016, at 12:19, Eyal Edri <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Dec 21, 2016 at 12:56 PM, Vinzenz Feenstra >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Dec 21, 2016, at 11:17 AM, Barak Korren <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> The test for running VMs had been failing since yesterday. >>>>>>>>>>>>> >>>>>>>>>>>>> The patch merged before the failures started was: >>>>>>>>>>>>> https://gerrit.ovirt.org/#/c/68826/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The error we`re seeing is a time-out (after two minutes) while >>>>>>>>>>>>> running >>>>>>>>>>>>> this API call: >>>>>>>>>>>>> >>>>>>>>>>>>> api.vms.get(VM0_NAME).status.state == ‘up' >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This is a REST API call, the patch above is Frontend. So this is >>>>>>>>>>>> unrelated. >>>>>>>>>>>> >>>>>>>>>>>> However on Host 0 I can see this: >>>>>>>>>>>> >>>>>>>>>>>> 2016-12-20 16:54:43,544 ERROR (vm/d299ab29) [virt.vm] >>>>>>>>>>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') The vm start process >>>>>>>>>>>> failed (vm:615) Traceback (most recent call last): File >>>>>>>>>>>> "/usr/share/vdsm/virt/vm.py", line 551, in _startUnderlyingVm >>>>>>>>>>>> self._run() File "/usr/share/vdsm/virt/vm.py", line 1991, in _run >>>>>>>>>>>> self._connection.createXML(domxml, flags), File >>>>>>>>>>>> "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line >>>>>>>>>>>> 123, in wrapper ret = f(*args, **kwargs) File >>>>>>>>>>>> "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in >>>>>>>>>>>> wrapper return func(inst, *args, **kwargs) File >>>>>>>>>>>> "/usr/lib64/python2.7/site-packages/libvirt.py", line 3782, in >>>>>>>>>>>> createXML if ret is None:raise libvirtError('virDomainCreateXML() >>>>>>>>>>>> failed', conn=self) libvirtError: internal error: process exited >>>>>>>>>>>> while connecting to monitor: 2016-12-20T21:54:43.044971Z qemu-kvm: >>>>>>>>>>>> warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 >>>>>>>>>>>> 10 11 12 13 14 15 2016-12-20T21:54:43.045164Z qemu-kvm: warning: >>>>>>>>>>>> All CPU(s) up to maxcpus should be described in NUMA config >>>>>>>>>>>> 2016-12-20T21:54:43.101886Z qemu-kvm: -device >>>>>>>>>>>> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying >>>>>>>>>>>> to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", >>>>>>>>>>>> port "1" (high speed) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> it is likely related to the recent USB patches >>>>>>>>>> investigating >>>>>>>>> >>>>>>>>> >>>>>>>>> hm, there are multiple problems (features/bugs depending on >>>>>>>>> prefferred point of view:) >>>>>>>>> but there is an easy “fix” taking care of this particular problem, so >>>>>>>>> we can start with that and figure out the proper approach later >>>>>>>>> arik will push that and merge it soon, likely today >>>>>>>> >>>>>>>> >>>>>>>> Thanks - if there is a quicker way to resolve this by reverting, I >>>>>>>> think it's a better option. >>>>>>> >>>>>>> >>>>>>> I really need to talk you out of this approach:-) >>>>>>> It does sound tempting and logical, but with our development model of >>>>>>> large patch series combined with late detection it really is quite >>>>>>> risky. Here it wouldn’t help much…and figuring out the right revert >>>>>>> patch is more complicated then fixing it. >>>>>> >>>>>> >>>>>> Can we start asking developers run OST before they merge so it will be >>>>>> early detection and not late detection? >>>>>> We have video sessions on how to use OST, so it shouldn't be any issues >>>>>> in running it on a patch. >>>>>> >>>>>>> >>>>>>> I believe the best is to identify it early and notify the maintainer >>>>>>> who merged that patch ASAP, as that person is in the best position to >>>>>>> asses if revert is safe or if there is a simple follow up patch he can >>>>>>> push right away >>>>>>> >>>>>>> We can surely improve on reporting, so Barak, how/why did you point to >>>>>>> that particular patch in your email? It should start failing on >>>>>>> 16c2ec236184b3152f1df8e874b43115f78d0989 (CommitDate: Fri Dec 16 >>>>>>> 01:56:07 2016 -0500) >>>>>>> Even though it may be that it was hidden because of >>>>>>> c46f653a7846c3c2a76507b8dcf5bc0391ec5709 (CommitDate: Mon Dec 19 >>>>>>> 15:16:40 2016 -0500) >>>>>>> >>>>>>> (fix is ready, waiting on CI now) >>>>>>> >>>>>>> Thanks, >>>>>>> michal >>>>>>> >>>>>>>> Y. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> 2016-12-20 16:54:43,550 INFO (vm/d299ab29) [virt.vm] >>>>>>>>>>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Changed state to >>>>>>>>>>>> Down: internal error: process exited while connecting to monitor: >>>>>>>>>>>> 2016-12-20T21:54:43.044971Z qemu-kvm: warning: CPU(s) not present >>>>>>>>>>>> in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 >>>>>>>>>>>> 2016-12-20T21:54:43.045164Z qemu-kvm: warning: All CPU(s) up to >>>>>>>>>>>> maxcpus should be described in NUMA config >>>>>>>>>>>> 2016-12-20T21:54:43.101886Z qemu-kvm: -device >>>>>>>>>>>> usb-ccid,id=ccid0,bus=usb.0,port=1: Warning: speed mismatch trying >>>>>>>>>>>> to attach usb device "QEMU USB CCID" (full speed) to bus "usb.0", >>>>>>>>>>>> port "1" (high speed) (code=1) (vm:1197) 2016-12-20 16:54:43,550 >>>>>>>>>>>> INFO (vm/d299ab29) [virt.vm] >>>>>>>>>>>> (vmId='d299ab29-284a-435c-a50f-183a6e54def2') Stopping connection >>>>>>>>>>>> (guestagent:430) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> And on The engine loads of these: >>>>>>>>>>>> >>>>>>>>>>>> 2016-12-20 16:53:57,844-05 ERROR >>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>>>>>>>>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Command >>>>>>>>>>>> 'PollVDSCommand(HostName = lago-basic-suite-4-1-host0, >>>>>>>>>>>> VdsIdVDSCommandParametersBase:{runAsync='true', >>>>>>>>>>>> hostId='994b5d79-605f-4415-94f2-02c79cfa246e'})' execution failed: >>>>>>>>>>>> VDSGenericException: VDSNetworkException: Timeout during rpc call >>>>>>>>>>>> 2016-12-20 16:53:57,849-05 DEBUG >>>>>>>>>>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL >>>>>>>>>>>> Stomp Reactor) [7971dfb4] MESSAGE content-length:80 >>>>>>>>>>>> destination:jms.topic.vdsm_responses content-type:application/json >>>>>>>>>>>> subscription:5b6494d5-d5a0-4771-941c-a8be70f72450 {"jsonrpc": >>>>>>>>>>>> "2.0", "id": "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": >>>>>>>>>>>> true}� 2016-12-20 16:53:57,850-05 DEBUG >>>>>>>>>>>> [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] >>>>>>>>>>>> (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": >>>>>>>>>>>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d", "result": true} 2016-12-20 >>>>>>>>>>>> 16:53:57,850-05 ERROR >>>>>>>>>>>> [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] >>>>>>>>>>>> Not able to update response for >>>>>>>>>>>> "3c95fdb0-5b77-4927-9f6e-adc7395c122d" 2016-12-20 16:53:57,844-05 >>>>>>>>>>>> DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>>>>>>>>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] >>>>>>>>>>>> Exception: >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: >>>>>>>>>>>> VDSGenericException: VDSNetworkException: Timeout during rpc call >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:73) >>>>>>>>>>>> [vdsbroker.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue(HostSetupNetworkPoller.java:56) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll(HostSetupNetworkPoller.java:41) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.invokeSetupNetworksCommand(HostSetupNetworksCommand.java:426) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:287) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1249) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1389) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2053) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) >>>>>>>>>>>> [utils.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) >>>>>>>>>>>> [utils.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1449) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:395) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) >>>>>>>>>>>> [:1.8.0_111] at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) >>>>>>>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) >>>>>>>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) >>>>>>>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) >>>>>>>>>>>> [bll.jar:] at >>>>>>>>>>>> sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) >>>>>>>>>>>> [:1.8.0_111] at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) >>>>>>>>>>>> [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at >>>>>>>>>>>> org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) >>>>>>>>>>>> [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) >>>>>>>>>>>> [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) >>>>>>>>>>>> [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >>>>>>>>>>>> at >>>>>>>>>>>> org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.common.interfaces.BackendLocal$$$view4.runAction(Unknown >>>>>>>>>>>> Source) [common.jar:] at >>>>>>>>>>>> org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:182) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.resource.BackendResource.performAction(BackendResource.java:170) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.resource.BackendHostResource.setupNetworks(BackendHostResource.java:212) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.v3.V3Server.adaptAction(V3Server.java:216) at >>>>>>>>>>>> org.ovirt.engine.api.v3.servers.V3HostServer.setupNetworks(V3HostServer.java:182) >>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>>>>>>>>>>> [resteasy-jaxrs-3.0.19.Final.jar:3.0.19.Final] at >>>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) >>>>>>>>>>>> [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:221) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:147) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:139) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:68) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:116) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:71) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) >>>>>>>>>>>> [aaa.jar:] at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) >>>>>>>>>>>> [aaa.jar:] at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:91) >>>>>>>>>>>> [aaa.jar:] at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) >>>>>>>>>>>> [aaa.jar:] at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) >>>>>>>>>>>> [aaa.jar:] at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) >>>>>>>>>>>> [aaa.jar:] at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> org.ovirt.engine.api.restapi.security.CORSSupportFilter.doFilter(CORSSupportFilter.java:183) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>>>>>>>>>> at >>>>>>>>>>>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) >>>>>>>>>>>> at >>>>>>>>>>>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) >>>>>>>>>>>> at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>>>>>>>>> [rt.jar:1.8.0_111] at >>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>>>>>>>> [rt.jar:1.8.0_111] at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>> [rt.jar:1.8.0_111] 2016-12-20 16:53:57,859-05 ERROR >>>>>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] >>>>>>>>>>>> (default task-17) [5ecd5a55-2b7a-4dd6-b42b-cc49bbfb3962] Timeout >>>>>>>>>>>> waiting for VDSM response: Internal timeout occured >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This is definitely not related to the patch linked above. >>>>>>>>>>>> >>>>>>>>>>>> However I am also not quite sure what is the cause of this right >>>>>>>>>>>> now, it might be a breaking change in VDSM. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The DC level is 4.0, is it possible that is the reason for the >>>>>>>>>>> failure? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Full test code can be seen here: >>>>>>>>>>>>> https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=basic-suite-4.1/test-scenarios/004_basic_sanity.py;hb=refs/heads/master#l291 >>>>>>>>>>>>> >>>>>>>>>>>>> Full test exception can be seen here: >>>>>>>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/testReport/junit/%28root%29/004_basic_sanity/vm_run/ >>>>>>>>>>>>> >>>>>>>>>>>>> Further logs can be seen in Jenkins: >>>>>>>>>>>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/13/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/ >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Barak Korren >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> RHCE, RHCi, RHV-DevOps Team >>>>>>>>>>>>> https://ifireball.wordpress.com/ >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Devel mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Devel mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Eyal Edri >>>>>>>>>>> Associate Manager >>>>>>>>>>> RHV DevOps >>>>>>>>>>> EMEA ENG Virtualization R&D >>>>>>>>>>> Red Hat Israel >>>>>>>>>>> >>>>>>>>>>> phone: +972-9-7692018 >>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list >>>>>>>>> [email protected] >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Eyal Edri >>>>>> Associate Manager >>>>>> RHV DevOps >>>>>> EMEA ENG Virtualization R&D >>>>>> Red Hat Israel >>>>>> >>>>>> phone: +972-9-7692018 >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>> >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> [email protected] >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> [email protected] >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>> Guys, >>>> 4.1 is broken for 11 hours now. >>>> It seems that the issues are accumulating; looking into >>>> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.1/28/artifact/exported-artifacts/basic_suite_4.1.sh-el7/exported-artifacts/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log_ovirt-engine/engine.log >>>> >>>> shows a duplicate entry in disk mapping: >>> >>> interesting error, Arik. Reruns should be added to lago >>> The root cause is few lines above and the same as was fixed yesterday on >>> master. It just needs a backport to 4.1 >>>> 2016-12-21 10:52:51,436-05 ERROR >>>> [org.ovirt.engine.core.bll.RunVmOnceCommand] >>>> (org.ovirt.thread.pool-6-thread-37) [] Error during ValidateFailure.: >>>> java.lang.IllegalStateException: Duplicate key >>>> org.ovirt.engine.core.common.businessentities.storage.DiskVm >>>> Element@8f24f60b >>>> at >>>> java.util.stream.Collectors.lambda$throwingMerger$0(Collecto >>>> rs.java:133) [rt.jar:1.8.0_111] >>>> at java.util.HashMap.merge(HashMa >>>> p.java:1253) [rt.jar:1.8.0_111] >>>> at >>>> java.util.stream.Collectors.lambda$toMap$58(Collectors.java: >>>> 1320) [rt.jar:1.8.0_111] >>>> at java.util.stream.ReduceOps$3Re >>>> ducingSink.accept(ReduceOps. >>>> java:169) [rt.jar:1.8.0_111] >>>> at java.util.ArrayList$ArrayListS >>>> pliterator.forEachRemaining( >>>> ArrayList.java:1374) [rt.jar:1.8.0_111] >>>> at java.util.stream.AbstractPipel >>>> ine.copyInto(AbstractPipeline. >>>> java:481) [rt.jar:1.8.0_111] >>>> at java.util.stream.AbstractPipel >>>> ine.wrapAndCopyInto(AbstractPi >>>> peline.java:471) [rt.jar:1.8.0_111] >>>> at java.util.stream.ReduceOps$Red >>>> uceOp.evaluateSequential(Reduc >>>> eOps.java:708) [rt.jar:1.8.0_111] >>>> at java.util.stream.AbstractPipel >>>> ine.evaluate(AbstractPipeline. >>>> java:234) [rt.jar:1.8.0_111] >>>> at java.util.stream.ReferencePipe >>>> line.collect(ReferencePipeline >>>> .java:499) [rt.jar:1.8.0_111] >>>> at org.ovirt.engine.core.bll.stor >>>> age.disk.DiskHandler.getDiskToDiskVmElementMap(DiskHandler. >>>> java:46) [bll.jar:] >>>> at org.ovirt.engine.core.bll.vali >>>> dator.RunVmValidator.validateDisksPassDiscard(RunVmValidator >>>> .java:318) [bll.jar:] >>>> at org.ovirt.engine.core.bll.vali >>>> dator.RunVmValidator.canRunVm( >>>> RunVmValidator.java:146) [bll.jar:] >>>> at org.ovirt.engine.core.bll.RunV >>>> mCommand.validate(RunVmCommand >>>> .java:920) [bll.jar:] >>>> at org.ovirt.engine.core.bll.RunV >>>> mOnceCommand.validate(RunVmOnc >>>> eCommand.java:58) [bll.jar:] >>>> at org.ovirt.engine.core.bll.Comm >>>> andBase.internalValidate(Comma >>>> ndBase.java:834) [bll.jar:] >>>> at org.ovirt.engine.core.bll.Comm >>>> andBase.executeAction(CommandB >>>> ase.java:388) [bll.jar:] >>>> at org.ovirt.engine.core.bll.RunV >>>> mCommandBase.reexecuteCommand( >>>> RunVmCommandBase.java:163) [bll.jar:] >>>> at org.ovirt.engine.core.bll.RunV >>>> mCommandBase.rerun(RunVmComman >>>> dBase.java:138) [bll.jar:] >>>> at org.ovirt.engine.core.bll.RunV >>>> mCommand.rerun(RunVmCommand.java:327) [bll.jar:] >> >> But as I understood from Arik this is indeed a separate problem in rerun, >> likely caused by recent storage code. Adding Tal >>>> >>>> I suggest we consider reverting to the last good state, and work out the >>>> issues from there. >> >> >> _______________________________________________ >> Devel mailing list >> [email protected] >> http://lists.ovirt.org/mailman/listinfo/devel >> >> > _______________________________________________ Devel mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/devel
