Hi,
Unfortunately CQ tester is still failing even after reverting back to nose.
Now it fails during host0 installation when trying to establish connection
after setup networks. On engine we have below exception:
2020-04-03 00:39:34,107-04 ERROR
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [7fb0fc5b] Host installation
failed for host 'efc5cbff-4255-4c6d-98fd-338169272e15',
'lago-basic-suite-master-host-0': Network error during communication
with the host
2020-04-03 00:39:34,107-04 DEBUG
[org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [7fb0fc5b] Exception:
org.ovirt.engine.core.bll.hostdeploy.VdsInstallException: Network
error during communication with the host
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.configureManagementNetwork(InstallVdsInternalCommand.java:447)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand.executeCommand(InstallVdsInternalCommand.java:166)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1169)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1327)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2003)
at
org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140)
at
org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1387)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:639)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
[email protected]//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
[email protected]//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:79)
at
[email protected]//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89)
at
[email protected]//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102)
at
[email protected]//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at
[email protected]//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216)
at
[email protected]//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418)
at
[email protected]//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
[email protected]//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
at
[email protected]//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at
[email protected]//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:627)
at
[email protected]//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at
[email protected]//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
[email protected]//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at
[email protected]//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at
[email protected]//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
at
[email protected]//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view4.runInternalAction(Unknown
Source)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
[email protected]//org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410)
at
[email protected]//org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134)
at
[email protected]//org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
at
[email protected]//org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68)
at
[email protected]//org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown
Source)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2381)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand.lambda$executeCommand$3(AddVdsCommand.java:219)
at
org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
2020-04-03 00:39:34,111-04 WARN
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable]
(EE-ManagedThreadFactory-engine-Thread-42) [7fb0fc5b] Retry failed
2020-04-03 00:39:34,111-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable]
(EE-ManagedThreadFactory-engine-Thread-42) [7fb0fc5b] null:
java.lang.InterruptedException
at
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
at
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
at
java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.lambda$waitForConnect$0(SSLStompClient.java:111)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.utils.retry.AwaitRetry.retry(AwaitRetry.java:15)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.waitForConnect(SSLStompClient.java:110)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient.sendMessage(SSLStompClient.java:80)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:93)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:92)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$timeBoundPollInternal$1(JsonRpcVdsServer.java:1025)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:499)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$FutureCallable.call(JsonRpcVdsServer.java:488)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
And on VDSM we have some uncatched exception:
2020-04-03 00:37:35,278-0400 INFO (Reactor thread)
[ProtocolDetector.AcceptorImpl] Accepted connection from
::ffff:192.168.201.4:47644 (protocoldetector:61)
2020-04-03 00:37:35,280-0400 ERROR (Reactor thread) [vds.dispatcher]
uncaptured python exception, closing channel
<yajsonrpc.betterAsyncore.Dispatcher connected
('::ffff:192.168.201.4', 47644, 0, 0) at 0x7fb1b7721c50> (<class
'ssl.SSLError'>:[X509] no certificate or crl found (_ssl.c:3771)
[/usr/lib64/python3.6/asyncore.py|readwrite|110]
[/usr/lib64/python3.6/asyncore.py|handle_write_event|442]
[/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|handle_write|75]
[/usr/lib/python3.6/site-packages/yajsonrpc/betterAsyncore.py|_delegate_call|171]
[/usr/lib/python3.6/site-packages/vdsm/sslutils.py|handle_write|190]
[/usr/lib/python3.6/site-packages/vdsm/sslutils.py|_handle_io|194]
[/usr/lib/python3.6/site-packages/vdsm/sslutils.py|_set_up_socket|154])
(betterAsyncore:182)
More details can be found at
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/22094
Marcin/Artur could it be related to fixes around swallowing SSLError
https://gerrit.ovirt.org/108016 ?
Thanks,
Martin
--
Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.
_______________________________________________
Devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/FB5VVAVITIYLPTO4EHJ5PWNCIFH7QWEH/
