[
https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210529#comment-15210529
]
ASF GitHub Bot commented on CLOUDSTACK-9285:
--------------------------------------------
Github user kiwiflyer commented on the pull request:
https://github.com/apache/cloudstack/pull/1430#issuecomment-200918379
@rafaelweingartner Yes, you are correct. I like your suggestion. I'll log
the exception with the failure.
In terms of the persistent connections, I believe the management server
pushes to the agents. It also relies on the agent connection to determine
health information for the agent and the VMs located on each host.
> Cloudstack 4.8 can't connect to XEN and KVM hosts
> -------------------------------------------------
>
> Key: CLOUDSTACK-9285
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Xen
> Affects Versions: 4.8.0
> Environment: CentOS 7
> Reporter: Jens Fettig
> Priority: Blocker
> Attachments: agentfailure.pcap
>
>
> We used Cloudstack 4.7 in our testing environment. For future uses we updated
> to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8
> cloudstack has some problems to connect to the hosts again. Here are the
> error logs.
> {code:title=XEN Server from the managment-server.log:|borderStyle=solid}
> 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly
> connected host 13(xen5)
> 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is
> running...
> 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is
> running...
> 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is
> running...
> 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is
> running...
> 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary
> storage VM for secondary storage host cdsdev-secondary
> 2016-02-15 11:22:11,018 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired
> async-jobs
> 2016-02-15 11:22:11,024 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired
> async-jobs
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool]
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create
> master connection to host(192.168.0.97) , due to The credentials given by the
> user are incorrect, so access has been denied, and you have not been issued a
> session handle.
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager
> Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled,
> Agent event = AgentDisconnected, Host id = 13, name = xen5]
> 2016-02-15 11:22:11,101 WARN [c.c.a.m.ClusteredAgentManagerImpl]
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) can not load
> directly connected host 13(xen5) due to
> com.cloud.utils.exception.CloudRuntimeException: Unable to create master
> connection to host(192.168.0.97) , due to The credentials given by the user
> are incorrect, so access has been denied, and you have not been issued a
> session handle.
> at
> com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827)
> at
> com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158)
> at
> com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
> at
> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: The credentials given by the user are incorrect, so access has
> been denied, and you have not been issued a session handle.
> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
> at com.xensource.xenapi.Session.loginWithPassword(Session.java:587)
> ... 19 more
> {code}
> This seems to be the problem:
> Unable to create master connection to host(192.168.0.97) , due to The
> credentials given by the user are incorrect, so access has been denied, and
> you have not been issued a session handle.
> I updated the vhd-utils like in the upgrade gourde from 4.6:
> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.7.0/upgrade/upgrade-4.6.html
> But maybe a need a new vhd-utils file for 4.8, because this one was for 4.7?
> {code:title=KVM from the agent.log:|borderStyle=solid}
> 2016-02-15 10:46:20,971 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> Agent started
> 2016-02-15 10:46:20,973 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> Implementation Version is 4.8.0
> 2016-02-15 10:46:20,974 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> agent.properties found at /etc/cloudstack/agent/agent.properties
> 2016-02-15 10:46:20,977 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> Defaulting to using properties file for storage
> 2016-02-15 10:46:20,978 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> Defaulting to the constant time backoff algorithm
> 2016-02-15 10:46:20,988 INFO [cloud.utils.LogUtils] (main:null) (logid:)
> log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
> 2016-02-15 10:46:20,999 INFO [cloud.agent.AgentShell] (main:null) (logid:)
> Preferring IPv4 address family for agent connection
> 2016-02-15 10:46:21,095 INFO [cloud.agent.Agent] (main:null) (logid:) id is
> 17
> 2016-02-15 10:46:21,116 INFO [kvm.resource.LibvirtConnection] (main:null)
> (logid:) No existing libvirtd connection found. Opening a new one
> 2016-02-15 10:46:21,373 INFO [org.reflections.Reflections] (main:null)
> (logid:) Reflections took 66 ms to scan 1 urls, producing 7 keys and 10 values
> 2016-02-15 10:46:21,514 INFO [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) No libvirt.vif.driver specified. Defaults to
> BridgeVifDriver.
> 2016-02-15 10:46:21,543 INFO [cloud.agent.Agent] (main:null) (logid:) Agent
> [id = 17 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5
> : host = 192.168.0.72 : port = 8250
> 2016-02-15 10:46:21,549 INFO [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 192.168.0.72:8250
> 2016-02-15 10:46:21,609 ERROR [utils.nio.NioConnection] (main:null) (logid:)
> Unable to initialize the threads.
> java.io.IOException: Connection closed with -1 on reading size.
> at com.cloud.utils.nio.Link.doHandshake(Link.java:513)
> at com.cloud.utils.nio.NioClient.init(NioClient.java:80)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:228)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at
> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:461)
> at com.cloud.agent.AgentShell.main(AgentShell.java:496)
> 2016-02-15 10:46:21,611 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) (logid:) Could not find exception:
> com.cloud.utils.exception.NioConnectionException in error code list for
> exceptions
> 2016-02-15 10:46:21,612 ERROR [cloud.agent.AgentShell] (main:null) (logid:)
> Unable to start agent:
> com.cloud.utils.exception.CloudRuntimeException: Unable to start the
> connection!
> at com.cloud.agent.Agent.start(Agent.java:230)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at
> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:461)
> at com.cloud.agent.AgentShell.main(AgentShell.java:496)
> Caused by: com.cloud.utils.exception.NioConnectionException: Connection
> closed with -1 on reading size.
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:94)
> at com.cloud.agent.Agent.start(Agent.java:228)
> ... 5 more
> Caused by: java.io.IOException: Connection closed with -1 on reading size.
> at com.cloud.utils.nio.Link.doHandshake(Link.java:513)
> at com.cloud.utils.nio.NioClient.init(NioClient.java:80)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> ... 6 more
> {code}
> cloudstack-agent on the kvm host and the cloudstack-management on the
> cloudstack server is running. Everything on version 4.8.0.
> Maybe there are some bugs or did I something wrong?
> Greetings J. Fettig
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
