Re: CCS (Cloudstack Container Service) in CloudStack 4.11

[Parth Patel]

Oh, ok. Thanks for the info Daan.

On Tue 13 Mar, 2018, 23:32 Daan Hoogland,  wrote:

> Don't bother. Ccs will not work with 4.9
>
> Biligual auto correct use.  Read at your own risico
>
> On 13 Mar 2018 18:05, "Parth Patel"  wrote:
>
> Hi Da'an,
>
> I will give that a try, till then I will try the ccs first with ACS 4.9 :P
>
> Thanks,
> Parth Patel
>
> On Tue, 13 Mar 2018 at 17:39 Daan Hoogland 
> wrote:
>
> > Parth, The plugin 1.0 won't work with any version beyond 4.9 (4.7 and 4.7
> > aren't tested). There is a branch, 'on-top-of-pr-2071-for-4.10
> > ',
> that
> > we are now porting to 4.11. If you have capability to build it you can
> try
> > that.
> >
> > On Tue, Mar 13, 2018 at 1:03 PM, Parth Patel 
> > wrote:
> >
> > > Hi,
> > >
> > > Maybe somebody working in ShapeBlue can help me. I am trying to
> integrate
> > > CCS service (https://github.com/shapeblue/ccs) with ACS 4.11 using
> > > http://downloads.shapeblue.com/ccs/1.0/Installation_and_
> > > Administration_Guide.pdf
> > > and
> > > I quickly came to know about huge architecture changes that ACS has
> gone
> > > through in the latest release. I somehow managed to bring the UI by
> > editing
> > > and placing the ccs folder in the new "webapp" folder instead of
> > "webapps"
> > > folder. But I am confused as to where to place the shapeblue-ccs and
> > > flyaway core jar files. I tried placing them in
> > > /usr/share/cloudstack-management/lib folder but it gave the following
> > > error
> > > in management-server.log:
> > >
> > >
> > > 2018-03-13 16:59:27,953 INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > > (main:null) (logid:) Loading module context [ccs] from URL
> > > [jar:file:/usr/share/cloudstack-management/webapp/
> > > WEB-INF/lib/cloud-plugin-shapeblue-ccs-1.0.1.jar!/META-
> > > INF/cloudstack/ccs/spring-ccs-context.xml]
> > > 2018-03-13 16:59:27,953 INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > > (main:null) (logid:) Loading module context [ccs] from URL
> > > [jar:file:/usr/share/cloudstack-management/lib/
> > > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/compute/spring-
> > > core-lifecycle-compute-context-inheritable.xml]
> > > 2018-03-13 16:59:27,953 INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > > (main:null) (logid:) Loading module context [ccs] from URL
> > > [jar:file:/usr/share/cloudstack-management/lib/
> > > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/core/spring-
> > > core-lifecycle-core-context-inheritable.xml]
> > > 2018-03-13 16:59:27,953 INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > > (main:null) (logid:) Loading module context [ccs] from URL
> > > [jar:file:/usr/share/cloudstack-management/lib/
> > > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > > framework-config-system-context-inheritable.xml]
> > > 2018-03-13 16:59:27,953 INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > > (main:null) (logid:) Loading module context [ccs] from URL
> > > [jar:file:/usr/share/cloudstack-management/lib/
> > > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > > core-system-context-inheritable.xml]
> > > 2018-03-13 16:59:27,953 INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > > (main:null) (logid:) Loading module context [ccs] from URL
> > > [jar:file:/usr/share/cloudstack-management/lib/
> > > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > > contrail-system-context-inheritable.xml]
> > > 2018-03-13 16:59:27,953 INFO
> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > > (main:null) (logid:) Loading module context [ccs] from URL
> > > [jar:file:/usr/share/cloudstack-management/lib/
> > > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/bootstrap/
> > > spring-bootstrap-context-inheritable.xml]
> > > 2018-03-13 16:59:28,098 WARN  [o.a.c.s.m.c.ResourceApplicationContext]
> > > (main:null) (logid:) Exception encountered during context
> initialization
> > -
> > > cancelling refresh attempt:
> > > org.springframework.beans.factory.BeanCreationException: Error creating
> > > bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> > > failed; nested exception is java.lang.IllegalStateException: Failed to
> > > introspect Class [com.cloud.containercluster.
> ContainerClusterManagerImpl]
> > > from ClassLoader [WebAppClassLoader=500772834@1dd92fe2]
> > > 2018-03-13 16:59:28,100 WARN  [o.e.j.w.WebAppContext] (main:null)
> > (logid:)
> > > Failed startup of context o.e.j.w.WebAppContext@365c30cc
> > > {/client,file:///usr/share/cloudstack-management/webapp/,
> > > UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
> > > org.springframework.beans.factory.BeanCreationException: Error creating
> > > bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> > > failed; nested exception is java.lang.IllegalStateException: Failed to
> > > introspect Class 

Re: Cloudstack installation on Ubuntu Xenial

[Rafael Weingärtner]

Looking at the logs you provided looks like something wrong with the
certificate used to secure communication with your KVM agent. I am not
familiar with KVM and ACS. I know however, that there is a CA pluging that
can issue and install certificates on hosts. Have you tried that?

On Tue, Mar 13, 2018 at 5:07 PM, Daniel Coric  wrote:

> Hello Rafael,
>
> Thank you for your response.
>
> I really did nothing except installing CS on a fresh installed Ubuntu VM -
> as I did it on the CentOS. On the CentOS everything worked out of the box -
> on the Ubuntu problems.
>
> I tried to install it from different package repositories (community,
> ShapeBlue, self-built), compared and followed Ubuntu specific installation
> instructions from two different sources (ACS, ShapeBlue) every time same
> errors in agent.log.
>
> So, I would rather say that there is something wrong either with the
> source or Ubuntu - but, as the first time CS user I could be wrong, of
> course.
>
> Regards
> Daniel Coric
>
> On 2018/03/13 18:43:46, Rafael Weingärtner 
> wrote:
> > The MySQL thing is only a warning and should not cause problems in your
> > POC. The other is an error. There is something wrong with your agent's
> > configurations/deployment.
> >
> > On Mon, Mar 12, 2018 at 9:57 PM, Daniel Coric  wrote:
> >
> > > Hello Everyone,
> > >
> > > I'm getting myself familiar with CloudStack so please excuse if I have
> > > overlooked something obvious.
> > >
> > > Using build and install instructions from the official documentation I
> > > have managed to successfully install CloudStack 4.11 on the neasted
> CentOS
> > > 7.4 KVM (from both community provided package repositories and
> self-built
> > > packages).
> > >
> > > I have tried some of the basic operations like: uploading iso images,
> > > adding volumes and users, creating templates, creating and using VMs
> (both
> > > as admin and user) etc.
> > > As far as I can tell, everything worked as expected - except the fact
> that
> > > CentOS VM took about half an hour to shut down.
> > >
> > > Than I decided to give it a try on Ubuntu too. And indeed, Ubuntu
> 16.04.4
> > > VM shut down normally.
> > >
> > > But, that was also the only thing that worked as expected on that
> Ubuntu
> > > VM.
> > >
> > > I have tried to find some solution on internet but the closest I could
> get
> > > was this thread:
> > > https://www.mail-archive.com/users@cloudstack.apache.org/msg22578.html
> > > and this documentation:
> > > http://docs.cloudstack.apache.org/projects/cloudstack-
> > > administration/en/latest/hosts.html#security
> > >
> > > And I'm not even sure if I am on the right path to the solution - any
> > > assistance would be much appreciated.
> > >
> > >
> > >
> > > Ubuntu 16.04.4 cloudstack-management.err is filled with:
> > >
> > > Mon Mar 12 20:30:24 CET 2018 WARN: Establishing SSL connection without
> > > server's identity verification is not recommended. According to MySQL
> > > 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be
> established
> > > by default if explicit option isn't set. For compliance with existing
> > > applications not using SSL the verifyServerCertificate property is set
> to
> > > 'false'. You need either to explicitly disable SSL by setting
> useSSL=false,
> > > or set useSSL=true and provide truststore for server certificate
> > > verification.
> > >
> > > Ubuntu 16.04.4 agent.log is filled with:
> > >
> > > 2018-03-12 20:43:58,782 INFO  [utils.exception.CSExceptionErrorCode]
> > > (main:null) (logid:) Could not find exception:
> com.cloud.utils.exception.NioConnectionException
> > > in error code list for exceptions
> > > 2018-03-12 20:43:58,782 WARN  [cloud.agent.Agent] (main:null) (logid:)
> NIO
> > > Connection Exception  com.cloud.utils.exception.
> NioConnectionException:
> > > SSL Handshake failed while connecting to host: 10.22.0.5 port: 8250
> > > 2018-03-12 20:43:58,782 INFO  [cloud.agent.Agent] (main:null) (logid:)
> > > Attempted to connect to the server, but received an unexpected
> exception,
> > > trying again...
> > > 2018-03-12 20:44:03,783 INFO  [cloud.agent.Agent] (main:null) (logid:)
> > > Connecting to host:10.22.0.5
> > > 2018-03-12 20:44:03,783 INFO  [utils.nio.NioClient] (main:null)
> (logid:)
> > > Connecting to 10.22.0.5:8250
> > > 2018-03-12 20:44:03,786 INFO  [utils.nio.Link] (main:null) (logid:)
> Conf
> > > file found: /etc/cloudstack/agent/agent.properties
> > > 2018-03-12 20:44:03,787 WARN  [utils.nio.Link] (main:null) (logid:)
> Failed
> > > to load keystore, using trust all manager
> > > 2018-03-12 20:44:03,858 ERROR [utils.nio.Link] (main:null) (logid:) SSL
> > > error caught during unwrap data: Received fatal alert:
> bad_certificate, for
> > > local address=/10.22.0.5:53356, remote address=/10.22.0.5:8250. The
> > > client may have invalid ca-certificates.
> > > 2018-03-12 20:44:03,858 ERROR [utils.nio.NioClient] (main:null)
> (logid:)
> 

RE: KVM HostHA

[Paul Angus]

The management server doesn't ping the host through IPMI.   However if IPMI is 
not available, you will not be able to use Host HA, as there is no way for 
CloudStack to 'fence' the host - that is shut it down to be sure that a VM 
cannot start again on that host.

I can explain why that is necessary if you wish.


Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Parth Patel  
Sent: 13 March 2018 16:57
To: users@cloudstack.apache.org
Cc: Jon Marshall 
Subject: Re: KVM HostHA

Hi Jon and Victor,

I think the management server pings your host using ipmi (I really don't hope 
this is the case).
In my case, I did not have OOBM enabled at all (my hardware didn't support
it)
I think you could disable OOBM and/or HA-Host and give that a try :)

On Tue, 13 Mar 2018 at 20:40 victor  wrote:

> Hello Guys,
>
> I have tried the following two cases.
>
> 1, "echo c > /proc/sysrq-trigger"
>
> 2, Pulled the network cable of one of the host
>
> In both cases, the following happened.
>
> =
> 2018-03-13 08:22:54,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentTaskPool-15:ctx-c8d9f5d2) (logid:c0a3d2da) Notifying other nodes 
> of to disconnect
> 2018-03-13 08:22:54,983 INFO [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is 
> disconnecting with event AgentDisconnected
> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is already 
> Alert
> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Deregistering link 
> for
> 4 with state Alert
> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Remove Agent : 4 
> =
>
> But nothing happened for the  vm's in that node. I have waited for one 
> hour and the VM's in that node has been migrated to the other 
> available hosts. I think the issue is that the management server still 
> thinks that the VM's in that host is running. Please check the 
> following logs
>
> ===
> 2018-03-13 11:08:25,882 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 1 VMs on host 4
> 2018-03-13 11:08:25,888 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 0 VM, not 
> running on host 4 
>
>
> On 03/13/2018 04:20 PM, Jon Marshall wrote:
> > I tried "echo c > /proc/sysrq-trigger" which stopped me getting into 
> > the
> server but it did not stop the server responding to an ipmitool 
> request on the manager eg -
> >
> >
> > "ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis status"
> >
> >
> > from the management server got an answer saying the chassis power 
> > was on
> so CS never registered the compute node as down.
> >
> >
> > I am obviously doing something wrong but cannot work it out.
> >
> >
> > The management server has one NIC - 172.16.7.4
> >
> >
> > Each compute node has 3 NICs -
> >
> >
> > cnode1
> cnode2
> >
> >
> > mangement NIC172.16.7.5   172.16.7.6
> >
> > vm NIC  172.16.6.130 172.16.6.131
> >
> > storage - 172.16.250.4   172.16.250.5
> >
> >
> > Dell LOM (for Idrac)   172.16.7.29172.16.7.30
> >
> >
> > the dell LOM IPs are the ones used to configure OOBM  in the UI
> >
> >
> >
> > If I pull the storage NIC presumably nothing will happen as the 
> > ipmitool
> check is running across the management NIC so I need to pull both ?
> >
> > My understanding of host HA was the management server monitored the
> compute nodes using ipmitool and if it did not get a response because 
> the host was down it would fence off that host and move the VMs to an 
> active compute node.
> >
> > This is obviously too simplistic so could someone explain how it is
> meant to work and what it is protecting against ?
> >
> > 
> > From: Paul Angus 
> > Sent: 13 March 2018 07:01
> > To: users@cloudstack.apache.org
> > Subject: RE: KVM HostHA
> >
> > Hi all,
> >
> > One small note, unplugging the management NIC will only cause an HA
> event if the storage is running over that NIC also.
> >
> > Is the storage is over a separate NIC then, the guest VMs will 
> > continue
> to run when the mgmt. NIC is unplugged, Host HA will detect the disk 
> activity and conclude that there is nothing it can do, as the VMs are 
> still running other than mark the hosts as degraded.
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<
> http://www.shapeblue.com/>
> >
> 

Re: Cloudstack installation on Ubuntu Xenial

[Daniel Coric]

Hello Rafael,

Thank you for your response. 

I really did nothing except installing CS on a fresh installed Ubuntu VM - as I 
did it on the CentOS. On the CentOS everything worked out of the box - on the 
Ubuntu problems. 

I tried to install it from different package repositories (community, 
ShapeBlue, self-built), compared and followed Ubuntu specific installation 
instructions from two different sources (ACS, ShapeBlue) every time same errors 
in agent.log.

So, I would rather say that there is something wrong either with the source or 
Ubuntu - but, as the first time CS user I could be wrong, of course.

Regards
Daniel Coric

On 2018/03/13 18:43:46, Rafael Weingärtner  
wrote: 
> The MySQL thing is only a warning and should not cause problems in your
> POC. The other is an error. There is something wrong with your agent's
> configurations/deployment.
> 
> On Mon, Mar 12, 2018 at 9:57 PM, Daniel Coric  wrote:
> 
> > Hello Everyone,
> >
> > I'm getting myself familiar with CloudStack so please excuse if I have
> > overlooked something obvious.
> >
> > Using build and install instructions from the official documentation I
> > have managed to successfully install CloudStack 4.11 on the neasted CentOS
> > 7.4 KVM (from both community provided package repositories and self-built
> > packages).
> >
> > I have tried some of the basic operations like: uploading iso images,
> > adding volumes and users, creating templates, creating and using VMs (both
> > as admin and user) etc.
> > As far as I can tell, everything worked as expected - except the fact that
> > CentOS VM took about half an hour to shut down.
> >
> > Than I decided to give it a try on Ubuntu too. And indeed, Ubuntu 16.04.4
> > VM shut down normally.
> >
> > But, that was also the only thing that worked as expected on that Ubuntu
> > VM.
> >
> > I have tried to find some solution on internet but the closest I could get
> > was this thread:
> > https://www.mail-archive.com/users@cloudstack.apache.org/msg22578.html
> > and this documentation:
> > http://docs.cloudstack.apache.org/projects/cloudstack-
> > administration/en/latest/hosts.html#security
> >
> > And I'm not even sure if I am on the right path to the solution - any
> > assistance would be much appreciated.
> >
> >
> >
> > Ubuntu 16.04.4 cloudstack-management.err is filled with:
> >
> > Mon Mar 12 20:30:24 CET 2018 WARN: Establishing SSL connection without
> > server's identity verification is not recommended. According to MySQL
> > 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> > by default if explicit option isn't set. For compliance with existing
> > applications not using SSL the verifyServerCertificate property is set to
> > 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> > or set useSSL=true and provide truststore for server certificate
> > verification.
> >
> > Ubuntu 16.04.4 agent.log is filled with:
> >
> > 2018-03-12 20:43:58,782 INFO  [utils.exception.CSExceptionErrorCode]
> > (main:null) (logid:) Could not find exception: 
> > com.cloud.utils.exception.NioConnectionException
> > in error code list for exceptions
> > 2018-03-12 20:43:58,782 WARN  [cloud.agent.Agent] (main:null) (logid:) NIO
> > Connection Exception  com.cloud.utils.exception.NioConnectionException:
> > SSL Handshake failed while connecting to host: 10.22.0.5 port: 8250
> > 2018-03-12 20:43:58,782 INFO  [cloud.agent.Agent] (main:null) (logid:)
> > Attempted to connect to the server, but received an unexpected exception,
> > trying again...
> > 2018-03-12 20:44:03,783 INFO  [cloud.agent.Agent] (main:null) (logid:)
> > Connecting to host:10.22.0.5
> > 2018-03-12 20:44:03,783 INFO  [utils.nio.NioClient] (main:null) (logid:)
> > Connecting to 10.22.0.5:8250
> > 2018-03-12 20:44:03,786 INFO  [utils.nio.Link] (main:null) (logid:) Conf
> > file found: /etc/cloudstack/agent/agent.properties
> > 2018-03-12 20:44:03,787 WARN  [utils.nio.Link] (main:null) (logid:) Failed
> > to load keystore, using trust all manager
> > 2018-03-12 20:44:03,858 ERROR [utils.nio.Link] (main:null) (logid:) SSL
> > error caught during unwrap data: Received fatal alert: bad_certificate, for
> > local address=/10.22.0.5:53356, remote address=/10.22.0.5:8250. The
> > client may have invalid ca-certificates.
> > 2018-03-12 20:44:03,858 ERROR [utils.nio.NioClient] (main:null) (logid:)
> > SSL Handshake failed while connecting to host: 10.22.0.5 port: 8250
> > 2018-03-12 20:44:03,858 ERROR [utils.nio.NioConnection] (main:null)
> > (logid:) Unable to initialize the threads.
> > java.io.IOException: SSL Handshake failed while connecting to host:
> > 10.22.0.5 port: 8250
> > at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
> > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
> > at com.cloud.agent.Agent.start(Agent.java:263)
> > at 

Re: Cloudstack installation on Ubuntu Xenial

[Rafael Weingärtner]

The MySQL thing is only a warning and should not cause problems in your
POC. The other is an error. There is something wrong with your agent's
configurations/deployment.

On Mon, Mar 12, 2018 at 9:57 PM, Daniel Coric  wrote:

> Hello Everyone,
>
> I'm getting myself familiar with CloudStack so please excuse if I have
> overlooked something obvious.
>
> Using build and install instructions from the official documentation I
> have managed to successfully install CloudStack 4.11 on the neasted CentOS
> 7.4 KVM (from both community provided package repositories and self-built
> packages).
>
> I have tried some of the basic operations like: uploading iso images,
> adding volumes and users, creating templates, creating and using VMs (both
> as admin and user) etc.
> As far as I can tell, everything worked as expected - except the fact that
> CentOS VM took about half an hour to shut down.
>
> Than I decided to give it a try on Ubuntu too. And indeed, Ubuntu 16.04.4
> VM shut down normally.
>
> But, that was also the only thing that worked as expected on that Ubuntu
> VM.
>
> I have tried to find some solution on internet but the closest I could get
> was this thread:
> https://www.mail-archive.com/users@cloudstack.apache.org/msg22578.html
> and this documentation:
> http://docs.cloudstack.apache.org/projects/cloudstack-
> administration/en/latest/hosts.html#security
>
> And I'm not even sure if I am on the right path to the solution - any
> assistance would be much appreciated.
>
>
>
> Ubuntu 16.04.4 cloudstack-management.err is filled with:
>
> Mon Mar 12 20:30:24 CET 2018 WARN: Establishing SSL connection without
> server's identity verification is not recommended. According to MySQL
> 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established
> by default if explicit option isn't set. For compliance with existing
> applications not using SSL the verifyServerCertificate property is set to
> 'false'. You need either to explicitly disable SSL by setting useSSL=false,
> or set useSSL=true and provide truststore for server certificate
> verification.
>
> Ubuntu 16.04.4 agent.log is filled with:
>
> 2018-03-12 20:43:58,782 INFO  [utils.exception.CSExceptionErrorCode]
> (main:null) (logid:) Could not find exception: 
> com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2018-03-12 20:43:58,782 WARN  [cloud.agent.Agent] (main:null) (logid:) NIO
> Connection Exception  com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 10.22.0.5 port: 8250
> 2018-03-12 20:43:58,782 INFO  [cloud.agent.Agent] (main:null) (logid:)
> Attempted to connect to the server, but received an unexpected exception,
> trying again...
> 2018-03-12 20:44:03,783 INFO  [cloud.agent.Agent] (main:null) (logid:)
> Connecting to host:10.22.0.5
> 2018-03-12 20:44:03,783 INFO  [utils.nio.NioClient] (main:null) (logid:)
> Connecting to 10.22.0.5:8250
> 2018-03-12 20:44:03,786 INFO  [utils.nio.Link] (main:null) (logid:) Conf
> file found: /etc/cloudstack/agent/agent.properties
> 2018-03-12 20:44:03,787 WARN  [utils.nio.Link] (main:null) (logid:) Failed
> to load keystore, using trust all manager
> 2018-03-12 20:44:03,858 ERROR [utils.nio.Link] (main:null) (logid:) SSL
> error caught during unwrap data: Received fatal alert: bad_certificate, for
> local address=/10.22.0.5:53356, remote address=/10.22.0.5:8250. The
> client may have invalid ca-certificates.
> 2018-03-12 20:44:03,858 ERROR [utils.nio.NioClient] (main:null) (logid:)
> SSL Handshake failed while connecting to host: 10.22.0.5 port: 8250
> 2018-03-12 20:44:03,858 ERROR [utils.nio.NioConnection] (main:null)
> (logid:) Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 10.22.0.5 port: 8250
> at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
> at com.cloud.agent.Agent.start(Agent.java:263)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:410)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(
> AgentShell.java:378)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:362)
> at com.cloud.agent.AgentShell.start(AgentShell.java:467)
> at com.cloud.agent.AgentShell.main(AgentShell.java:502)
>
>
>
> Regards
> D.Coric
>



-- 
Rafael Weingärtner

Re: CCS (Cloudstack Container Service) in CloudStack 4.11

[Daan Hoogland]

Don't bother. Ccs will not work with 4.9

Biligual auto correct use.  Read at your own risico

On 13 Mar 2018 18:05, "Parth Patel"  wrote:

Hi Da'an,

I will give that a try, till then I will try the ccs first with ACS 4.9 :P

Thanks,
Parth Patel

On Tue, 13 Mar 2018 at 17:39 Daan Hoogland  wrote:

> Parth, The plugin 1.0 won't work with any version beyond 4.9 (4.7 and 4.7
> aren't tested). There is a branch, 'on-top-of-pr-2071-for-4.10
> ', that
> we are now porting to 4.11. If you have capability to build it you can try
> that.
>
> On Tue, Mar 13, 2018 at 1:03 PM, Parth Patel 
> wrote:
>
> > Hi,
> >
> > Maybe somebody working in ShapeBlue can help me. I am trying to
integrate
> > CCS service (https://github.com/shapeblue/ccs) with ACS 4.11 using
> > http://downloads.shapeblue.com/ccs/1.0/Installation_and_
> > Administration_Guide.pdf
> > and
> > I quickly came to know about huge architecture changes that ACS has gone
> > through in the latest release. I somehow managed to bring the UI by
> editing
> > and placing the ccs folder in the new "webapp" folder instead of
> "webapps"
> > folder. But I am confused as to where to place the shapeblue-ccs and
> > flyaway core jar files. I tried placing them in
> > /usr/share/cloudstack-management/lib folder but it gave the following
> > error
> > in management-server.log:
> >
> >
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/webapp/
> > WEB-INF/lib/cloud-plugin-shapeblue-ccs-1.0.1.jar!/META-
> > INF/cloudstack/ccs/spring-ccs-context.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/compute/spring-
> > core-lifecycle-compute-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/core/spring-
> > core-lifecycle-core-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > framework-config-system-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > core-system-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > contrail-system-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/bootstrap/
> > spring-bootstrap-context-inheritable.xml]
> > 2018-03-13 16:59:28,098 WARN  [o.a.c.s.m.c.ResourceApplicationContext]
> > (main:null) (logid:) Exception encountered during context initialization
> -
> > cancelling refresh attempt:
> > org.springframework.beans.factory.BeanCreationException: Error creating
> > bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> > failed; nested exception is java.lang.IllegalStateException: Failed to
> > introspect Class [com.cloud.containercluster.
ContainerClusterManagerImpl]
> > from ClassLoader [WebAppClassLoader=500772834@1dd92fe2]
> > 2018-03-13 16:59:28,100 WARN  [o.e.j.w.WebAppContext] (main:null)
> (logid:)
> > Failed startup of context o.e.j.w.WebAppContext@365c30cc
> > {/client,file:///usr/share/cloudstack-management/webapp/,
> > UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
> > org.springframework.beans.factory.BeanCreationException: Error creating
> > bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> > failed; nested exception is java.lang.IllegalStateException: Failed to
> > introspect Class [com.cloud.containercluster.
ContainerClusterManagerImpl]
> > from ClassLoader [WebAppClassLoader=500772834@1dd92fe2]
> > at
> > org.springframework.beans.factory.annotation.
> > AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(
> > AutowiredAnnotationBeanPostProcessor.java:262)
> > at
> > 

Re: Migrate system VMs volumes to new storage

[Andrija Panic]

Good, cheers

On 13 March 2018 at 15:37, Ugo Vasi  wrote:

> Hi, I put the server in maintenance mode and then I could delete it
> directly from the web interface.
>
> Bye
>
>
> Il 13/03/2018 15:25, Ugo Vasi ha scritto:
>
>> Hi Andrija,
>> I followed your procedures and confirm that it works fine!
>> Now I have to dismiss the storage server and I'm wondering if you just
>> put it in maintenance mode forever or if there's another way to delete it.
>>
>> Thanks
>>
>> Il 15/02/2018 18:03, Andrija Panic ha scritto:
>>
>>> Hi Ugo,
>>>
>>> though not preferred at all, feel free to start DB experiments :) since
>>> this is very often required (well, in some cases) - this is not the rule,
>>> buy often you have to make your hand dirty.
>>>
>>> For you specific issue, here is output of my DB below
>>>
>>> The important thing is obviously the TAGS column, just enter value which
>>> you also added to Storage Tags and that is all (wait for 5min, go get some
>>> coffee, not sure why, needs some time to be applied)
>>>
>>> Also note the special column "unique_name" - if you have created
>>> multiple offerings for i.e. SSVM (more CPU or more RAM, whatever...) then
>>> ACS needs to know WHICH of the offering for SSVM to actually use when
>>> creating new SSVM.
>>> SO make sure that, in this specific example, you move the value "
>>> Cloud.com-SecondaryStorage" to correct ROW - only single ROW can exist with
>>> this specific value.
>>>
>>>
>>> FOr LOCAL systemVMs, there is similar record, but I guess you are on
>>> shared storage anyway
>>>
>>>
>>> mysql> select id,domain_id,name,display_text,tags,unique_name,system_use
>>> from disk_offering where name like "%system_offering%%" and name not like
>>> "%local%";
>>> +-+---+-
>>> ---+
>>> ++++
>>>
>>> | id  | domain_id | name  | display_text
>>>  | tags   | unique_name   | system_use |
>>> +-+---+-
>>> ---+
>>> ++++
>>>
>>> |  19 |  NULL | System Offering For Software Router 1Gbps
>>> | System Offering For Software Router 1Gbps   | SolidFire1 |
>>> Cloud.Com-SoftwareRouter   |  1 |
>>> |  22 |  NULL | System Offering For Internal LB VM| NULL
>>>  | SolidFire1 | Cloud.Com-InternalLBVm|  1 |
>>> |  25 |  NULL | System Offering For Console Proxy | NULL
>>>| SolidFire1 | Cloud.com-ConsoleProxy |
>>> 1 |
>>> |  28 |  NULL | System Offering For Secondary Storage VM
>>>| NULL  | SolidFire1 | Cloud.com-SecondaryStorage |
>>> 1 |
>>> |  31 |  NULL | System Offering For Elastic LB VM | NULL
>>>| SolidFire1 | Cloud.Com-ElasticLBVm |
>>> 1 |
>>> | 166 |  NULL | System Offering For Software Router 500Mbps
>>>   | System Offering For Software Router 500Mbps   | SolidFire1
>>> | NULL  |  1 |
>>> | 169 |  NULL | System Offering For Software Router 200Mbps
>>>   | System Offering For Software Router 200Mbps   | SolidFire1
>>> | NULL  |  1 |
>>> | 243 |  NULL | System Offering For Software Router 100Mbps
>>>   | System Offering For Software Router 100Mbps   | SolidFire1
>>> | NULL  |  1 |
>>> +-+---+-
>>> ---+
>>> ++++
>>>
>>>
>>> Let me know if you need more help
>>>
>>> Cheers
>>>
>>>
>>> On 14 February 2018 at 12:10, Ugo Vasi > ugo.v...@procne.it>> wrote:
>>>
>>> Hi Andrija,
>>> the web interface does not allow to add / change tags for system
>>> VM system offerings.
>>>
>>> In a test system with the same configuration (as much as possible)
>>> I added a new entry for the proxy console with the tags for the
>>> new strorage and, after destroying the console VM, it was
>>> recreated in the new storage as I expected.
>>>
>>> I tried to take the same steps in the production environment but
>>> the system persists in recreating the VM in the old storage. I
>>> also tried to change the order of appearance of the system
>>> offerings list but the result does not change.
>>>
>>> I thought about intervening in the database but I do not like it
>>> as a solution.
>>>
>>> Any idea?
>>>
>>>
>>>
>>>
>>> Il 07/02/2018 21:30, Andrija Panic ha scritto:
>>>
>>> Hi Ugo,
>>>
>>> I dont know if you have multiple SSVM (and other system)
>>> offerings, 

Re: CCS (Cloudstack Container Service) in CloudStack 4.11

[Parth Patel]

Hi Da'an,

I will give that a try, till then I will try the ccs first with ACS 4.9 :P

Thanks,
Parth Patel

On Tue, 13 Mar 2018 at 17:39 Daan Hoogland  wrote:

> Parth, The plugin 1.0 won't work with any version beyond 4.9 (4.7 and 4.7
> aren't tested). There is a branch, 'on-top-of-pr-2071-for-4.10
> ', that
> we are now porting to 4.11. If you have capability to build it you can try
> that.
>
> On Tue, Mar 13, 2018 at 1:03 PM, Parth Patel 
> wrote:
>
> > Hi,
> >
> > Maybe somebody working in ShapeBlue can help me. I am trying to integrate
> > CCS service (https://github.com/shapeblue/ccs) with ACS 4.11 using
> > http://downloads.shapeblue.com/ccs/1.0/Installation_and_
> > Administration_Guide.pdf
> > and
> > I quickly came to know about huge architecture changes that ACS has gone
> > through in the latest release. I somehow managed to bring the UI by
> editing
> > and placing the ccs folder in the new "webapp" folder instead of
> "webapps"
> > folder. But I am confused as to where to place the shapeblue-ccs and
> > flyaway core jar files. I tried placing them in
> > /usr/share/cloudstack-management/lib folder but it gave the following
> > error
> > in management-server.log:
> >
> >
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/webapp/
> > WEB-INF/lib/cloud-plugin-shapeblue-ccs-1.0.1.jar!/META-
> > INF/cloudstack/ccs/spring-ccs-context.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/compute/spring-
> > core-lifecycle-compute-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/core/spring-
> > core-lifecycle-core-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > framework-config-system-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > core-system-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> > contrail-system-context-inheritable.xml]
> > 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> > (main:null) (logid:) Loading module context [ccs] from URL
> > [jar:file:/usr/share/cloudstack-management/lib/
> > cloudstack-4.11.0.0.jar!/META-INF/cloudstack/bootstrap/
> > spring-bootstrap-context-inheritable.xml]
> > 2018-03-13 16:59:28,098 WARN  [o.a.c.s.m.c.ResourceApplicationContext]
> > (main:null) (logid:) Exception encountered during context initialization
> -
> > cancelling refresh attempt:
> > org.springframework.beans.factory.BeanCreationException: Error creating
> > bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> > failed; nested exception is java.lang.IllegalStateException: Failed to
> > introspect Class [com.cloud.containercluster.ContainerClusterManagerImpl]
> > from ClassLoader [WebAppClassLoader=500772834@1dd92fe2]
> > 2018-03-13 16:59:28,100 WARN  [o.e.j.w.WebAppContext] (main:null)
> (logid:)
> > Failed startup of context o.e.j.w.WebAppContext@365c30cc
> > {/client,file:///usr/share/cloudstack-management/webapp/,
> > UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
> > org.springframework.beans.factory.BeanCreationException: Error creating
> > bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> > failed; nested exception is java.lang.IllegalStateException: Failed to
> > introspect Class [com.cloud.containercluster.ContainerClusterManagerImpl]
> > from ClassLoader [WebAppClassLoader=500772834@1dd92fe2]
> > at
> > org.springframework.beans.factory.annotation.
> > AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(
> > AutowiredAnnotationBeanPostProcessor.java:262)
> > at
> > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> > tory.determineConstructorsFromBeanPostProcessors(
> > AbstractAutowireCapableBeanFactory.java:1198)
> > at
> > 

Re: KVM HostHA

[Parth Patel]

Hi Jon and Victor,

I think the management server pings your host using ipmi (I really don't
hope this is the case).
In my case, I did not have OOBM enabled at all (my hardware didn't support
it)
I think you could disable OOBM and/or HA-Host and give that a try :)

On Tue, 13 Mar 2018 at 20:40 victor  wrote:

> Hello Guys,
>
> I have tried the following two cases.
>
> 1, "echo c > /proc/sysrq-trigger"
>
> 2, Pulled the network cable of one of the host
>
> In both cases, the following happened.
>
> =
> 2018-03-13 08:22:54,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentTaskPool-15:ctx-c8d9f5d2) (logid:c0a3d2da) Notifying other nodes
> of to disconnect
> 2018-03-13 08:22:54,983 INFO [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is disconnecting
> with event AgentDisconnected
> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is already Alert
> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Deregistering link for
> 4 with state Alert
> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Remove Agent : 4
> =
>
> But nothing happened for the  vm's in that node. I have waited for one
> hour and the VM's in that node has been migrated to the other available
> hosts. I think the issue is that the management server still thinks that
> the VM's in that host is running. Please check the following logs
>
> ===
> 2018-03-13 11:08:25,882 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 1 VMs on host 4
> 2018-03-13 11:08:25,888 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 0 VM, not running
> on host 4
> 
>
>
> On 03/13/2018 04:20 PM, Jon Marshall wrote:
> > I tried "echo c > /proc/sysrq-trigger" which stopped me getting into the
> server but it did not stop the server responding to an ipmitool request on
> the manager eg -
> >
> >
> > "ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis status"
> >
> >
> > from the management server got an answer saying the chassis power was on
> so CS never registered the compute node as down.
> >
> >
> > I am obviously doing something wrong but cannot work it out.
> >
> >
> > The management server has one NIC - 172.16.7.4
> >
> >
> > Each compute node has 3 NICs -
> >
> >
> > cnode1
> cnode2
> >
> >
> > mangement NIC172.16.7.5   172.16.7.6
> >
> > vm NIC  172.16.6.130 172.16.6.131
> >
> > storage - 172.16.250.4   172.16.250.5
> >
> >
> > Dell LOM (for Idrac)   172.16.7.29172.16.7.30
> >
> >
> > the dell LOM IPs are the ones used to configure OOBM  in the UI
> >
> >
> >
> > If I pull the storage NIC presumably nothing will happen as the ipmitool
> check is running across the management NIC so I need to pull both ?
> >
> > My understanding of host HA was the management server monitored the
> compute nodes using ipmitool and if it did not get a response because the
> host was down it would fence off that host and move the VMs to an active
> compute node.
> >
> > This is obviously too simplistic so could someone explain how it is
> meant to work and what it is protecting against ?
> >
> > 
> > From: Paul Angus 
> > Sent: 13 March 2018 07:01
> > To: users@cloudstack.apache.org
> > Subject: RE: KVM HostHA
> >
> > Hi all,
> >
> > One small note, unplugging the management NIC will only cause an HA
> event if the storage is running over that NIC also.
> >
> > Is the storage is over a separate NIC then, the guest VMs will continue
> to run when the mgmt. NIC is unplugged, Host HA will detect the disk
> activity and conclude that there is nothing it can do, as the VMs are still
> running other than mark the hosts as degraded.
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<
> http://www.shapeblue.com/>
> >
> > Shapeblue - The CloudStack Company
> > www.shapeblue.com
> > Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is
> a framework developed by ShapeBlue to deli
> ver
> the rapid deployment of a standardised ...
> >
> >
> >
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Parth Patel 
> > Sent: 12 March 2018 17:35
> > To: users@cloudstack.apache.org
> > Subject: Re: KVM HostHA
> >
> >> Hi Jon,
> >>
> >> As I said, in my case, making the host HA didn't work 

RE: new version 4.9.3 coming

[Jochim, Ingo]

There is a bug in DNS which got fixed in 4.11
We hoped to get a fixed 4.9 version which includes that.

Thanks,
Ingo

-Original Message-
From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] 
Sent: Dienstag, 13. März 2018 15:54
To: users 
Subject: Re: new version 4.9.3 coming

There is one thing that escape my memory. There is a security release about
to come for 4.9.3.0. However, other issues/improvements/bug fixes that are
on 4.11 will not go into 4.9.3.0.

On Tue, Mar 13, 2018 at 11:50 AM, Jochim, Ingo 
wrote:

> Hi Rafael,
>
> ok, thanks.
>
> Regards,
> Ingo
>
> -Original Message-
> From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> Sent: Dienstag, 13. März 2018 11:48
> To: users 
> Subject: Re: new version 4.9.3 coming
>
> Not that I know of. I believe the 4.9.* are end of life this July.
> Therefore, I would not expect anything other than very critical things to
> be into them.
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/LTS
>
> On Tue, Mar 13, 2018 at 7:36 AM, Jochim, Ingo 
> wrote:
>
> > Hello all,
> >
> > are there any plans to release a version 4.9.3 which includes the latest
> > bug fixes available in 4.11?
> >
> > Many thanks already.
> > Regards,
> > Ingo
> >
>
>
>
> --
> Rafael Weingärtner
>



-- 
Rafael Weingärtner

Re: KVM HostHA

[victor]

Hello Guys,

I have tried the following two cases.

1, "echo c > /proc/sysrq-trigger"

2, Pulled the network cable of one of the host

In both cases, the following happened.

=
2018-03-13 08:22:54,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
(AgentTaskPool-15:ctx-c8d9f5d2) (logid:c0a3d2da) Notifying other nodes 
of to disconnect
2018-03-13 08:22:54,983 INFO [c.c.a.m.AgentManagerImpl] 
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is disconnecting 
with event AgentDisconnected
2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is already Alert
2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Deregistering link for 
4 with state Alert
2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl] 
(AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Remove Agent : 4

=

But nothing happened for the  vm's in that node. I have waited for one 
hour and the VM's in that node has been migrated to the other available 
hosts. I think the issue is that the management server still thinks that 
the VM's in that host is running. Please check the following logs


===
2018-03-13 11:08:25,882 DEBUG [c.c.c.CapacityManagerImpl] 
(CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 1 VMs on host 4
2018-03-13 11:08:25,888 DEBUG [c.c.c.CapacityManagerImpl] 
(CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 0 VM, not running 
on host 4




On 03/13/2018 04:20 PM, Jon Marshall wrote:

I tried "echo c > /proc/sysrq-trigger" which stopped me getting into the server 
but it did not stop the server responding to an ipmitool request on the manager eg -


"ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis status"


from the management server got an answer saying the chassis power was on so CS 
never registered the compute node as down.


I am obviously doing something wrong but cannot work it out.


The management server has one NIC - 172.16.7.4


Each compute node has 3 NICs -


cnode1cnode2


mangement NIC172.16.7.5   172.16.7.6

vm NIC  172.16.6.130 172.16.6.131

storage - 172.16.250.4   172.16.250.5


Dell LOM (for Idrac)   172.16.7.29172.16.7.30


the dell LOM IPs are the ones used to configure OOBM  in the UI



If I pull the storage NIC presumably nothing will happen as the ipmitool check 
is running across the management NIC so I need to pull both ?

My understanding of host HA was the management server monitored the compute 
nodes using ipmitool and if it did not get a response because the host was down 
it would fence off that host and move the VMs to an active compute node.

This is obviously too simplistic so could someone explain how it is meant to 
work and what it is protecting against ?


From: Paul Angus 
Sent: 13 March 2018 07:01
To: users@cloudstack.apache.org
Subject: RE: KVM HostHA

Hi all,

One small note, unplugging the management NIC will only cause an HA event if 
the storage is running over that NIC also.

Is the storage is over a separate NIC then, the guest VMs will continue to run 
when the mgmt. NIC is unplugged, Host HA will detect the disk activity and 
conclude that there is nothing it can do, as the VMs are still running other 
than mark the hosts as degraded.


Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

Shapeblue - The CloudStack Company
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a 
framework developed by ShapeBlue to deliver the rapid deployment of a 
standardised ...



53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-Original Message-
From: Parth Patel 
Sent: 12 March 2018 17:35
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA


Hi Jon,

As I said, in my case, making the host HA didn't work but by just
having a HA VM running on host and executing - (WARNING) "echo c >
/proc/sysrq-trigger" to simulate a kernel crash on host, the
management server registered it as down and started the VM on another
host. I know I've suggested this before but I insist you give this a
try. Also, you don't need to completely power off the machine manually
but just plugging out the network cable works fine. The cloudstack
agent after losing connection to management server auto reboots
because of KVM heartbeat check shell script mentioned by Rohit Yadav
to one of my earlier queries in other thread.

On Mon 12 Mar, 2018, 21:23 Jon Marshall,  wrote:
Hi Paul


Thanks for the response.


I think I am not understanding how it was meant to work 

Re: new version 4.9.3 coming

[Rafael Weingärtner]

There is one thing that escape my memory. There is a security release about
to come for 4.9.3.0. However, other issues/improvements/bug fixes that are
on 4.11 will not go into 4.9.3.0.

On Tue, Mar 13, 2018 at 11:50 AM, Jochim, Ingo 
wrote:

> Hi Rafael,
>
> ok, thanks.
>
> Regards,
> Ingo
>
> -Original Message-
> From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> Sent: Dienstag, 13. März 2018 11:48
> To: users 
> Subject: Re: new version 4.9.3 coming
>
> Not that I know of. I believe the 4.9.* are end of life this July.
> Therefore, I would not expect anything other than very critical things to
> be into them.
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/LTS
>
> On Tue, Mar 13, 2018 at 7:36 AM, Jochim, Ingo 
> wrote:
>
> > Hello all,
> >
> > are there any plans to release a version 4.9.3 which includes the latest
> > bug fixes available in 4.11?
> >
> > Many thanks already.
> > Regards,
> > Ingo
> >
>
>
>
> --
> Rafael Weingärtner
>



-- 
Rafael Weingärtner

RE: new version 4.9.3 coming

[Jochim, Ingo]

Hi Rafael,

ok, thanks.

Regards,
Ingo

-Original Message-
From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] 
Sent: Dienstag, 13. März 2018 11:48
To: users 
Subject: Re: new version 4.9.3 coming

Not that I know of. I believe the 4.9.* are end of life this July.
Therefore, I would not expect anything other than very critical things to
be into them.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/LTS

On Tue, Mar 13, 2018 at 7:36 AM, Jochim, Ingo 
wrote:

> Hello all,
>
> are there any plans to release a version 4.9.3 which includes the latest
> bug fixes available in 4.11?
>
> Many thanks already.
> Regards,
> Ingo
>



-- 
Rafael Weingärtner

Re: Migrate system VMs volumes to new storage

[Ugo Vasi]

Hi, I put the server in maintenance mode and then I could delete it 
directly from the web interface.


Bye

Il 13/03/2018 15:25, Ugo Vasi ha scritto:

Hi Andrija,
I followed your procedures and confirm that it works fine!
Now I have to dismiss the storage server and I'm wondering if you just 
put it in maintenance mode forever or if there's another way to delete 
it.


Thanks

Il 15/02/2018 18:03, Andrija Panic ha scritto:

Hi Ugo,

though not preferred at all, feel free to start DB experiments :) 
since this is very often required (well, in some cases) - this is not 
the rule,  buy often you have to make your hand dirty.


For you specific issue, here is output of my DB below

The important thing is obviously the TAGS column, just enter value 
which you also added to Storage Tags and that is all (wait for 5min, 
go get some coffee, not sure why, needs some time to be applied)


Also note the special column "unique_name" - if you have created 
multiple offerings for i.e. SSVM (more CPU or more RAM, whatever...) 
then ACS needs to know WHICH of the offering for SSVM to actually use 
when creating new SSVM.
SO make sure that, in this specific example, you move the value " 
Cloud.com-SecondaryStorage" to correct ROW - only single ROW can 
exist with this specific value.



FOr LOCAL systemVMs, there is similar record, but I guess you are on 
shared storage anyway



mysql> select 
id,domain_id,name,display_text,tags,unique_name,system_use from 
disk_offering where name like "%system_offering%%" and name not like 
"%local%";
+-+---++++++ 

| id  | domain_id | name                      | display_text         
         | tags       | unique_name   | system_use |
+-+---++++++ 

|  19 |      NULL | System Offering For Software Router 1Gbps         
      | System Offering For Software Router 1Gbps           | 
SolidFire1 | Cloud.Com-SoftwareRouter   |          1 |
|  22 |      NULL | System Offering For Internal LB VM            | 
NULL                      | SolidFire1 | Cloud.Com-InternalLBVm    |  
        1 |
|  25 |      NULL | System Offering For Console Proxy             | 
NULL                        | SolidFire1 | Cloud.com-ConsoleProxy    
 |          1 |
|  28 |      NULL | System Offering For Secondary Storage VM         
       | NULL                      | SolidFire1 | 
Cloud.com-SecondaryStorage |          1 |
|  31 |      NULL | System Offering For Elastic LB VM             | 
NULL                        | SolidFire1 | Cloud.Com-ElasticLBVm     
|          1 |
| 166 |      NULL | System Offering For Software Router 500Mbps       
        | System Offering For Software Router 500Mbps           | 
SolidFire1 | NULL      |          1 |
| 169 |      NULL | System Offering For Software Router 200Mbps       
        | System Offering For Software Router 200Mbps           | 
SolidFire1 | NULL      |          1 |
| 243 |      NULL | System Offering For Software Router 100Mbps       
        | System Offering For Software Router 100Mbps           | 
SolidFire1 | NULL      |          1 |
+-+---++++++ 



Let me know if you need more help

Cheers


On 14 February 2018 at 12:10, Ugo Vasi > wrote:


    Hi Andrija,
    the web interface does not allow to add / change tags for system
    VM system offerings.

    In a test system with the same configuration (as much as possible)
    I added a new entry for the proxy console with the tags for the
    new strorage and, after destroying the console VM, it was
    recreated in the new storage as I expected.

    I tried to take the same steps in the production environment but
    the system persists in recreating the VM in the old storage. I
    also tried to change the order of appearance of the system
    offerings list but the result does not change.

    I thought about intervening in the database but I do not like it
    as a solution.

    Any idea?




    Il 07/02/2018 21:30, Andrija Panic ha scritto:

    Hi Ugo,

    I dont know if you have multiple SSVM (and other system)
    offerings, but we
    have 2 default ones for every role (SSVM on local and on
    shared storage,
    then CPVM on local and shared, then VR for both also...) -
    and we just
    edit the "shared one" i.e. System Offering For Secondary
    Storage VM and set
    tag here, which was previously also set on new storage.

    They way tagging works 

Re: Migrate system VMs volumes to new storage

[Ugo Vasi]

Hi Andrija,
I followed your procedures and confirm that it works fine!
Now I have to dismiss the storage server and I'm wondering if you just 
put it in maintenance mode forever or if there's another way to delete it.


Thanks

Il 15/02/2018 18:03, Andrija Panic ha scritto:

Hi Ugo,

though not preferred at all, feel free to start DB experiments :) 
since this is very often required (well, in some cases) - this is not 
the rule,  buy often you have to make your hand dirty.


For you specific issue, here is output of my DB below

The important thing is obviously the TAGS column, just enter value 
which you also added to Storage Tags and that is all (wait for 5min, 
go get some coffee, not sure why, needs some time to be applied)


Also note the special column "unique_name" - if you have created 
multiple offerings for i.e. SSVM (more CPU or more RAM, whatever...) 
then ACS needs to know WHICH of the offering for SSVM to actually use 
when creating new SSVM.
SO make sure that, in this specific example, you move the value " 
Cloud.com-SecondaryStorage" to correct ROW - only single ROW can exist 
with this specific value.



FOr LOCAL systemVMs, there is similar record, but I guess you are on 
shared storage anyway



mysql> select 
id,domain_id,name,display_text,tags,unique_name,system_use from 
disk_offering where name like "%system_offering%%" and name not like 
"%local%";

+-+---++++++
| id  | domain_id | name                      | display_text           
           | tags       | unique_name   | system_use |

+-+---++++++
|  19 |      NULL | System Offering For Software Router 1Gbps          
        | System Offering For Software Router 1Gbps                  | 
SolidFire1 | Cloud.Com-SoftwareRouter   |          1 |
|  22 |      NULL | System Offering For Internal LB VM                 
     | NULL                      | SolidFire1 | Cloud.Com-InternalLBVm 
   |          1 |
|  25 |      NULL | System Offering For Console Proxy                 
      | NULL                        | SolidFire1 | 
Cloud.com-ConsoleProxy     |          1 |
|  28 |      NULL | System Offering For Secondary Storage VM          
         | NULL                      | SolidFire1 | 
Cloud.com-SecondaryStorage |          1 |
|  31 |      NULL | System Offering For Elastic LB VM                 
      | NULL                        | SolidFire1 | 
Cloud.Com-ElasticLBVm     |          1 |
| 166 |      NULL | System Offering For Software Router 500Mbps        
        | System Offering For Software Router 500Mbps                | 
SolidFire1 | NULL      |          1 |
| 169 |      NULL | System Offering For Software Router 200Mbps        
        | System Offering For Software Router 200Mbps                | 
SolidFire1 | NULL      |          1 |
| 243 |      NULL | System Offering For Software Router 100Mbps        
        | System Offering For Software Router 100Mbps                | 
SolidFire1 | NULL      |          1 |

+-+---++++++

Let me know if you need more help

Cheers


On 14 February 2018 at 12:10, Ugo Vasi > wrote:


Hi Andrija,
the web interface does not allow to add / change tags for system
VM system offerings.

In a test system with the same configuration (as much as possible)
I added a new entry for the proxy console with the tags for the
new strorage and, after destroying the console VM, it was
recreated in the new storage as I expected.

I tried to take the same steps in the production environment but
the system persists in recreating the VM in the old storage. I
also tried to change the order of appearance of the system
offerings list but the result does not change.

I thought about intervening in the database but I do not like it
as a solution.

Any idea?




Il 07/02/2018 21:30, Andrija Panic ha scritto:

Hi Ugo,

I dont know if you have multiple SSVM (and other system)
offerings, but we
have 2 default ones for every role (SSVM on local and on
shared storage,
then CPVM on local and shared, then VR for both also...) - 
and we just
edit the "shared one" i.e. System Offering For Secondary
Storage VM and set
tag here, which was previously also set on new storage.

They way tagging works for volumes (descriptive) as following,
assuming you
have setup "TAG1" as 

Re: CCS (Cloudstack Container Service) in CloudStack 4.11

[Daan Hoogland]

Parth, The plugin 1.0 won't work with any version beyond 4.9 (4.7 and 4.7
aren't tested). There is a branch, 'on-top-of-pr-2071-for-4.10
', that
we are now porting to 4.11. If you have capability to build it you can try
that.

On Tue, Mar 13, 2018 at 1:03 PM, Parth Patel 
wrote:

> Hi,
>
> Maybe somebody working in ShapeBlue can help me. I am trying to integrate
> CCS service (https://github.com/shapeblue/ccs) with ACS 4.11 using
> http://downloads.shapeblue.com/ccs/1.0/Installation_and_
> Administration_Guide.pdf
> and
> I quickly came to know about huge architecture changes that ACS has gone
> through in the latest release. I somehow managed to bring the UI by editing
> and placing the ccs folder in the new "webapp" folder instead of "webapps"
> folder. But I am confused as to where to place the shapeblue-ccs and
> flyaway core jar files. I tried placing them in
> /usr/share/cloudstack-management/lib folder but it gave the following
> error
> in management-server.log:
>
>
> 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (main:null) (logid:) Loading module context [ccs] from URL
> [jar:file:/usr/share/cloudstack-management/webapp/
> WEB-INF/lib/cloud-plugin-shapeblue-ccs-1.0.1.jar!/META-
> INF/cloudstack/ccs/spring-ccs-context.xml]
> 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (main:null) (logid:) Loading module context [ccs] from URL
> [jar:file:/usr/share/cloudstack-management/lib/
> cloudstack-4.11.0.0.jar!/META-INF/cloudstack/compute/spring-
> core-lifecycle-compute-context-inheritable.xml]
> 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (main:null) (logid:) Loading module context [ccs] from URL
> [jar:file:/usr/share/cloudstack-management/lib/
> cloudstack-4.11.0.0.jar!/META-INF/cloudstack/core/spring-
> core-lifecycle-core-context-inheritable.xml]
> 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (main:null) (logid:) Loading module context [ccs] from URL
> [jar:file:/usr/share/cloudstack-management/lib/
> cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> framework-config-system-context-inheritable.xml]
> 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (main:null) (logid:) Loading module context [ccs] from URL
> [jar:file:/usr/share/cloudstack-management/lib/
> cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> core-system-context-inheritable.xml]
> 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (main:null) (logid:) Loading module context [ccs] from URL
> [jar:file:/usr/share/cloudstack-management/lib/
> cloudstack-4.11.0.0.jar!/META-INF/cloudstack/system/spring-
> contrail-system-context-inheritable.xml]
> 2018-03-13 16:59:27,953 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
> (main:null) (logid:) Loading module context [ccs] from URL
> [jar:file:/usr/share/cloudstack-management/lib/
> cloudstack-4.11.0.0.jar!/META-INF/cloudstack/bootstrap/
> spring-bootstrap-context-inheritable.xml]
> 2018-03-13 16:59:28,098 WARN  [o.a.c.s.m.c.ResourceApplicationContext]
> (main:null) (logid:) Exception encountered during context initialization -
> cancelling refresh attempt:
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> failed; nested exception is java.lang.IllegalStateException: Failed to
> introspect Class [com.cloud.containercluster.ContainerClusterManagerImpl]
> from ClassLoader [WebAppClassLoader=500772834@1dd92fe2]
> 2018-03-13 16:59:28,100 WARN  [o.e.j.w.WebAppContext] (main:null) (logid:)
> Failed startup of context o.e.j.w.WebAppContext@365c30cc
> {/client,file:///usr/share/cloudstack-management/webapp/,
> UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'ContainerClusterManagerImpl': Lookup method resolution
> failed; nested exception is java.lang.IllegalStateException: Failed to
> introspect Class [com.cloud.containercluster.ContainerClusterManagerImpl]
> from ClassLoader [WebAppClassLoader=500772834@1dd92fe2]
> at
> org.springframework.beans.factory.annotation.
> AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(
> AutowiredAnnotationBeanPostProcessor.java:262)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.determineConstructorsFromBeanPostProcessors(
> AbstractAutowireCapableBeanFactory.java:1198)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1123)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFac
> tory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
> at
> 

Re: KVM HostHA

[Jon Marshall]

Update on below.


I pulled the NICs for both management and storage from cnode 1.


1) The UI immediately showed the power state as Unknown but the state was Up.

2) The HA state on cnode1 showed as suspect. The HA state on cnode2 showed as 
available.

3) After about 4 mins the state on cnode1 went from Up to Alert

4) The HA state on cnode1 showed as Fencing and the HA state on cnode2 showed 
as Ineligible.


The HA enabled VMs on cnode1 never switched over to the working node cnode2.


Any ideas ?



From: Jon Marshall 
Sent: 13 March 2018 10:50
To: users@cloudstack.apache.org
Subject: Re: KVM HostHAtot stop the server responding to an ipmitool request on 
the manager eg -


"ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis status"


from the management server got an answer saying the chassis power was on so CS 
never registered the compute node as down.


I am obviously doing something wrong but cannot work it out.


The management server has one NIC - 172.16.7.4


Each compute node has 3 NICs -


   cnode1cnode2


mangement NIC172.16.7.5   172.16.7.6

vm NIC  172.16.6.130 172.16.6.131

storage - 172.16.250.4   172.16.250.5


Dell LOM (for Idrac)   172.16.7.29172.16.7.30


the dell LOM IPs are the ones used to configure OOBM  in the UI



If I pull the storage NIC presumably nothing will happen as the ipmitool check 
is running across the management NIC so I need to pull both ?

My understanding of host HA was the management server monitored the compute 
nodes using ipmitool and if it did not get a response because the host was down 
it would fence off that host and move the VMs to an active compute node.

This is obviously too simplistic so could someone explain how it is meant to 
work and what it is protecting against ?


From: Paul Angus 
Sent: 13 March 2018 07:01
To: users@cloudstack.apache.org
Subject: RE: KVM HostHA

Hi all,

One small note, unplugging the management NIC will only cause an HA event if 
the storage is running over that NIC also.

Is the storage is over a separate NIC then, the guest VMs will continue to run 
when the mgmt. NIC is unplugged, Host HA will detect the disk activity and 
conclude that there is nothing it can do, as the VMs are still running other 
than mark the hosts as degraded.


Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

Shapeblue - The CloudStack Company
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a 
framework developed by ShapeBlue to deliver the rapid deployment of a 
standardised ...



[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

Shapeblue - The CloudStack Company
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

Shapeblue - The CloudStack Company
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a 
framework developed by ShapeBlue to deliver the rapid deployment of a 
standardised ...



www.shapeblue.com
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

Shapeblue - The CloudStack Company
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a 
framework developed by ShapeBlue to deliver the rapid deployment of a 
standardised ...



Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a 
framework developed by ShapeBlue to deliver the rapid deployment of a 
standardised ...



53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-Original Message-
From: Parth Patel 
Sent: 12 March 2018 17:35
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA

>
> Hi Jon,
>
> As I said, in my case, making the host HA didn't work but by just
> having a HA VM running on host and executing - (WARNING) "echo c >
> /proc/sysrq-trigger" to simulate a kernel crash on host, the
> management server registered it as down and started the VM on another
> host. I know I've suggested this before but I insist you give this a
> try. Also, you don't need to completely power off the machine manually
> but just plugging out the network cable works fine. The cloudstack
> agent after losing connection to management server auto reboots
> because of KVM heartbeat check shell script mentioned by Rohit Yadav
> to one of my earlier queries in other thread.
>
> On Mon 12 Mar, 2018, 21:23 Jon Marshall, 

Re: KVM HostHA

[Jon Marshall]

I tried "echo c > /proc/sysrq-trigger" which stopped me getting into the server 
but it did not stop the server responding to an ipmitool request on the manager 
eg -


"ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis status"


from the management server got an answer saying the chassis power was on so CS 
never registered the compute node as down.


I am obviously doing something wrong but cannot work it out.


The management server has one NIC - 172.16.7.4


Each compute node has 3 NICs -


   cnode1cnode2


mangement NIC172.16.7.5   172.16.7.6

vm NIC  172.16.6.130 172.16.6.131

storage - 172.16.250.4   172.16.250.5


Dell LOM (for Idrac)   172.16.7.29172.16.7.30


the dell LOM IPs are the ones used to configure OOBM  in the UI



If I pull the storage NIC presumably nothing will happen as the ipmitool check 
is running across the management NIC so I need to pull both ?

My understanding of host HA was the management server monitored the compute 
nodes using ipmitool and if it did not get a response because the host was down 
it would fence off that host and move the VMs to an active compute node.

This is obviously too simplistic so could someone explain how it is meant to 
work and what it is protecting against ?


From: Paul Angus 
Sent: 13 March 2018 07:01
To: users@cloudstack.apache.org
Subject: RE: KVM HostHA

Hi all,

One small note, unplugging the management NIC will only cause an HA event if 
the storage is running over that NIC also.

Is the storage is over a separate NIC then, the guest VMs will continue to run 
when the mgmt. NIC is unplugged, Host HA will detect the disk activity and 
conclude that there is nothing it can do, as the VMs are still running other 
than mark the hosts as degraded.


Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

Shapeblue - The CloudStack Company
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a 
framework developed by ShapeBlue to deliver the rapid deployment of a 
standardised ...



53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-Original Message-
From: Parth Patel 
Sent: 12 March 2018 17:35
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA

>
> Hi Jon,
>
> As I said, in my case, making the host HA didn't work but by just
> having a HA VM running on host and executing - (WARNING) "echo c >
> /proc/sysrq-trigger" to simulate a kernel crash on host, the
> management server registered it as down and started the VM on another
> host. I know I've suggested this before but I insist you give this a
> try. Also, you don't need to completely power off the machine manually
> but just plugging out the network cable works fine. The cloudstack
> agent after losing connection to management server auto reboots
> because of KVM heartbeat check shell script mentioned by Rohit Yadav
> to one of my earlier queries in other thread.
>
> On Mon 12 Mar, 2018, 21:23 Jon Marshall,  wrote:
> Hi Paul
>
>
> Thanks for the response.
>
>
> I think I am not understanding how it was meant to work then. My
> understanding was that the manager used ipmitool to just keep querying
> the compute nodes as to their status so I assumed it didn't matter how
> you shut the node down, once it was down the manager would get no
> response and mark it as down (which it does).
>
>
> I am in testing mode so I think I will just go and pull the power and
> see what happens :)
>
>
> Thanks
>
>
> Jon
>
>
> 
> From: Paul Angus 
> Sent: 12 March 2018 15:31
> To: users@cloudstack.apache.org
> Subject: RE: KVM HostHA
> Hi Jon,
>
> I think that what you guys are finding, is that a controlled host
> shutdown, which will cause the agent to shutdown cleanly; Is not
> considered an HA event. I wouldn't expect CloudStack to take any
> action if you shut down a host, only if the host (agent) stops responding.
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

Shapeblue - The CloudStack Company
www.shapeblue.com
Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge is a 
framework developed by ShapeBlue to deliver the rapid deployment of a 
standardised ...



> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png

[http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]

]<
> http://www.shapeblue.com/>

Re: new version 4.9.3 coming

[Rafael Weingärtner]

Not that I know of. I believe the 4.9.* are end of life this July.
Therefore, I would not expect anything other than very critical things to
be into them.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/LTS

On Tue, Mar 13, 2018 at 7:36 AM, Jochim, Ingo 
wrote:

> Hello all,
>
> are there any plans to release a version 4.9.3 which includes the latest
> bug fixes available in 4.11?
>
> Many thanks already.
> Regards,
> Ingo
>



-- 
Rafael Weingärtner

new version 4.9.3 coming

[Jochim, Ingo]

Hello all,

are there any plans to release a version 4.9.3 which includes the latest bug 
fixes available in 4.11?

Many thanks already.
Regards,
Ingo

RE: KVM HostHA

[Paul Angus]

Hi all,

One small note, unplugging the management NIC will only cause an HA event if 
the storage is running over that NIC also.

Is the storage is over a separate NIC then, the guest VMs will continue to run 
when the mgmt. NIC is unplugged, Host HA will detect the disk activity and 
conclude that there is nothing it can do, as the VMs are still running other 
than mark the hosts as degraded.


Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Parth Patel  
Sent: 12 March 2018 17:35
To: users@cloudstack.apache.org
Subject: Re: KVM HostHA

>
> Hi Jon,
>
> As I said, in my case, making the host HA didn't work but by just 
> having a HA VM running on host and executing - (WARNING) "echo c > 
> /proc/sysrq-trigger" to simulate a kernel crash on host, the 
> management server registered it as down and started the VM on another 
> host. I know I've suggested this before but I insist you give this a 
> try. Also, you don't need to completely power off the machine manually 
> but just plugging out the network cable works fine. The cloudstack 
> agent after losing connection to management server auto reboots 
> because of KVM heartbeat check shell script mentioned by Rohit Yadav 
> to one of my earlier queries in other thread.
>
> On Mon 12 Mar, 2018, 21:23 Jon Marshall,  wrote:
> Hi Paul
>
>
> Thanks for the response.
>
>
> I think I am not understanding how it was meant to work then. My 
> understanding was that the manager used ipmitool to just keep querying 
> the compute nodes as to their status so I assumed it didn't matter how 
> you shut the node down, once it was down the manager would get no 
> response and mark it as down (which it does).
>
>
> I am in testing mode so I think I will just go and pull the power and 
> see what happens :)
>
>
> Thanks
>
>
> Jon
>
>
> 
> From: Paul Angus 
> Sent: 12 March 2018 15:31
> To: users@cloudstack.apache.org
> Subject: RE: KVM HostHA
> Hi Jon,
>
> I think that what you guys are finding, is that a controlled host 
> shutdown, which will cause the agent to shutdown cleanly; Is not 
> considered an HA event. I wouldn't expect CloudStack to take any 
> action if you shut down a host, only if the host (agent) stops responding.
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]<
> http://www.shapeblue.com/>
>
> Shapeblue - The CloudStack Company
> www.shapeblue.com
> Rapid deployment framework for Apache CloudStack IaaS Clouds. CSForge 
> is a framework developed by ShapeBlue to deliver the rapid deployment 
> of a standardised ...
>
>
>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
>
>
>
>
> -Original Message-
> From: Jon Marshall 
> Sent: 12 March 2018 15:15
> To: users@cloudstack.apache.org
> Subject: Re: KVM HostHA
>
> I have the same issue here and am not entirely sure what the behaviour
> should be.
>
>
> I have one manager node and 2 compute nodes running 4.11 with ipmi working
> correctly.
>
>
> From the UI under HA -
>
>
> HA Enabled Yes
> HA State Available
> HA Provider kvmhaprovider
>
>
> although interestingly from the "Details" tab it shows -
>
>
> HA enabled No
>
>
> which I assume is a cosmetic issue ?
>
>
> On each compute node I have one HA enabled VM and one non HA enabled VM.
>
>
> I power off a compute node and the UI updates the host status and the VMs
> on that node stop responding but they never fail over to the other node.
>
>
> Couple of things I noticed -
>
>
> 1) as soon as i power off the compute node the HA state on the other node
> shows "Ineligible"
>
>
> 2) In the UI the instances all still show as green even though two of them
> are not available
>
>
> Any help much appreciated
>
>
>
>
> 
> From: victor 
> Sent: 07 March 2018 17:01
> To: users@cloudstack.apache.org
> Subject: KVM HostHA
>
> Hello Guys,
>
> I have installed cloudstack 4.11. I have enabled HA for each hosts I have
> added. I have also added ipmi successfully (using ipmi driver).
> The hosts are showing like the following.
>
> ===
>
> HA Enabled Yes
> HA State Available
> HA Provider kvmhaprovider
>
> ==
>
> Also the host is showing the following correctly
>
> Resource state --> Enabled
> State --> UP
> Power state --> On
>
> So I have shutdown one of the hosts to see how the KVM hosts Ha is
> working. I have waited for half an hour. But nothing has happened. What
> will happen to the VM's in that host, if the host failed to back up.
> There isn't much from logs.
>
> Regards
> Victor
>