Hi Piotr, Did you roll back and start the upgrade again, or just run the sql statements?
paul.an...@shapeblue.comĀ www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -----Original Message----- From: Piotr Pisz <pp...@pulab.pl> Sent: 07 March 2018 07:42 To: Paul Angus <paul.an...@shapeblue.com>; users@cloudstack.apache.org Subject: RE: CS fail after upgrade to 4.11 Hi Paul, You were right and the script helped, the upgrade was a success. Unfortunately, we have a strange symptom, in UI all hosts are not visible (see picture). There are no major errors in the log. On DB all hosts have status Up but all operation like VM power on ending with info "No host are available". What could be the reason? Regards, Piotr 2018-03-07 08:53:07,405 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) HostStatsCollector is running... 2018-03-07 08:53:07,451 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 3-8238491093344190514: Received: { Ans: , MgmtId: 119496918425704, via: 3(dc13-r4-wpr1-h1.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,497 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 5-6845752908579864613: Received: { Ans: , MgmtId: 119496918425704, via: 5(dc13-r1-wpr1-h2.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,541 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 25-3432587340986449985: Received: { Ans: , MgmtId: 119496918425704, via: 25(dc13-r19-wpr2-n3.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,585 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 29-4613937818241073220: Received: { Ans: , MgmtId: 119496918425704, via: 29(dc13-r17-wpr2-n2.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,630 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 37-8468737624293507140: Received: { Ans: , MgmtId: 119496918425704, via: 37(dc13-r16-wpr2-n1.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,673 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 48-1969480412044460068: Received: { Ans: , MgmtId: 119496918425704, via: 48(dc13-r7-wpr2-v12-n1.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,718 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 52-3613294276034691098: Received: { Ans: , MgmtId: 119496918425704, via: 52(dc13-r9-wpr2-v12-n2.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,761 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-383073e7) (logid:a0a38b47) Seq 64-4504162577323917343: Received: { Ans: , MgmtId: 119496918425704, via: 64(dc13-r11-wpr2-v12-n4.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2018-03-07 08:53:07,879 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-75d73769) (logid:4333f9a0) Begin cleanup expired async-jobs 2018-03-07 08:53:07,881 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-75d73769) (logid:4333f9a0) End cleanup expired async-jobs 2018-03-07 08:53:09,743 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e2109ed5) (logid:a7e15ece) Seq 3-8238491093344190515: Received: { Ans: , MgmtId: 119496918425704, via: 3(dc13-r4-wpr1-h1.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2018-03-07 08:53:09,841 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e2109ed5) (logid:a7e15ece) Seq 25-3432587340986449986: Received: { Ans: , MgmtId: 119496918425704, via: 25(dc13-r19-wpr2-n3.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2018-03-07 08:53:09,909 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e2109ed5) (logid:a7e15ece) Seq 29-4613937818241073221: Received: { Ans: , MgmtId: 119496918425704, via: 29(dc13-r17-wpr2-n2.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2018-03-07 08:53:09,991 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e2109ed5) (logid:a7e15ece) Seq 37-8468737624293507141: Received: { Ans: , MgmtId: 119496918425704, via: 37(dc13-r16-wpr2-n1.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2018-03-07 08:53:10,042 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e2109ed5) (logid:a7e15ece) Seq 48-1969480412044460069: Received: { Ans: , MgmtId: 119496918425704, via: 48(dc13-r7-wpr2-v12-n1.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2018-03-07 08:53:10,110 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-e2109ed5) (logid:a7e15ece) Seq 64-4504162577323917344: Received: { Ans: , MgmtId: 119496918425704, via: 64(dc13-r11-wpr2-v12-n4.nubilum.opegieka.pl), Ver: v1, Flags: 10, { GetVmStatsAnswer } } 2018-03-07 08:53:15,190 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Ping from 5(dc13-r1-wpr1-h2.nubilum.opegieka.pl) 2018-03-07 08:53:15,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Process host VM state report from ping process. host: 5 2018-03-07 08:53:15,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Process VM state report. host: 5, number of records in report: 0 2018-03-07 08:53:15,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Done with process of VM state report. host: 5 -----Original Message----- From: Paul Angus [mailto:paul.an...@shapeblue.com] Sent: Tuesday, March 6, 2018 8:39 PM To: users@cloudstack.apache.org; pp...@pulab.pl Subject: RE: CS fail after upgrade to 4.11 Hi Piotr, Did you see this section in the upgrade notes for 4.10 to 4.11 ? Apache CloudStack 4.10.0.0 users who are upgrading to 4.11.0.0 should read the following discussion and workaround for a db-upgrade issue: http://markmail.org/message/f42kqr3mx4r4hgih paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -----Original Message----- From: Piotr Pisz <pp...@pulab.pl> Sent: 06 March 2018 18:20 To: users@cloudstack.apache.org Subject: CS fail after upgrade to 4.11 All, After upgrade from 4.10 to 4.11 We have these errors in managemen log. Is this a problem with our db or an incorrect upgrade? We repeated the upgrade but with the same result. What can we do? Please help J Regards, Piotr 2018-03-06 19:52:43,719 INFO [c.c.s.ConfigurationServerImpl] (main:null) (logid:) Injected public and private keys into systemvm iso with result : null 2018-03-06 19:52:43,738 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:) Start configuring cluster manager : ClusterManagerImpl 2018-03-06 19:52:43,739 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:) Cluster node IP : 172.16.109.46 2018-03-06 19:52:43,744 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:) Cluster manager is configured. 2018-03-06 19:52:43,744 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null) (logid:) Start AsyncJobManager API executor thread pool in size 125 2018-03-06 19:52:43,744 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null) (logid:) Start AsyncJobManager Work executor thread pool in size 166 2018-03-06 19:52:43,745 INFO [c.c.h.v.m.VmwareManagerImpl] (main:null) (logid:) Configure VmwareManagerImpl, manager name: VmwareManagerImpl 2018-03-06 19:52:43,745 INFO [c.c.h.v.m.VmwareManagerImpl] (main:null) (logid:) VmwareManagerImpl config - instance.name: VM 2018-03-06 19:52:43,747 INFO [c.c.h.v.m.VmwareManagerImpl] (main:null) (logid:) VmwareManagerImpl config - _mountParent: /var/cloudstack/mnt/VM 2018-03-06 19:52:43,752 INFO [c.c.h.v.m.VmwareManagerImpl] (main:null) (logid:) VmwareManagerImpl config - vmware.vcenter.session.timeout: 1200000 2018-03-06 19:52:43,753 INFO [c.c.h.v.m.VmwareManagerImpl] (main:null) (logid:) Additional VNC port allocation range is settled at 50000 to 51000 2018-03-06 19:52:43,753 INFO [c.c.h.v.m.VmwareStorageManagerImpl] (main:null) (logid:) Configure VmwareStorageManagerImpl 2018-03-06 19:52:43,753 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener VmwareManagerImpl with id 1 2018-03-06 19:52:43,753 INFO [c.c.h.v.m.VmwareManagerImpl] (main:null) (logid:) VmwareManagerImpl has been successfully configured 2018-03-06 19:52:43,754 INFO [c.c.a.m.ClusteredAgentManagerImpl] (main:null) (logid:) Configuring ClusterAgentManagerImpl. management server node id(msid): 119496918425704 2018-03-06 19:52:43,756 INFO [c.c.c.ClusterManagerImpl] (main:null) (logid:) register cluster listener class com.cloud.agent.manager.ClusteredAgentManagerImpl 2018-03-06 19:52:43,757 INFO [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Ping Timeout is 2.5 2018-03-06 19:52:43,758 INFO [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Configuring AgentManagerImpl. management server node id(msid): 119496918425704 2018-03-06 19:52:43,789 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener BehindOnPingListener with id 2 2018-03-06 19:52:43,790 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener SetHostParamsListener with id 3 2018-03-06 19:52:43,792 INFO [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Listening on 8250 with 5 workers 2018-03-06 19:52:43,793 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Created DirectAgentAttache pool with size: 500 2018-03-06 19:52:43,827 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (main:null) (logid:) Router configurations: ramsize=256 2018-03-06 19:52:43,828 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener SshKeysDistriMonitor with id 4 2018-03-06 19:52:43,831 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener VpcVirtualNetworkApplianceManagerImpl with id 5 2018-03-06 19:52:43,831 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (main:null) (logid:) DomainRouterManager is configured. 2018-03-06 19:52:43,832 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener SecurityGroupListener with id 6 2018-03-06 19:52:43,832 INFO [c.c.n.s.SecurityGroupManagerImpl] (main:null) (logid:) SecurityGroupManager: num worker threads=50, time between cleanups=120 global lock timeout=300 2018-03-06 19:52:43,833 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:null) (logid:) Background Poll Manager received task: CABackgroundTask 2018-03-06 19:52:43,834 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener DeploymentPlanningManagerImpl with id 7 2018-03-06 19:52:43,838 INFO [c.c.n.IpAddressManagerImpl] (main:null) (logid:) IPAddress Manager is configured. 2018-03-06 19:52:43,839 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener StorageCapacityListener with id 8 2018-03-06 19:52:43,839 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener ComputeCapacityListener with id 9 2018-03-06 19:52:43,840 INFO [o.a.c.s.SecondaryStorageManagerImpl] (main:null) (logid:) Start configuring secondary storage vm manager : PremiumSecondaryStorageManagerImpl 2018-03-06 19:52:43,842 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener SecondaryStorageListener with id 10 2018-03-06 19:52:43,846 INFO [o.a.c.s.SecondaryStorageManagerImpl] (main:null) (logid:) Secondary storage vm Manager is configured. 2018-03-06 19:52:43,848 INFO [o.a.c.f.m.MessageDispatcher] (main:null) (logid:) Build message handler cache for com.cloud.api.ApiServer 2018-03-06 19:52:43,848 INFO [o.a.c.f.m.MessageDispatcher] (main:null) (logid:) Add message handler com.cloud.api.ApiServer.handleAsyncJobPublishEvent to cache 2018-03-06 19:52:43,848 INFO [o.a.c.f.m.MessageDispatcher] (main:null) (logid:) Done building message handler cache for com.cloud.api.ApiServer 2018-03-06 19:52:43,850 INFO [o.a.c.n.l.InternalLoadBalancerVMManagerImpl] (main:null) (logid:) InternalLoadBalancerVMManagerImpl has been configured 2018-03-06 19:52:43,865 INFO [c.c.n.NetworkServiceImpl] (main:null) (logid:) Network Service is configured. 2018-03-06 19:52:43,865 INFO [c.c.c.ConsoleProxyManagerImpl] (main:null) (logid:) Start configuring console proxy manager : ConsoleProxyManagerImpl 2018-03-06 19:52:43,865 WARN [c.c.c.ConsoleProxyManagerImpl] (main:null) (logid:) Empty console proxy domain, explicitly disabling SSL 2018-03-06 19:52:43,866 INFO [c.c.c.ConsoleProxyManagerImpl] (main:null) (logid:) Console proxy max session soft limit : 50 2018-03-06 19:52:43,866 INFO [c.c.c.ConsoleProxyManagerImpl] (main:null) (logid:) Console proxy standby capacity : 10 2018-03-06 19:52:43,867 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener ConsoleProxyListener with id 11 2018-03-06 19:52:43,872 INFO [c.c.c.ConsoleProxyManagerImpl] (main:null) (logid:) Console Proxy Manager is configured. 2018-03-06 19:52:43,876 INFO [c.c.n.l.LBHealthCheckManagerImpl] (main:null) (logid:) Configuring LBHealthCheck Manager null 2018-03-06 19:52:43,891 DEBUG [o.a.c.p.BackgroundPollManagerImpl] (main:null) (logid:) Background Poll Manager received task: OutOfBandManagementPowerStatePollTask 2018-03-06 19:52:43,891 INFO [o.a.c.o.OutOfBandManagementServiceImpl] (main:null) (logid:) Starting out-of-band management background sync executor with thread pool-size=50 2018-03-06 19:52:43,894 INFO [c.c.s.s.SnapshotSchedulerImpl] (main:null) (logid:) Snapshot Scheduler is configured. 2018-03-06 19:52:43,899 INFO [c.c.s.s.SnapshotManagerImpl] (main:null) (logid:) Snapshot Manager is configured. 2018-03-06 19:52:43,900 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (main:null) (logid:) Router configurations: ramsize=256 2018-03-06 19:52:43,900 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener SshKeysDistriMonitor with id 12 2018-03-06 19:52:43,902 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener VirtualNetworkApplianceManagerImpl with id 13 2018-03-06 19:52:43,902 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (main:null) (logid:) DomainRouterManager is configured. 2018-03-06 19:52:43,905 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 2 Name = tx-3; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-Transaction Legacy.close:656-Transaction.execute:43-Transaction.execute:47-NetworkOrches trator.configure:450-CloudStackExtendedLifeCycle$3.with:115-CloudStackExtend edLifeCycle.with:154-CloudStackExtendedLifeCycle.configure:111-CloudStackExt endedLifeCycle.start:56-DefaultLifecycleProcessor.doStart:183-DefaultLifecyc leProcessor.access$200:52 2018-03-06 19:52:43,905 WARN [o.a.c.s.m.c.ResourceApplicationContext] (main:null) (logid:) Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@1cf4d454 <mailto:com.mysql.jdbc.JDBC4PreparedStatement@1cf4d454> : SELECT network_offerings.id, network_offerings.name, network_offerings.unique_name, network_offerings.display_text, network_offerings.nw_rate, network_offerings.mc_rate, network_offerings.traffic_type, network_offerings.specify_vlan, network_offerings.system_only, network_offerings.service_offering_id, network_offerings.tags, network_offerings.default, network_offerings.availability, network_offerings.state, network_offerings.removed, network_offerings.created, network_offerings.guest_type, network_offerings.dedicated_lb_service, network_offerings.shared_source_nat_service, network_offerings.specify_ip_ranges, network_offerings.sort_key, network_offerings.uuid, network_offerings.redundant_router_service, network_offerings.conserve_mode, network_offerings.elastic_ip_service, network_offerings.eip_associate_public_ip, network_offerings.elastic_lb_service, network_offerings.inline, network_offerings.is_persistent, network_offerings.for_vpc, network_offerings.egress_default_policy, network_offerings.concurrent_connections, network_offerings.keep_alive_enabled, network_offerings.supports_streched_l2, network_offerings.supports_public_access, network_offerings.internal_lb, network_offerings.public_lb, network_offerings.service_package_id FROM network_offerings WHERE network_offerings.unique_name = x'517569636B436C6F75644E6F5365727669636573' AND network_offerings.removed IS NULL ORDER BY RAND() LIMIT 1 2018-03-06 19:52:43,906 WARN [o.e.j.w.WebAppContext] (main:null) (logid:) Failed startup of context o.e.j.w.WebAppContext@23bb8443{/client,file:///usr/share/cloudstack-manageme nt/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp} <mailto:o.e.j.w.WebAppContext@23bb8443%7b/client,file:///usr/share/cloudstac k-management/webapp/,UNAVAILABLE%7d%7b/usr/share/cloudstack-management/webap p%7d> org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@1cf4d454 <mailto:com.mysql.jdbc.JDBC4PreparedStatement@1cf4d454> : SELECT network_offerings.id, network_offerings.name, network_offerings.unique_name, network_offerings.display_text, network_offerings.nw_rate, network_offerings.mc_rate, network_offerings.traffic_type, network_offerings.specify_vlan, network_offerings.system_only, network_offerings.service_offering_id, network_offerings.tags, network_offerings.default, network_offerings.availability, network_offerings.state, network_offerings.removed, network_offerings.created, network_offerings.guest_type, network_offerings.dedicated_lb_service, network_offerings.shared_source_nat_service, network_offerings.specify_ip_ranges, network_offerings.sort_key, network_offerings.uuid, network_offerings.redundant_router_service, network_offerings.conserve_mode, network_offerings.elastic_ip_service, network_offerings.eip_associate_public_ip, network_offerings.elastic_lb_service, network_offerings.inline, network_offerings.is_persistent, network_offerings.for_vpc, network_offerings.egress_default_policy, network_offerings.concurrent_connections, network_offerings.keep_alive_enabled, network_offerings.supports_streched_l2, network_offerings.supports_public_access, network_offerings.internal_lb, network_offerings.public_lb, network_offerings.service_package_id FROM network_offerings WHERE network_offerings.unique_name = x'517569636B436C6F75644E6F5365727669636573' AND network_offerings.removed IS NULL ORDER BY RAND() LIMIT 1 at org.springframework.context.support.DefaultLifecycleProcessor.doStart(Defaul tLifecycleProcessor.java:186) at org.springframework.context.support.DefaultLifecycleProcessor.access$200(Def aultLifecycleProcessor.java:52) at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup .start(DefaultLifecycleProcessor.java:358) at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(Def aultLifecycleProcessor.java:159) at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(Defa ultLifecycleProcessor.java:123) at org.springframework.context.support.AbstractApplicationContext.finishRefresh (AbstractApplicationContext.java:884) at org.springframework.context.support.AbstractApplicationContext.refresh(Abstr actApplicationContext.java:552) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.lo adContext(DefaultModuleDefinitionSet.java:145) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2. with(DefaultModuleDefinitionSet.java:122) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.wi thModule(DefaultModuleDefinitionSet.java:245) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.wi thModule(DefaultModuleDefinitionSet.java:250) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.wi thModule(DefaultModuleDefinitionSet.java:250) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.wi thModule(DefaultModuleDefinitionSet.java:233) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.lo adContexts(DefaultModuleDefinitionSet.java:117) at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.lo ad(DefaultModuleDefinitionSet.java:79) at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadMo dules(ModuleBasedContextFactory.java:37) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(Clo udStackSpringContext.java:71) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(C loudStackSpringContext.java:58) at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(C loudStackSpringContext.java:62) at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.cont extInitialized(CloudStackContextLoaderListener.java:52) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(Conte xtHandler.java:890) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(Servl etContextHandler.java:532) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler. java:853) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextH andler.java:344) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java: 785) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandle r.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle .java:133) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCyc le.java:107) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.jav a:113) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:2 73) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle .java:133) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCyc le.java:115) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.jav a:113) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle .java:133) at org.eclipse.jetty.server.Server.start(Server.java:418) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCyc le.java:107) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.jav a:113) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.j ava:68) at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:241) Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@1cf4d454 <mailto:com.mysql.jdbc.JDBC4PreparedStatement@1cf4d454> : SELECT network_offerings.id, network_offerings.name, network_offerings.unique_name, network_offerings.display_text, network_offerings.nw_rate, network_offerings.mc_rate, network_offerings.traffic_type, network_offerings.specify_vlan, network_offerings.system_only, network_offerings.service_offering_id, network_offerings.tags, network_offerings.default, network_offerings.availability, network_offerings.state, network_offerings.removed, network_offerings.created, network_offerings.guest_type, network_offerings.dedicated_lb_service, network_offerings.shared_source_nat_service, network_offerings.specify_ip_ranges, network_offerings.sort_key, network_offerings.uuid, network_offerings.redundant_router_service, network_offerings.conserve_mode, network_offerings.elastic_ip_service, network_offerings.eip_associate_public_ip, network_offerings.elastic_lb_service, network_offerings.inline, network_offerings.is_persistent, network_offerings.for_vpc, network_offerings.egress_default_policy, network_offerings.concurrent_connections, network_offerings.keep_alive_enabled, network_offerings.supports_streched_l2, network_offerings.supports_public_access, network_offerings.internal_lb, network_offerings.public_lb, network_offerings.service_package_id FROM network_offerings WHERE network_offerings.unique_name = x'517569636B436C6F75644E6F5365727669636573' AND network_offerings.removed IS NULL ORDER BY RAND() LIMIT 1 at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java :428) at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java :364) at com.cloud.utils.db.GenericDaoBase.findOneIncludingRemovedBy(GenericDaoBase.j ava:892) ... 76 more Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'network_offerings.service_package_id' in 'field list' at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces sorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc torAccessorImpl.java:45) ... 92 more 2018-03-06 19:52:43,908 INFO [o.e.j.s.h.ContextHandler] (main:null) (logid:) Started o.e.j.s.h.MovedContextHandler@1e4a7dd4{/,null,AVAILABLE} <mailto:o.e.j.s.h.MovedContextHandler@1e4a7dd4%7b/,null,AVAILABLE%7d> 2018-03-06 19:52:43,925 INFO [o.e.j.s.AbstractConnector] (main:null) (logid:) Started ServerConnector@76f2b07d{HTTP/1.1,[http/1.1]}{:::8080} <mailto:ServerConnector@76f2b07d%7bHTTP/1.1,%5bhttp/1.1%5d%7d%7b:::8080%7d> 2018-03-06 19:52:43,925 INFO [o.e.j.s.Server] (main:null) (logid:) Started @25928ms 2018-03-06 19:53:43,863 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-7e960663) (logid:26fa7a85) Zone 3 is not ready to launch secondary storage VM yet 2018-03-06 19:53:43,893 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-dcddc9fb) (logid:16287824) Zone 3 is not ready to launch console proxy yet 2018-03-06 19:54:13,856 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-215ad2c1) (logid:8bc41e24) Zone 3 is not ready to launch secondary storage VM yet 2018-03-06 19:54:13,880 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-3f88b880) (logid:4d897d1f) Zone 3 is not ready to launch console proxy yet 2018-03-06 19:54:43,857 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-8c3ea998) (logid:d8b3049e) Zone 3 is not ready to launch secondary storage VM yet 2018-03-06 19:54:43,884 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-8bd27be7) (logid:8230c30b) Zone 3 is not ready to launch console proxy yet ^C [root@dc13-r4-wpr1-h1 ~]# ^C [root@dc13-r4-wpr1-h1 ~]# ^C [root@dc13-r4-wpr1-h1 ~]#