Hi Josh, To send a smaller file, you could clear the log file and wait for 2 or 3 iterations of the SSVMs to spin up and fail:
1. Stop Cloudstack Management Service 2. Stop Cloudstack Usage Service 3. Copy off the /var/log/cloudstack/management/management-server.log file to another location 4. Truncate the /var/log/cloudstack/management/management-server.log file 5. Start the CloudStack Management Service You can tail the /var/log/cloudstack/management/management-server.log file to keep an eye on it, also have your Hypervisor console open to monitor if the SSVM Templates are being copied over etc. Let the SSVMs cycle 5 or so times, stop the management service and paste us the /var/log/cloudstack/management/management-server.log logs. There are a number of things which could cause this (as mentioned by previous posters) Networking, Secondary Storage etc. Lets see what the logs have to say. Kind Regards, Timothy Lothering -----Original Message----- From: Sanjeev Neelarapu [mailto:sanjeev.neelar...@accelerite.com] Sent: Tuesday, 10 May 2016 8:44 AM To: users@cloudstack.apache.org Subject: RE: System VMs keeps failing to start Hi Josh, Since you are using clustered management server with HA, make sure that you have set the global setting parameter "host" to the load balance/virtual IP address. If not, please set , restart management server and destroy system vms. Best Regards, Sanjeev N Chief Product Engineer, Accelerite Off: +91 40 6722 9368 | EMail: sanjeev.neelar...@accelerite.com -----Original Message----- From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] Sent: Monday, May 09, 2016 4:46 PM To: users@cloudstack.apache.org Subject: Re: System VMs keeps failing to start System VM connect back to management server on port 8250. Some of the other ports to be aware of are here : https://cwiki.apache.org/confluence/display/CLOUDSTACK/Ports+used+by+CloudStack On 04/03/16, 4:28 PM, "cloudstackh...@outlook.com" <cloudstackh...@outlook.com> wrote: > > >Hi, > > >Before I dig through that I was just wondering if maybe it's really just a >networking issue. Here's how my network is setup right now: > > >a. Management server cluster running galera sitting behind a gateway that NATs >port 8080 to a HAproxy load balancer. All public traffic that is a result from >MS requests get allowed through. Everything else is rejected. > > >b. Xenserver HVs with 4 NICs connected to 4 different switches: > >1. L3 switch with connection to internet (public subnet) > >2. L2 switch for management network which is connected to the MS >cluster and the secondary NFS share (192.168.2.0/24) > >3. L2 switch for storage network where the EQL SAN sits on >(192.168.10.0/24) > >4. L2 switch for the guest network (10.10.1.0/16) > > >I'm not too sure if I'm doing stuff wrongly. > > >Josh > > > > > > >On Thu, Mar 3, 2016 at 2:07 PM -0800, "Ahmad Emneina" <aemne...@gmail.com> >wrote: > > > > > >If you see any exceptions, I'd ideally like to see a few hundred lines >above and below. It might be easiest to stop the management service, >rename the log file. Restart the service, observe the system vm's go >through their life cycle... stop the management server and post that. >Whatever works best for you. > >On Thu, Mar 3, 2016 at 1:43 PM, <cloudstackh...@outlook.com> wrote: > >> >> >> It's really huge. Which part am I looking for exactly? >> >> >> >> >> >> >> On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina" >> <aemne...@gmail.com> >> wrote: >> >> >> >> >> >> Josh, can you share the logs off the management server. Namely: >> /var/log/cloudstack/management/management-server.log >> Post as much as you can to pastebin or similar. That'll help identify >> what part of the process is failing... >> >> Ahmad E >> >> > On Mar 3, 2016, at 12:44 PM, <cloudstackh...@outlook.com> < >> cloudstackh...@outlook.com> wrote: >> > >> > >> > >> > Nothing's being spun up on the HVs. I'm using separate networks for >> > each >> component (public, management, guest, storage). They all have a >> dedicated NIC each. On the HVs it seems like CS created its own cloud >> link local network but the link status is <none> >> > >> > >> > Thanks >> > >> > Josh >> > >> > >> > >> > >> > >> > >> > On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina" < >> aemne...@gmail.com> wrote: >> > >> > >> > >> > >> > >> > Hey Josh, >> > >> > Are you seeing the VM's being spun up on the backend (hypervisor)? >> > If so, I'd imagine its a communication issue between the management >> > server and >> the >> > public interface on the system vm. If you use VLAN tagging for your >> public >> > network, make sure the VLAN is trunked to your hypervisors in the cloud. >> I >> > recommend you stop the management service. Once restarted >> > CloudStack will try to recycle those vm's and spin them up again >> > (so no worries should be had there). If you're able to time it >> > correctly, you can stop the management service before the system >> > vm's get shut down and log into them... make sure the respective >> > interfaces can reach their next hops... >> > that would be a good first step. >> > >> >> On Thu, Mar 3, 2016 at 12:32 PM, <cloudstackh...@outlook.com> wrote: >> >> >> >> >> >> >> >> Hi Ron and all, >> >> >> >> >> >> >> >> You mentioned that SVMs need to be started. I'm finding that my >> >> installation keeps looping the command "Creating system VMs (this >> >> may >> take >> >> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems >> >> to >> want >> >> to keep going on until it crashes. >> >> >> >> >> >> >> >> I'm thinking I should just kill the process but I'm worried I >> >> can't >> start >> >> the process again later. Is there a way to re-run this again later on? >> >> >> >> >> >> >> >> Why is it failing to start the VMs? Why is everything null? My >> >> networks are starting fine. Apologies for the lack of formating. >> >> Sending this via phone. >> >> >> >> >> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: >> >> null, private IP: N/A1004 Mar 2016 04:27:04 >> >> >> >> Console proxy creation failure. zone: Public Cloud, error details: >> >> null1004 Mar 2016 04:27:04 >> >> >> >> Secondary Storage Vm creation failure. zone: Public Cloud, error >> details: >> >> null1904 Mar 2016 04:27:00 >> >> >> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: >> >> null, private IP: N/A1004 Mar 2016 04:26:34 >> >> >> >> Console proxy creation failure. zone: Public Cloud, error details: >> >> null1004 Mar 2016 04:26:34 >> >> >> >> >> >> Thanks in advance! >> >> >> >> Josh >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" < >> >> rwhee...@artifact-software.com> wrote: >> >> >> >> >> >> >> >> >> >> >> >> I have been using Linux and the Internet since the mid 1990s. >> >> There are still 3 consecutively numbered C class registered to me >> >> for different clients back in the days when c-class networks were >> >> given out from massive ranges of free numbers. >> >> I have set up small ISP operations for clients with multiple >> >> domains including web sites, e-mail servers, fileservers, etc. >> >> I have done this on SCO , Mandrake, CentOS 4 to 7. >> >> >> >> I should not have to struggle to figure out how set up Cloudstack >> >> in a small configuration with a few servers and a single public IP. >> >> >> >> The documentation on networking is jumbled about and so unclear >> >> that I can only point out why it is not clear but can not figure >> >> out the truth sufficiently well to actually fix it. >> >> I still don't know where the sources for the drawings are kept >> >> even though I have asked several times. >> >> >> >> It needs a team approach with someone who knows the truth and >> >> someone who can write it down so that someone who did not write >> >> the code can figure out what to do. >> >> >> >> The biggest problem with programmers writing the user >> >> documentation is that they are so caught up in the exceptions and special >> >> cases. >> >> They spent a lot of time figuring out how to handle these oddball >> >> cases that they feel that these triumphs must be on the front page. >> >> They forget to explain the 95% case and lace the description of >> >> the main flow with notes about these interesting exceptions. >> >> >> >> That is not just true for Cloudstack but is a general problem with >> >> documentation just because we are all human. >> >> >> >> They also forget that the user does not want to be an expert in >> >> the topic but wants to know enough to get the thing running. >> >> The user has a lot of other problems and does not to become a >> >> developer in order to get this to work. >> >> >> >> In my case, I really need to get some internal applications >> >> (accounting, SCM, issue tracking, Maven repo, 20 web sites etc.) >> >> running on virtual machines in an environment that is easy to >> >> manage. I want to support clients who I am supporting as users of >> >> other systems - just want simple low volume services to support my >> >> supporting of their users. >> >> >> >> I only expect to have 4 servers, one NIC per machine to support 1 >> >> transaction per second on a busy day I may get down to 2 servers >> >> if Cloudstack works well and allows me to manage test servers and >> >> run docker nicely. >> >> >> >> I do not want to know enough to be the network administrator at >> >> Google or Amazon. >> >> >> >> This should not be hard to implement and from what I have seen it >> >> is not but the networking docs are a major barrier to acceptance >> >> by mid-market companies - 300-1000 users with 1 or 2 System Admins >> >> who have to support all of the operations requirements and help >> >> developers and application support teams test and keep production systems >> >> running. >> >> >> >> Ron >> >> >> >> >> >>> On 03/03/2016 6:22 AM, Mario Giammarco wrote: >> >>> Simon Weller <sweller@...> writes: >> >>> >> >>>> I do agree that the docs are confusing, especially if you have a >> limited >> >>> knowledge of networking concepts. >> >>>> In terms of the complexity, a lot of that has to do with the >> >>>> fact that >> >>> every company has different service >> >>>> requirements and ACS needs to be flexible enough to accommodate >> >>>> very >> >>> different underlying needs. >> >>> Not agree. Even with good knowledge documentation is confusing because: >> >>> >> >>> - it assumes you are always in the use case of "I have plenty of >> >> routable ips" >> >>> - it forgets to say that two system vms are create to manage >> >>> routing >> and >> >>> secondary storage >> >>> - it does not say that cloudstack manager can rewrite your host >> >> configuration >> >>> >> >>> >> >>> >> >>>> It's always best to start with a basic zone, unless you REALLY >> >>>> need >> some >> >>> functionality within an advanced >> >>>> zone. As soon as you move into advanced zone networking, you >> >>>> need to >> >> have >> >>> a good understanding of layer 2/3 >> >>>> networking. >> >>> I was able to make my cloudstack network working only when I >> >>> skipped >> >> basic >> >>> zone and used advanced zone >> >> >> >> >> >> -- >> >> Ron Wheeler >> >> President >> >> Artifact Software Inc >> >> email: rwhee...@artifact-software.com >> >> skype: ronaldmwheeler >> >> phone: 866-970-2435, ext 102 >> >> >> >> >> Regards, Abhinandan Prateek abhinandan.prat...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails. Timothy Lothering Solutions Architect Managed Services T: +27877415535 F: +27877415100 C: +27824904099 E: tlother...@datacentrix.co.za DISCLAIMER NOTICE: Everything in this e-mail and any attachments relating to the official business of Datacentrix Holdings Ltd. and its subsidiaries ('Datacentrix') is proprietary to Datacentrix. It is confidential, legally privileged and protected by law. Datacentrix does not own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of Datacentrix. The person addressed in the e-mail is the sole authorised recipient. Please notify the sender immediately if it has unintentionally reached you and do not read, disclose or use the content in any way. Datacentrix cannot assure that the integrity of this communication has been maintained nor that it is free of errors, virus, interception or interference.
