RE: System VMs keeps failing to start

[Timothy Lothering]

Hi Josh,

To send a smaller file, you could clear the log file and wait for 2 or 3 
iterations of the SSVMs to spin up and fail:

1. Stop Cloudstack Management Service
2. Stop Cloudstack Usage Service
3. Copy off the /var/log/cloudstack/management/management-server.log file to 
another location
4. Truncate the /var/log/cloudstack/management/management-server.log file
5. Start the CloudStack Management Service

You can tail the /var/log/cloudstack/management/management-server.log file to 
keep an eye on it, also have your Hypervisor console open to monitor if the 
SSVM Templates are being copied over etc. Let the SSVMs cycle 5 or so times, 
stop the management service and paste us the 
/var/log/cloudstack/management/management-server.log logs.

There are a number of things which could cause this (as mentioned by previous 
posters) Networking, Secondary Storage etc. Lets see what the logs have to say.

Kind Regards,
Timothy Lothering

-Original Message-
From: Sanjeev Neelarapu [mailto:sanjeev.neelar...@accelerite.com] 
Sent: Tuesday, 10 May 2016 8:44 AM
To: users@cloudstack.apache.org
Subject: RE: System VMs keeps failing to start

Hi Josh,

Since you are using clustered management server with HA, make sure that you 
have set the global setting parameter "host" to the load balance/virtual IP 
address. If not, please set , restart management server and destroy system vms.

Best Regards,
Sanjeev N
Chief Product Engineer, Accelerite
Off: +91 40 6722 9368 | EMail: sanjeev.neelar...@accelerite.com 



-Original Message-
From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
Sent: Monday, May 09, 2016 4:46 PM
To: users@cloudstack.apache.org
Subject: Re: System VMs keeps failing to start

System VM connect back to management server on port 8250. Some of the other 
ports to be aware of are here : 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Ports+used+by+CloudStack




On 04/03/16, 4:28 PM, "cloudstackh...@outlook.com" <cloudstackh...@outlook.com> 
wrote:

>
>
>Hi,
>
>
>Before I dig through that I was just wondering if maybe it's really just a 
>networking issue. Here's how my network is setup right now:
>
>
>a. Management server cluster running galera sitting behind a gateway that NATs 
>port 8080 to a HAproxy load balancer. All public traffic that is a result from 
>MS requests get allowed through. Everything else is rejected.
>
>
>b. Xenserver HVs with 4 NICs connected to 4 different switches:
>
>1. L3 switch with connection to internet (public subnet)
>
>2. L2 switch for management network which is connected to the MS 
>cluster and the secondary NFS share (192.168.2.0/24)
>
>3. L2 switch for storage network where the EQL SAN sits on
>(192.168.10.0/24)
>
>4. L2 switch for the guest network (10.10.1.0/16)
>
>
>I'm not too sure if I'm doing stuff wrongly.
>
>
>Josh
>
>
>
>
>
>
>On Thu, Mar 3, 2016 at 2:07 PM -0800, "Ahmad Emneina" <aemne...@gmail.com> 
>wrote:
>
>
>
>
>
>If you see any exceptions, I'd ideally like to see a few hundred lines 
>above and below. It might be easiest to stop the management service, 
>rename the log file. Restart the service, observe the system vm's go 
>through their life cycle... stop the management server and post that.
>Whatever works best for you.
>
>On Thu, Mar 3, 2016 at 1:43 PM, <cloudstackh...@outlook.com> wrote:
>
>>
>>
>> It's really huge. Which part am I looking for exactly?
>>
>>
>>
>>
>>
>>
>> On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina" 
>> <aemne...@gmail.com>
>> wrote:
>>
>>
>>
>>
>>
>> Josh, can you share the logs off the management server. Namely:
>> /var/log/cloudstack/management/management-server.log
>> Post as much as you can to pastebin or similar. That'll help identify 
>> what part of the process is failing...
>>
>> Ahmad E
>>
>> > On Mar 3, 2016, at 12:44 PM, <cloudstackh...@outlook.com> <
>> cloudstackh...@outlook.com> wrote:
>> >
>> >
>> >
>> > Nothing's being spun up on the HVs. I'm using separate networks for 
>> > each
>> component (public, management, guest, storage). They all have a 
>> dedicated NIC each. On the HVs it seems like CS created its own cloud 
>> link local network but the link status is 
>> >
>> >
>> > Thanks
>> >
>> > Josh
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina" <
>> aemne...@gmail.com> wrote:
>> >
>> >
>> >
>> >
>> >

RE: System VMs keeps failing to start

[Glenn Wagner]

Hi Josh,

Once you have done that , could you paste your management server logs to 
pastbin,
We are looking at the time around the deployment of the system vm's, if there 
is a failure it will be an exception (ERROR message)

Regards
Glenn



Regards,

Glenn Wagner

glenn.wag...@shapeblue.com 
www.shapeblue.com
2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West, Cape Town  7130South 
Africa
@shapeblue

-Original Message-
From: Sanjeev Neelarapu [mailto:sanjeev.neelar...@accelerite.com] 
Sent: Tuesday, 10 May 2016 8:44 AM
To: users@cloudstack.apache.org
Subject: RE: System VMs keeps failing to start

Hi Josh,

Since you are using clustered management server with HA, make sure that you 
have set the global setting parameter "host" to the load balance/virtual IP 
address. If not, please set , restart management server and destroy system vms.

Best Regards,
Sanjeev N
Chief Product Engineer, Accelerite
Off: +91 40 6722 9368 | EMail: sanjeev.neelar...@accelerite.com 



-Original Message-
From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
Sent: Monday, May 09, 2016 4:46 PM
To: users@cloudstack.apache.org
Subject: Re: System VMs keeps failing to start

System VM connect back to management server on port 8250. Some of the other 
ports to be aware of are here : 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Ports+used+by+CloudStack




On 04/03/16, 4:28 PM, "cloudstackh...@outlook.com" <cloudstackh...@outlook.com> 
wrote:

>
>
>Hi,
>
>
>Before I dig through that I was just wondering if maybe it's really just a 
>networking issue. Here's how my network is setup right now:
>
>
>a. Management server cluster running galera sitting behind a gateway that NATs 
>port 8080 to a HAproxy load balancer. All public traffic that is a result from 
>MS requests get allowed through. Everything else is rejected.
>
>
>b. Xenserver HVs with 4 NICs connected to 4 different switches:
>
>1. L3 switch with connection to internet (public subnet)
>
>2. L2 switch for management network which is connected to the MS 
>cluster and the secondary NFS share (192.168.2.0/24)
>
>3. L2 switch for storage network where the EQL SAN sits on
>(192.168.10.0/24)
>
>4. L2 switch for the guest network (10.10.1.0/16)
>
>
>I'm not too sure if I'm doing stuff wrongly.
>
>
>Josh
>
>
>
>
>
>
>On Thu, Mar 3, 2016 at 2:07 PM -0800, "Ahmad Emneina" <aemne...@gmail.com> 
>wrote:
>
>
>
>
>
>If you see any exceptions, I'd ideally like to see a few hundred lines 
>above and below. It might be easiest to stop the management service, 
>rename the log file. Restart the service, observe the system vm's go 
>through their life cycle... stop the management server and post that.
>Whatever works best for you.
>
>On Thu, Mar 3, 2016 at 1:43 PM, <cloudstackh...@outlook.com> wrote:
>
>>
>>
>> It's really huge. Which part am I looking for exactly?
>>
>>
>>
>>
>>
>>
>> On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina" 
>> <aemne...@gmail.com>
>> wrote:
>>
>>
>>
>>
>>
>> Josh, can you share the logs off the management server. Namely:
>> /var/log/cloudstack/management/management-server.log
>> Post as much as you can to pastebin or similar. That'll help identify 
>> what part of the process is failing...
>>
>> Ahmad E
>>
>> > On Mar 3, 2016, at 12:44 PM, <cloudstackh...@outlook.com> <
>> cloudstackh...@outlook.com> wrote:
>> >
>> >
>> >
>> > Nothing's being spun up on the HVs. I'm using separate networks for 
>> > each
>> component (public, management, guest, storage). They all have a 
>> dedicated NIC each. On the HVs it seems like CS created its own cloud 
>> link local network but the link status is 
>> >
>> >
>> > Thanks
>> >
>> > Josh
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina" <
>> aemne...@gmail.com> wrote:
>> >
>> >
>> >
>> >
>> >
>> > Hey Josh,
>> >
>> > Are you seeing the VM's being spun up on the backend (hypervisor)? 
>> > If so, I'd imagine its a communication issue between the management 
>> > server and
>> the
>> > public interface on the system vm. If you use VLAN tagging for your
>> public
>> > network, make sure the VLAN is trunked to your hypervisors in the cloud.
>> I
>> > recommend you stop the management service. Once restarted 
>> > Clou

RE: System VMs keeps failing to start

[Sanjeev Neelarapu]

Hi Josh,

Since you are using clustered management server with HA, make sure that you 
have set the global setting parameter "host" to the load balance/virtual IP 
address. If not, please set , restart management server and destroy system vms.

Best Regards,
Sanjeev N
Chief Product Engineer, Accelerite
Off: +91 40 6722 9368 | EMail: sanjeev.neelar...@accelerite.com 



-Original Message-
From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] 
Sent: Monday, May 09, 2016 4:46 PM
To: users@cloudstack.apache.org
Subject: Re: System VMs keeps failing to start

System VM connect back to management server on port 8250. Some of the other 
ports to be aware of are here : 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Ports+used+by+CloudStack




On 04/03/16, 4:28 PM, "cloudstackh...@outlook.com" <cloudstackh...@outlook.com> 
wrote:

>
>
>Hi,
>
>
>Before I dig through that I was just wondering if maybe it's really just a 
>networking issue. Here's how my network is setup right now:
>
>
>a. Management server cluster running galera sitting behind a gateway that NATs 
>port 8080 to a HAproxy load balancer. All public traffic that is a result from 
>MS requests get allowed through. Everything else is rejected.
>
>
>b. Xenserver HVs with 4 NICs connected to 4 different switches:
>
>1. L3 switch with connection to internet (public subnet)
>
>2. L2 switch for management network which is connected to the MS 
>cluster and the secondary NFS share (192.168.2.0/24)
>
>3. L2 switch for storage network where the EQL SAN sits on 
>(192.168.10.0/24)
>
>4. L2 switch for the guest network (10.10.1.0/16)
>
>
>I'm not too sure if I'm doing stuff wrongly.
>
>
>Josh
>
>
>
>
>
>
>On Thu, Mar 3, 2016 at 2:07 PM -0800, "Ahmad Emneina" <aemne...@gmail.com> 
>wrote:
>
>
>
>
>
>If you see any exceptions, I'd ideally like to see a few hundred lines 
>above and below. It might be easiest to stop the management service, 
>rename the log file. Restart the service, observe the system vm's go 
>through their life cycle... stop the management server and post that. 
>Whatever works best for you.
>
>On Thu, Mar 3, 2016 at 1:43 PM, <cloudstackh...@outlook.com> wrote:
>
>>
>>
>> It's really huge. Which part am I looking for exactly?
>>
>>
>>
>>
>>
>>
>> On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina" 
>> <aemne...@gmail.com>
>> wrote:
>>
>>
>>
>>
>>
>> Josh, can you share the logs off the management server. Namely:
>> /var/log/cloudstack/management/management-server.log
>> Post as much as you can to pastebin or similar. That'll help identify 
>> what part of the process is failing...
>>
>> Ahmad E
>>
>> > On Mar 3, 2016, at 12:44 PM, <cloudstackh...@outlook.com> <
>> cloudstackh...@outlook.com> wrote:
>> >
>> >
>> >
>> > Nothing's being spun up on the HVs. I'm using separate networks for 
>> > each
>> component (public, management, guest, storage). They all have a 
>> dedicated NIC each. On the HVs it seems like CS created its own cloud 
>> link local network but the link status is 
>> >
>> >
>> > Thanks
>> >
>> > Josh
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina" <
>> aemne...@gmail.com> wrote:
>> >
>> >
>> >
>> >
>> >
>> > Hey Josh,
>> >
>> > Are you seeing the VM's being spun up on the backend (hypervisor)? 
>> > If so, I'd imagine its a communication issue between the management 
>> > server and
>> the
>> > public interface on the system vm. If you use VLAN tagging for your
>> public
>> > network, make sure the VLAN is trunked to your hypervisors in the cloud.
>> I
>> > recommend you stop the management service. Once restarted 
>> > CloudStack will try to recycle those vm's and spin them up again 
>> > (so no worries should be had there). If you're able to time it 
>> > correctly, you can stop the management service before the system 
>> > vm's get shut down and log into them... make sure the respective 
>> > interfaces can reach their next hops...
>> > that would be a good first step.
>> >
>> >> On Thu, Mar 3, 2016 at 12:32 PM, <cloudstackh...@outlook.com> wrote:
>> >>
>> >>
>> >>
>> >> Hi Ron and all,
>> >>
>> >>
&

Re: System VMs keeps failing to start

[Abhinandan Prateek]

System VM connect back to management server on port 8250. Some of the other 
ports to be aware of are here : 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Ports+used+by+CloudStack




On 04/03/16, 4:28 PM, "cloudstackh...@outlook.com"  
wrote:

>
>
>Hi,
>
>
>Before I dig through that I was just wondering if maybe it's really just a 
>networking issue. Here's how my network is setup right now:
>
>
>a. Management server cluster running galera sitting behind a gateway that NATs 
>port 8080 to a HAproxy load balancer. All public traffic that is a result from 
>MS requests get allowed through. Everything else is rejected.
>
>
>b. Xenserver HVs with 4 NICs connected to 4 different switches:
>
>1. L3 switch with connection to internet (public subnet)
>
>2. L2 switch for management network which is connected to the MS cluster and 
>the secondary NFS share (192.168.2.0/24)
>
>3. L2 switch for storage network where the EQL SAN sits on (192.168.10.0/24)
>
>4. L2 switch for the guest network (10.10.1.0/16)
>
>
>I'm not too sure if I'm doing stuff wrongly.
>
>
>Josh
>
>
>
>
>
>
>On Thu, Mar 3, 2016 at 2:07 PM -0800, "Ahmad Emneina"  
>wrote:
>
>
>
>
>
>If you see any exceptions, I'd ideally like to see a few hundred lines
>above and below. It might be easiest to stop the management service, rename
>the log file. Restart the service, observe the system vm's go through their
>life cycle... stop the management server and post that. Whatever works best
>for you.
>
>On Thu, Mar 3, 2016 at 1:43 PM,  wrote:
>
>>
>>
>> It's really huge. Which part am I looking for exactly?
>>
>>
>>
>>
>>
>>
>> On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina" 
>> wrote:
>>
>>
>>
>>
>>
>> Josh, can you share the logs off the management server. Namely:
>> /var/log/cloudstack/management/management-server.log
>> Post as much as you can to pastebin or similar. That'll help identify what
>> part of the process is failing...
>>
>> Ahmad E
>>
>> > On Mar 3, 2016, at 12:44 PM,  <
>> cloudstackh...@outlook.com> wrote:
>> >
>> >
>> >
>> > Nothing's being spun up on the HVs. I'm using separate networks for each
>> component (public, management, guest, storage). They all have a dedicated
>> NIC each. On the HVs it seems like CS created its own cloud link local
>> network but the link status is 
>> >
>> >
>> > Thanks
>> >
>> > Josh
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina" <
>> aemne...@gmail.com> wrote:
>> >
>> >
>> >
>> >
>> >
>> > Hey Josh,
>> >
>> > Are you seeing the VM's being spun up on the backend (hypervisor)? If so,
>> > I'd imagine its a communication issue between the management server and
>> the
>> > public interface on the system vm. If you use VLAN tagging for your
>> public
>> > network, make sure the VLAN is trunked to your hypervisors in the cloud.
>> I
>> > recommend you stop the management service. Once restarted CloudStack will
>> > try to recycle those vm's and spin them up again (so no worries should be
>> > had there). If you're able to time it correctly, you can stop the
>> > management service before the system vm's get shut down and log into
>> > them... make sure the respective interfaces can reach their next hops...
>> > that would be a good first step.
>> >
>> >> On Thu, Mar 3, 2016 at 12:32 PM,  wrote:
>> >>
>> >>
>> >>
>> >> Hi Ron and all,
>> >>
>> >>
>> >>
>> >> You mentioned that SVMs need to be started. I'm finding that my
>> >> installation keeps looping the command "Creating system VMs (this may
>> take
>> >> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems to
>> want
>> >> to keep going on until it crashes.
>> >>
>> >>
>> >>
>> >> I'm thinking I should just kill the process but I'm worried I can't
>> start
>> >> the process again later. Is there a way to re-run this again later on?
>> >>
>> >>
>> >>
>> >> Why is it failing to start the VMs? Why is everything null? My networks
>> >> are starting fine. Apologies for the lack of formating. Sending this via
>> >> phone.
>> >>
>> >>
>> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
>> >> private IP: N/A1004 Mar 2016 04:27:04
>> >>
>> >> Console proxy creation failure. zone: Public Cloud, error details:
>> >> null1004 Mar 2016 04:27:04
>> >>
>> >> Secondary Storage Vm creation failure. zone: Public Cloud, error
>> details:
>> >> null1904 Mar 2016 04:27:00
>> >>
>> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
>> >> private IP: N/A1004 Mar 2016 04:26:34
>> >>
>> >> Console proxy creation failure. zone: Public Cloud, error details:
>> >> null1004 Mar 2016 04:26:34
>> >>
>> >>
>> >> Thanks in advance!
>> >>
>> >> Josh
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" <
>> >> rwhee...@artifact-software.com> wrote:
>> >>
>> >>
>> >>
>> >>
>> >>
>> 

Re: System VMs keeps failing to start

[cloudstackhelp]

Hi,


Before I dig through that I was just wondering if maybe it's really just a 
networking issue. Here's how my network is setup right now:


a. Management server cluster running galera sitting behind a gateway that NATs 
port 8080 to a HAproxy load balancer. All public traffic that is a result from 
MS requests get allowed through. Everything else is rejected.


b. Xenserver HVs with 4 NICs connected to 4 different switches:

1. L3 switch with connection to internet (public subnet)

2. L2 switch for management network which is connected to the MS cluster and 
the secondary NFS share (192.168.2.0/24)

3. L2 switch for storage network where the EQL SAN sits on (192.168.10.0/24)

4. L2 switch for the guest network (10.10.1.0/16)


I'm not too sure if I'm doing stuff wrongly.


Josh






On Thu, Mar 3, 2016 at 2:07 PM -0800, "Ahmad Emneina"  
wrote:





If you see any exceptions, I'd ideally like to see a few hundred lines
above and below. It might be easiest to stop the management service, rename
the log file. Restart the service, observe the system vm's go through their
life cycle... stop the management server and post that. Whatever works best
for you.

On Thu, Mar 3, 2016 at 1:43 PM,  wrote:

>
>
> It's really huge. Which part am I looking for exactly?
>
>
>
>
>
>
> On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina" 
> wrote:
>
>
>
>
>
> Josh, can you share the logs off the management server. Namely:
> /var/log/cloudstack/management/management-server.log
> Post as much as you can to pastebin or similar. That'll help identify what
> part of the process is failing...
>
> Ahmad E
>
> > On Mar 3, 2016, at 12:44 PM,  <
> cloudstackh...@outlook.com> wrote:
> >
> >
> >
> > Nothing's being spun up on the HVs. I'm using separate networks for each
> component (public, management, guest, storage). They all have a dedicated
> NIC each. On the HVs it seems like CS created its own cloud link local
> network but the link status is 
> >
> >
> > Thanks
> >
> > Josh
> >
> >
> >
> >
> >
> >
> > On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina" <
> aemne...@gmail.com> wrote:
> >
> >
> >
> >
> >
> > Hey Josh,
> >
> > Are you seeing the VM's being spun up on the backend (hypervisor)? If so,
> > I'd imagine its a communication issue between the management server and
> the
> > public interface on the system vm. If you use VLAN tagging for your
> public
> > network, make sure the VLAN is trunked to your hypervisors in the cloud.
> I
> > recommend you stop the management service. Once restarted CloudStack will
> > try to recycle those vm's and spin them up again (so no worries should be
> > had there). If you're able to time it correctly, you can stop the
> > management service before the system vm's get shut down and log into
> > them... make sure the respective interfaces can reach their next hops...
> > that would be a good first step.
> >
> >> On Thu, Mar 3, 2016 at 12:32 PM,  wrote:
> >>
> >>
> >>
> >> Hi Ron and all,
> >>
> >>
> >>
> >> You mentioned that SVMs need to be started. I'm finding that my
> >> installation keeps looping the command "Creating system VMs (this may
> take
> >> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems to
> want
> >> to keep going on until it crashes.
> >>
> >>
> >>
> >> I'm thinking I should just kill the process but I'm worried I can't
> start
> >> the process again later. Is there a way to re-run this again later on?
> >>
> >>
> >>
> >> Why is it failing to start the VMs? Why is everything null? My networks
> >> are starting fine. Apologies for the lack of formating. Sending this via
> >> phone.
> >>
> >>
> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> >> private IP: N/A1004 Mar 2016 04:27:04
> >>
> >> Console proxy creation failure. zone: Public Cloud, error details:
> >> null1004 Mar 2016 04:27:04
> >>
> >> Secondary Storage Vm creation failure. zone: Public Cloud, error
> details:
> >> null1904 Mar 2016 04:27:00
> >>
> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> >> private IP: N/A1004 Mar 2016 04:26:34
> >>
> >> Console proxy creation failure. zone: Public Cloud, error details:
> >> null1004 Mar 2016 04:26:34
> >>
> >>
> >> Thanks in advance!
> >>
> >> Josh
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" <
> >> rwhee...@artifact-software.com> wrote:
> >>
> >>
> >>
> >>
> >>
> >> I have been using Linux and the Internet since the mid 1990s.
> >> There are still 3 consecutively numbered C class registered to me for
> >> different clients back in the days when  c-class networks were given out
> >> from massive ranges of free numbers.
> >> I have set up small ISP operations for clients with multiple domains
> >> including web sites, e-mail servers, fileservers, etc.
> >> I have done this on SCO , Mandrake, CentOS 4 to 7.
> >>
> >> I 

Re: System VMs keeps failing to start

[Ahmad Emneina]

If you see any exceptions, I'd ideally like to see a few hundred lines
above and below. It might be easiest to stop the management service, rename
the log file. Restart the service, observe the system vm's go through their
life cycle... stop the management server and post that. Whatever works best
for you.

On Thu, Mar 3, 2016 at 1:43 PM,  wrote:

>
>
> It's really huge. Which part am I looking for exactly?
>
>
>
>
>
>
> On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina" 
> wrote:
>
>
>
>
>
> Josh, can you share the logs off the management server. Namely:
> /var/log/cloudstack/management/management-server.log
> Post as much as you can to pastebin or similar. That'll help identify what
> part of the process is failing...
>
> Ahmad E
>
> > On Mar 3, 2016, at 12:44 PM,  <
> cloudstackh...@outlook.com> wrote:
> >
> >
> >
> > Nothing's being spun up on the HVs. I'm using separate networks for each
> component (public, management, guest, storage). They all have a dedicated
> NIC each. On the HVs it seems like CS created its own cloud link local
> network but the link status is 
> >
> >
> > Thanks
> >
> > Josh
> >
> >
> >
> >
> >
> >
> > On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina" <
> aemne...@gmail.com> wrote:
> >
> >
> >
> >
> >
> > Hey Josh,
> >
> > Are you seeing the VM's being spun up on the backend (hypervisor)? If so,
> > I'd imagine its a communication issue between the management server and
> the
> > public interface on the system vm. If you use VLAN tagging for your
> public
> > network, make sure the VLAN is trunked to your hypervisors in the cloud.
> I
> > recommend you stop the management service. Once restarted CloudStack will
> > try to recycle those vm's and spin them up again (so no worries should be
> > had there). If you're able to time it correctly, you can stop the
> > management service before the system vm's get shut down and log into
> > them... make sure the respective interfaces can reach their next hops...
> > that would be a good first step.
> >
> >> On Thu, Mar 3, 2016 at 12:32 PM,  wrote:
> >>
> >>
> >>
> >> Hi Ron and all,
> >>
> >>
> >>
> >> You mentioned that SVMs need to be started. I'm finding that my
> >> installation keeps looping the command "Creating system VMs (this may
> take
> >> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems to
> want
> >> to keep going on until it crashes.
> >>
> >>
> >>
> >> I'm thinking I should just kill the process but I'm worried I can't
> start
> >> the process again later. Is there a way to re-run this again later on?
> >>
> >>
> >>
> >> Why is it failing to start the VMs? Why is everything null? My networks
> >> are starting fine. Apologies for the lack of formating. Sending this via
> >> phone.
> >>
> >>
> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> >> private IP: N/A1004 Mar 2016 04:27:04
> >>
> >> Console proxy creation failure. zone: Public Cloud, error details:
> >> null1004 Mar 2016 04:27:04
> >>
> >> Secondary Storage Vm creation failure. zone: Public Cloud, error
> details:
> >> null1904 Mar 2016 04:27:00
> >>
> >> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> >> private IP: N/A1004 Mar 2016 04:26:34
> >>
> >> Console proxy creation failure. zone: Public Cloud, error details:
> >> null1004 Mar 2016 04:26:34
> >>
> >>
> >> Thanks in advance!
> >>
> >> Josh
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" <
> >> rwhee...@artifact-software.com> wrote:
> >>
> >>
> >>
> >>
> >>
> >> I have been using Linux and the Internet since the mid 1990s.
> >> There are still 3 consecutively numbered C class registered to me for
> >> different clients back in the days when  c-class networks were given out
> >> from massive ranges of free numbers.
> >> I have set up small ISP operations for clients with multiple domains
> >> including web sites, e-mail servers, fileservers, etc.
> >> I have done this on SCO , Mandrake, CentOS 4 to 7.
> >>
> >> I should not have to struggle to figure out how set up Cloudstack in a
> >> small configuration with a few servers and a single public IP.
> >>
> >> The documentation on networking is jumbled about and so unclear that I
> >> can only point out why it is not clear but can not figure out the truth
> >> sufficiently well to actually fix it.
> >> I still don't know where the sources for the drawings are kept even
> >> though I have asked several times.
> >>
> >> It needs a team approach with someone who knows the truth and someone
> >> who can write it down so that someone who did not write the code can
> >> figure out what to do.
> >>
> >> The biggest problem with programmers writing the user documentation is
> >> that they are so caught up in the exceptions and special cases.
> >> They spent a lot of time figuring out how to handle these oddball cases
> >> that they feel that these 

Re: System VMs keeps failing to start

[cloudstackhelp]

It's really huge. Which part am I looking for exactly?






On Thu, Mar 3, 2016 at 1:38 PM -0800, "Ahmad Emneina"  
wrote:





Josh, can you share the logs off the management server. Namely: 
/var/log/cloudstack/management/management-server.log
Post as much as you can to pastebin or similar. That'll help identify what part 
of the process is failing...

Ahmad E

> On Mar 3, 2016, at 12:44 PM,  
>  wrote:
>
>
>
> Nothing's being spun up on the HVs. I'm using separate networks for each 
> component (public, management, guest, storage). They all have a dedicated NIC 
> each. On the HVs it seems like CS created its own cloud link local network 
> but the link status is 
>
>
> Thanks
>
> Josh
>
>
>
>
>
>
> On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina"  
> wrote:
>
>
>
>
>
> Hey Josh,
>
> Are you seeing the VM's being spun up on the backend (hypervisor)? If so,
> I'd imagine its a communication issue between the management server and the
> public interface on the system vm. If you use VLAN tagging for your public
> network, make sure the VLAN is trunked to your hypervisors in the cloud. I
> recommend you stop the management service. Once restarted CloudStack will
> try to recycle those vm's and spin them up again (so no worries should be
> had there). If you're able to time it correctly, you can stop the
> management service before the system vm's get shut down and log into
> them... make sure the respective interfaces can reach their next hops...
> that would be a good first step.
>
>> On Thu, Mar 3, 2016 at 12:32 PM,  wrote:
>>
>>
>>
>> Hi Ron and all,
>>
>>
>>
>> You mentioned that SVMs need to be started. I'm finding that my
>> installation keeps looping the command "Creating system VMs (this may take
>> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems to want
>> to keep going on until it crashes.
>>
>>
>>
>> I'm thinking I should just kill the process but I'm worried I can't start
>> the process again later. Is there a way to re-run this again later on?
>>
>>
>>
>> Why is it failing to start the VMs? Why is everything null? My networks
>> are starting fine. Apologies for the lack of formating. Sending this via
>> phone.
>>
>>
>> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
>> private IP: N/A1004 Mar 2016 04:27:04
>>
>> Console proxy creation failure. zone: Public Cloud, error details:
>> null1004 Mar 2016 04:27:04
>>
>> Secondary Storage Vm creation failure. zone: Public Cloud, error details:
>> null1904 Mar 2016 04:27:00
>>
>> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
>> private IP: N/A1004 Mar 2016 04:26:34
>>
>> Console proxy creation failure. zone: Public Cloud, error details:
>> null1004 Mar 2016 04:26:34
>>
>>
>> Thanks in advance!
>>
>> Josh
>>
>>
>>
>>
>>
>>
>> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" <
>> rwhee...@artifact-software.com> wrote:
>>
>>
>>
>>
>>
>> I have been using Linux and the Internet since the mid 1990s.
>> There are still 3 consecutively numbered C class registered to me for
>> different clients back in the days when  c-class networks were given out
>> from massive ranges of free numbers.
>> I have set up small ISP operations for clients with multiple domains
>> including web sites, e-mail servers, fileservers, etc.
>> I have done this on SCO , Mandrake, CentOS 4 to 7.
>>
>> I should not have to struggle to figure out how set up Cloudstack in a
>> small configuration with a few servers and a single public IP.
>>
>> The documentation on networking is jumbled about and so unclear that I
>> can only point out why it is not clear but can not figure out the truth
>> sufficiently well to actually fix it.
>> I still don't know where the sources for the drawings are kept even
>> though I have asked several times.
>>
>> It needs a team approach with someone who knows the truth and someone
>> who can write it down so that someone who did not write the code can
>> figure out what to do.
>>
>> The biggest problem with programmers writing the user documentation is
>> that they are so caught up in the exceptions and special cases.
>> They spent a lot of time figuring out how to handle these oddball cases
>> that they feel that these triumphs must be on the front page.
>> They forget to explain the 95% case and lace the description of the main
>> flow with notes about these interesting exceptions.
>>
>> That is not just true for Cloudstack but is a general problem with
>> documentation just because we are all human.
>>
>> They also forget that the user does not want to be an expert in the
>> topic but wants to know enough to get the thing running.
>> The user has a lot of other problems and does not to become a developer
>> in order to get this to work.
>>
>> In my case, I really need to get some internal applications (accounting,
>> SCM, issue tracking, Maven repo, 20 

Re: System VMs keeps failing to start

[Ahmad Emneina]

Josh, can you share the logs off the management server. Namely: 
/var/log/cloudstack/management/management-server.log 
Post as much as you can to pastebin or similar. That'll help identify what part 
of the process is failing...

Ahmad E

> On Mar 3, 2016, at 12:44 PM,  
>  wrote:
> 
> 
> 
> Nothing's being spun up on the HVs. I'm using separate networks for each 
> component (public, management, guest, storage). They all have a dedicated NIC 
> each. On the HVs it seems like CS created its own cloud link local network 
> but the link status is 
> 
> 
> Thanks
> 
> Josh
> 
> 
> 
> 
> 
> 
> On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina"  
> wrote:
> 
> 
> 
> 
> 
> Hey Josh,
> 
> Are you seeing the VM's being spun up on the backend (hypervisor)? If so,
> I'd imagine its a communication issue between the management server and the
> public interface on the system vm. If you use VLAN tagging for your public
> network, make sure the VLAN is trunked to your hypervisors in the cloud. I
> recommend you stop the management service. Once restarted CloudStack will
> try to recycle those vm's and spin them up again (so no worries should be
> had there). If you're able to time it correctly, you can stop the
> management service before the system vm's get shut down and log into
> them... make sure the respective interfaces can reach their next hops...
> that would be a good first step.
> 
>> On Thu, Mar 3, 2016 at 12:32 PM,  wrote:
>> 
>> 
>> 
>> Hi Ron and all,
>> 
>> 
>> 
>> You mentioned that SVMs need to be started. I'm finding that my
>> installation keeps looping the command "Creating system VMs (this may take
>> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems to want
>> to keep going on until it crashes.
>> 
>> 
>> 
>> I'm thinking I should just kill the process but I'm worried I can't start
>> the process again later. Is there a way to re-run this again later on?
>> 
>> 
>> 
>> Why is it failing to start the VMs? Why is everything null? My networks
>> are starting fine. Apologies for the lack of formating. Sending this via
>> phone.
>> 
>> 
>> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
>> private IP: N/A1004 Mar 2016 04:27:04
>> 
>> Console proxy creation failure. zone: Public Cloud, error details:
>> null1004 Mar 2016 04:27:04
>> 
>> Secondary Storage Vm creation failure. zone: Public Cloud, error details:
>> null1904 Mar 2016 04:27:00
>> 
>> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
>> private IP: N/A1004 Mar 2016 04:26:34
>> 
>> Console proxy creation failure. zone: Public Cloud, error details:
>> null1004 Mar 2016 04:26:34
>> 
>> 
>> Thanks in advance!
>> 
>> Josh
>> 
>> 
>> 
>> 
>> 
>> 
>> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" <
>> rwhee...@artifact-software.com> wrote:
>> 
>> 
>> 
>> 
>> 
>> I have been using Linux and the Internet since the mid 1990s.
>> There are still 3 consecutively numbered C class registered to me for
>> different clients back in the days when  c-class networks were given out
>> from massive ranges of free numbers.
>> I have set up small ISP operations for clients with multiple domains
>> including web sites, e-mail servers, fileservers, etc.
>> I have done this on SCO , Mandrake, CentOS 4 to 7.
>> 
>> I should not have to struggle to figure out how set up Cloudstack in a
>> small configuration with a few servers and a single public IP.
>> 
>> The documentation on networking is jumbled about and so unclear that I
>> can only point out why it is not clear but can not figure out the truth
>> sufficiently well to actually fix it.
>> I still don't know where the sources for the drawings are kept even
>> though I have asked several times.
>> 
>> It needs a team approach with someone who knows the truth and someone
>> who can write it down so that someone who did not write the code can
>> figure out what to do.
>> 
>> The biggest problem with programmers writing the user documentation is
>> that they are so caught up in the exceptions and special cases.
>> They spent a lot of time figuring out how to handle these oddball cases
>> that they feel that these triumphs must be on the front page.
>> They forget to explain the 95% case and lace the description of the main
>> flow with notes about these interesting exceptions.
>> 
>> That is not just true for Cloudstack but is a general problem with
>> documentation just because we are all human.
>> 
>> They also forget that the user does not want to be an expert in the
>> topic but wants to know enough to get the thing running.
>> The user has a lot of other problems and does not to become a developer
>> in order to get this to work.
>> 
>> In my case, I really need to get some internal applications (accounting,
>> SCM, issue tracking, Maven repo, 20 web sites etc.) running on virtual
>> machines in an environment that is easy to manage.  I 

Re: System VMs keeps failing to start

[cloudstackhelp]

Nothing's being spun up on the HVs. I'm using separate networks for each 
component (public, management, guest, storage). They all have a dedicated NIC 
each. On the HVs it seems like CS created its own cloud link local network but 
the link status is 


Thanks

Josh






On Thu, Mar 3, 2016 at 12:39 PM -0800, "Ahmad Emneina"  
wrote:





Hey Josh,

Are you seeing the VM's being spun up on the backend (hypervisor)? If so,
I'd imagine its a communication issue between the management server and the
public interface on the system vm. If you use VLAN tagging for your public
network, make sure the VLAN is trunked to your hypervisors in the cloud. I
recommend you stop the management service. Once restarted CloudStack will
try to recycle those vm's and spin them up again (so no worries should be
had there). If you're able to time it correctly, you can stop the
management service before the system vm's get shut down and log into
them... make sure the respective interfaces can reach their next hops...
that would be a good first step.

On Thu, Mar 3, 2016 at 12:32 PM,  wrote:

>
>
> Hi Ron and all,
>
>
>
> You mentioned that SVMs need to be started. I'm finding that my
> installation keeps looping the command "Creating system VMs (this may take
> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems to want
> to keep going on until it crashes.
>
>
>
> I'm thinking I should just kill the process but I'm worried I can't start
> the process again later. Is there a way to re-run this again later on?
>
>
>
> Why is it failing to start the VMs? Why is everything null? My networks
> are starting fine. Apologies for the lack of formating. Sending this via
> phone.
>
>
> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> private IP: N/A1004 Mar 2016 04:27:04
>
> Console proxy creation failure. zone: Public Cloud, error details:
> null1004 Mar 2016 04:27:04
>
> Secondary Storage Vm creation failure. zone: Public Cloud, error details:
> null1904 Mar 2016 04:27:00
>
> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> private IP: N/A1004 Mar 2016 04:26:34
>
> Console proxy creation failure. zone: Public Cloud, error details:
> null1004 Mar 2016 04:26:34
>
>
> Thanks in advance!
>
> Josh
>
>
>
>
>
>
> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" <
> rwhee...@artifact-software.com> wrote:
>
>
>
>
>
> I have been using Linux and the Internet since the mid 1990s.
> There are still 3 consecutively numbered C class registered to me for
> different clients back in the days when  c-class networks were given out
> from massive ranges of free numbers.
> I have set up small ISP operations for clients with multiple domains
> including web sites, e-mail servers, fileservers, etc.
> I have done this on SCO , Mandrake, CentOS 4 to 7.
>
> I should not have to struggle to figure out how set up Cloudstack in a
> small configuration with a few servers and a single public IP.
>
> The documentation on networking is jumbled about and so unclear that I
> can only point out why it is not clear but can not figure out the truth
> sufficiently well to actually fix it.
> I still don't know where the sources for the drawings are kept even
> though I have asked several times.
>
> It needs a team approach with someone who knows the truth and someone
> who can write it down so that someone who did not write the code can
> figure out what to do.
>
> The biggest problem with programmers writing the user documentation is
> that they are so caught up in the exceptions and special cases.
> They spent a lot of time figuring out how to handle these oddball cases
> that they feel that these triumphs must be on the front page.
> They forget to explain the 95% case and lace the description of the main
> flow with notes about these interesting exceptions.
>
> That is not just true for Cloudstack but is a general problem with
> documentation just because we are all human.
>
> They also forget that the user does not want to be an expert in the
> topic but wants to know enough to get the thing running.
> The user has a lot of other problems and does not to become a developer
> in order to get this to work.
>
> In my case, I really need to get some internal applications (accounting,
> SCM, issue tracking, Maven repo, 20 web sites etc.) running on virtual
> machines in an environment that is easy to manage.  I want to support
> clients who I am supporting as users of other systems - just want simple
> low volume services to support my supporting of their users.
>
> I only expect to have 4 servers, one NIC per machine to support 1
> transaction per second on a busy day
> I may get down to 2 servers  if Cloudstack works well and allows me to
> manage test servers and run docker nicely.
>
> I do not want to know enough to be the network administrator at Google
> or Amazon.
>
> This should not be hard to implement and from what I have seen it is not
> but the 

Re: System VMs keeps failing to start

[Ahmad Emneina]

Hey Josh,

Are you seeing the VM's being spun up on the backend (hypervisor)? If so,
I'd imagine its a communication issue between the management server and the
public interface on the system vm. If you use VLAN tagging for your public
network, make sure the VLAN is trunked to your hypervisors in the cloud. I
recommend you stop the management service. Once restarted CloudStack will
try to recycle those vm's and spin them up again (so no worries should be
had there). If you're able to time it correctly, you can stop the
management service before the system vm's get shut down and log into
them... make sure the respective interfaces can reach their next hops...
that would be a good first step.

On Thu, Mar 3, 2016 at 12:32 PM,  wrote:

>
>
> Hi Ron and all,
>
>
>
> You mentioned that SVMs need to be started. I'm finding that my
> installation keeps looping the command "Creating system VMs (this may take
> a while)". Right now, it's done its 70th cycle (s-70-VM) and seems to want
> to keep going on until it crashes.
>
>
>
> I'm thinking I should just kill the process but I'm worried I can't start
> the process again later. Is there a way to re-run this again later on?
>
>
>
> Why is it failing to start the VMs? Why is everything null? My networks
> are starting fine. Apologies for the lack of formating. Sending this via
> phone.
>
>
> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> private IP: N/A1004 Mar 2016 04:27:04
>
> Console proxy creation failure. zone: Public Cloud, error details:
> null1004 Mar 2016 04:27:04
>
> Secondary Storage Vm creation failure. zone: Public Cloud, error details:
> null1904 Mar 2016 04:27:00
>
> Console proxy up in zone: Public Cloud, proxy: v-72-VM, public IP: null,
> private IP: N/A1004 Mar 2016 04:26:34
>
> Console proxy creation failure. zone: Public Cloud, error details:
> null1004 Mar 2016 04:26:34
>
>
> Thanks in advance!
>
> Josh
>
>
>
>
>
>
> On Thu, Mar 3, 2016 at 11:02 AM -0800, "Ron Wheeler" <
> rwhee...@artifact-software.com> wrote:
>
>
>
>
>
> I have been using Linux and the Internet since the mid 1990s.
> There are still 3 consecutively numbered C class registered to me for
> different clients back in the days when  c-class networks were given out
> from massive ranges of free numbers.
> I have set up small ISP operations for clients with multiple domains
> including web sites, e-mail servers, fileservers, etc.
> I have done this on SCO , Mandrake, CentOS 4 to 7.
>
> I should not have to struggle to figure out how set up Cloudstack in a
> small configuration with a few servers and a single public IP.
>
> The documentation on networking is jumbled about and so unclear that I
> can only point out why it is not clear but can not figure out the truth
> sufficiently well to actually fix it.
> I still don't know where the sources for the drawings are kept even
> though I have asked several times.
>
> It needs a team approach with someone who knows the truth and someone
> who can write it down so that someone who did not write the code can
> figure out what to do.
>
> The biggest problem with programmers writing the user documentation is
> that they are so caught up in the exceptions and special cases.
> They spent a lot of time figuring out how to handle these oddball cases
> that they feel that these triumphs must be on the front page.
> They forget to explain the 95% case and lace the description of the main
> flow with notes about these interesting exceptions.
>
> That is not just true for Cloudstack but is a general problem with
> documentation just because we are all human.
>
> They also forget that the user does not want to be an expert in the
> topic but wants to know enough to get the thing running.
> The user has a lot of other problems and does not to become a developer
> in order to get this to work.
>
> In my case, I really need to get some internal applications (accounting,
> SCM, issue tracking, Maven repo, 20 web sites etc.) running on virtual
> machines in an environment that is easy to manage.  I want to support
> clients who I am supporting as users of other systems - just want simple
> low volume services to support my supporting of their users.
>
> I only expect to have 4 servers, one NIC per machine to support 1
> transaction per second on a busy day
> I may get down to 2 servers  if Cloudstack works well and allows me to
> manage test servers and run docker nicely.
>
> I do not want to know enough to be the network administrator at Google
> or Amazon.
>
> This should not be hard to implement and from what I have seen it is not
> but the networking docs are a major barrier to acceptance by mid-market
> companies - 300-1000 users with 1 or 2 System Admins who have to support
> all of the operations requirements and help developers and application
> support teams test and keep production systems running.
>
> Ron
>
>
> On 03/03/2016 6:22 AM, Mario Giammarco wrote:
> > Simon Weller