date:20170615

Re: ethNone problem with VPC

2017-06-15 Thread Simon Weller

Roman,


Do you have a traffic label setup for your public network?


Can you put the host in debug mode: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug


- Si




From: Roman Ledovskiy 
Sent: Wednesday, June 14, 2017 7:59 AM
To: users@cloudstack.apache.org
Subject: ethNone problem with VPC

Hi Everyone,

Does anyone encounter "ethNone" interfaces in VPC routers?

I'm running cloudstack 4.9.2.0 with KVM hypervisor and every time I create VPC 
router it fails to do proper setup, with errors:

Traceback (most recent call last):
  File "/opt/cloud/bin/configure.py", line 931, in main
config.address().process()
  File "/opt/cloud/bin/cs/CsAddress.py", line 103, in process
ip = CsIP(dev, self.config)
  File "/opt/cloud/bin/cs/CsAddress.py", line 256, in __init__
self.dnum = hex(int(dev[3:]))
ValueError: invalid literal for int() with base 10: 'None'


Only local link IP (eth0) is up.

/etc/cloudstack/ips.json inside my VPC routers always look like this (I guess 
this explains above):

{
"eth0": [
{
"add": true,
"broadcast": "169.254.255.255",
"cidr": "169.254.1.238/16",
"device": "eth0",
.
...
}
],
"ethNone": [
{
"add": true,
"broadcast": "X.Y.Z.255",
"cidr": "Y.Y.Z.43/24",
"device": "ethNone",
.

}
],
"id": "ips"
}


X.Y.Z.0/24 is my public network

If I manually edit /etc/cloudstack/ips.json (change ethNone to eth1) - then 
restart VPC router manually (via virsh) - everything works, interfaces/iptables 
are up..
Until next time I do anything with this VPC from GUI - then ethNone reappears 
and everything is broken again..

Appreciate if anyone has any thoughts.


Thanks
Roman

RE: Recreating SystemVM's

2017-06-15 Thread Jeremy Peterson

What type of networking are you using on the XenServers?
XenServers are connected with 6 nic's per host connected to separate 
nexus 5k switches
NIC 0 and NIC 1 are Bond 0+1 10Gb nics
NIC 2 and NIC 3 are Bond 2+3 10Gb nics
NIC 4 and NIC 5 are Bond 4+5 2Gb nics 
Cloudstack is running Advanced networking
Bond 0+1 is primary storage
Bond 2+3 is secondary storage 
Bond 4+5 is Management
What version of os does the ms run on?
CentOS release 6.9 (Final)
What are the systemvm templates defined in your env?

http://cloudstack.apt-get.eu/systemvm/4.5/systemvm64template-4.5-xen.vhd.bz2
What is the version of the systemvm.iso?
Successfully installed system VM template  to 
/secondary/template/tmpl/1/1/
I just reinstalled systemvm's from the above 4.5-xen.vhd
What is the capacity you have in your (test) environment?
This is a production enviroment and currently cloudstack shows the 
following.
Public IP Addresses 61%
VLAN 35%
Management IP Addresses 20%
Primary Storage 44%
CPU 21%
Memory 5%
Of cource Secondary Storage shows 0%
What is the host os version for the hypervisors?
XenServer 6.5 SP1
What is the management network range?
management.network.cidr 10.90.1.0/24
What are the other physical networks?
?? Not sure what more you need
What storage do you use?
Primary - ISCSI
Secondary - NFS
Is it reachable from the systemvm?
All of my CS management servers have internet access
Is the big bad internet reachable for your SSVM’s public interface?
My SSVM does not go online but yes the public network is the same as 
the VR public vlan and all instances behind VR's are connected to the internet 
at this time

Jeremy


-Original Message-
From: Daan Hoogland [mailto:daan.hoogl...@shapeblue.com] 
Sent: Thursday, June 15, 2017 9:34 AM
To: users@cloudstack.apache.org; S. Brüseke - proIO GmbH 
Subject: Re: Recreating SystemVM's

Your problem might be like what Swen says, Jeremy but also a wrong systemvm 
offering or a fault in your management network definition.
I am going to sum up some trivialities so bear with me;

What type of networking are you using on the XenServers?
What version of os does the ms run on?
What are the systemvm templates defined in your env?
What is the version of the systemvm.iso?
What is the capacity you have in your (test) environment?
What is the host os version for the hypervisors?
What is the management network range?
What are the other physical networks?
What storage do you use?
Is it reachable from the systemvm?
Is the big bad internet reachable for your SSVM’s public interface?

And of course,

How is the weather, where you are at?

I am not sure any of these question is going to lead you in the right direction 
but one of them should.

On 15/06/17 13:56, "S. Brüseke - proIO GmbH"  wrote:

I once did have some similar problem with my systemvms and my root cause 
was that in the global settings it referred to the wrong systemvm template. I 
am not sure if this helps you, but wanted to tell you.

Mit freundlichen Grüßen / With kind regards,

Swen

-Ursprüngliche Nachricht-
Von: Jeremy Peterson [mailto:jpeter...@acentek.net] 
Gesendet: Donnerstag, 15. Juni 2017 01:55
An: users@cloudstack.apache.org
Betreff: RE: Recreating SystemVM's

Hahaha.  The best response ever. 

I dug through these emails and someone had soft of the same log messages 
cannot attach network and blamed xenserver. Ok I'm cool with that but why oh 
why is it only system vms?

Jeremy

From: Imran Ahmed [im...@eaxiom.net]
Sent: Wednesday, June 14, 2017 6:22 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Yes,

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 9:59 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Is there anyone out there reading these messages?

Am I just not seeing responses?

Jeremy


-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 8:12 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

I opened an issue since this is still an issue.  CLOUDSTACK-9960

Jeremy

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Sunday, June 11, 2017 9:10 AM
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

Any other suggestions?

I am going to be scheduling to run XenServer updates.  But this all points 
back to CANNOT_ATTACH_NETWORk.

I've verified

Re: Recreating SystemVM's

2017-06-15 Thread Dag Sonstebo

Hi Jeremy,

as a general rule of thumb XenServer errors are pretty accurate – hence 
“HOST_CANNOT_ATTACH_NETWORK, OpaqueRef:65d0c844-bd70-81e9-4518-8809e1dc0ee7, 
OpaqueRef:0093ac3f-9f3a-37e1-9cdb-581398d27ba2” is where you should start. The 
reason you are hitting issues just with your system VMs is they are attached to 
the management interface (which user VMs aren’t).

So a couple of things to check:
1) Check the bonds on all your XS hosts – are they consistent? Do you have the 
same PIFs attached to each bond on each host?
2) Check networks on all your XS hosts – in particular check your “Host 
internal management network” and “cloud_link_local_network” networks (xe 
network-list params=all). Your network UUIDs should match between hosts for the 
same networks. About 250 years ago XenServer sometimes pulled a trick where 
networks appeared with different UUIDs on different hosts.
3) Check if those opaquerefs match up to anything – not sure if they will. 
Vague article on difference between UUIDs and opaquerefs here - 
https://wiki.xenproject.org/wiki/OpaqueRef_and_uuid_relationship_in_xapi 
4) You may also want to check if this is happening on all hosts by disabling 
(not maintence) all but one, check, enable, move on to next, rinse and repeat. 
If you find you have one dodgy host then evacuate it from the pool, ideally 
rebuild it and rejoin it to the pool.
5) Have you tried to force reconnect the hosts? 

These are all shots in the dark, but hopefully you’ll come across something not 
looking right.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 15/06/2017, 12:56, "S. Brüseke - proIO GmbH"  wrote:

I once did have some similar problem with my systemvms and my root cause 
was that in the global settings it referred to the wrong systemvm template. I 
am not sure if this helps you, but wanted to tell you.

Mit freundlichen Grüßen / With kind regards,

Swen

-Ursprüngliche Nachricht-
Von: Jeremy Peterson [mailto:jpeter...@acentek.net] 
Gesendet: Donnerstag, 15. Juni 2017 01:55
An: users@cloudstack.apache.org
Betreff: RE: Recreating SystemVM's

Hahaha.  The best response ever. 

I dug through these emails and someone had soft of the same log messages 
cannot attach network and blamed xenserver. Ok I'm cool with that but why oh 
why is it only system vms?

Jeremy

From: Imran Ahmed [im...@eaxiom.net]
Sent: Wednesday, June 14, 2017 6:22 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Yes,

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 9:59 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Is there anyone out there reading these messages?

Am I just not seeing responses?

Jeremy


-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 8:12 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

I opened an issue since this is still an issue.  CLOUDSTACK-9960

Jeremy

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Sunday, June 11, 2017 9:10 AM
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

Any other suggestions?

I am going to be scheduling to run XenServer updates.  But this all points 
back to CANNOT_ATTACH_NETWORk.

I've verified nothing is active on the Public IP space that those two VM's 
were living on.

Jeremy

From: Jeremy Peterson 
Sent: Friday, June 9, 2017 9:58 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

I see the vm's try to create on a host that I just removed from maintenance 
mode to install updates and here are the logs

I don't see anything that sticks out to me as a failure message.

Jun  9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'del', '169.254.0.0/16']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13068] ['ifconfig', 'xapi12', '169.254.0.1', 
'netmask', '255.255.0.0']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'add', '169.254.0.0/16', 
'dev', 'xapi12', 'src', '169.254.0.1']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'del', '169.254.0.0/16']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ifconfig', 'xapi12', '169.254.0.1', 
'netmask', '255.255.0.0']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'add', '169.254.0.0/16', 
'dev', 'xapi12', 'src',

Re: Recreating SystemVM's

2017-06-15 Thread Daan Hoogland

Your problem might be like what Swen says, Jeremy but also a wrong systemvm 
offering or a fault in your management network definition.
I am going to sum up some trivialities so bear with me;

What type of networking are you using on the XenServers?
What version of os does the ms run on?
What are the systemvm templates defined in your env?
What is the version of the systemvm.iso?
What is the capacity you have in your (test) environment?
What is the host os version for the hypervisors?
What is the management network range?
What are the other physical networks?
What storage do you use?
Is it reachable from the systemvm?
Is the big bad internet reachable for your SSVM’s public interface?

And of course,

How is the weather, where you are at?

I am not sure any of these question is going to lead you in the right direction 
but one of them should.

On 15/06/17 13:56, "S. Brüseke - proIO GmbH"  wrote:

I once did have some similar problem with my systemvms and my root cause 
was that in the global settings it referred to the wrong systemvm template. I 
am not sure if this helps you, but wanted to tell you.

Mit freundlichen Grüßen / With kind regards,

Swen

-Ursprüngliche Nachricht-
Von: Jeremy Peterson [mailto:jpeter...@acentek.net] 
Gesendet: Donnerstag, 15. Juni 2017 01:55
An: users@cloudstack.apache.org
Betreff: RE: Recreating SystemVM's

Hahaha.  The best response ever. 

I dug through these emails and someone had soft of the same log messages 
cannot attach network and blamed xenserver. Ok I'm cool with that but why oh 
why is it only system vms?

Jeremy

From: Imran Ahmed [im...@eaxiom.net]
Sent: Wednesday, June 14, 2017 6:22 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Yes,

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 9:59 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Is there anyone out there reading these messages?

Am I just not seeing responses?

Jeremy


-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 8:12 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

I opened an issue since this is still an issue.  CLOUDSTACK-9960

Jeremy

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Sunday, June 11, 2017 9:10 AM
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

Any other suggestions?

I am going to be scheduling to run XenServer updates.  But this all points 
back to CANNOT_ATTACH_NETWORk.

I've verified nothing is active on the Public IP space that those two VM's 
were living on.

Jeremy

From: Jeremy Peterson 
Sent: Friday, June 9, 2017 9:58 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

I see the vm's try to create on a host that I just removed from maintenance 
mode to install updates and here are the logs

I don't see anything that sticks out to me as a failure message.

Jun  9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'del', '169.254.0.0/16']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13068] ['ifconfig', 'xapi12', '169.254.0.1', 
'netmask', '255.255.0.0']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'add', '169.254.0.0/16', 
'dev', 'xapi12', 'src', '169.254.0.1']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'del', '169.254.0.0/16']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ifconfig', 'xapi12', '169.254.0.1', 
'netmask', '255.255.0.0']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'add', '169.254.0.0/16', 
'dev', 'xapi12', 'src', '169.254.0.1']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS


Jun  9 09:54:00 Xen3 SM: [13115] on-slave.multi: {'vgName':
'VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2', 'lvName1':
'VHD-68a7-6c40-4aa6-b88e-c798b6fdc04d', 'action1':
'deactivateNoRefcount', 'action2': 'cleanupLock', 'uuid2':
'68a7-6c40-4aa6-b88e-c798b6fdc04d', 'ns2':
'lvm-469b6dcd-8466-3d03-de0e-cc3983e1b6e2'}
Jun  9 09:54:00 Xen3 SM: [13115] LVMCache created for
VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2
Jun  9 09:54:00 Xen3 SM: [13115] on-slave.action 1: deactivateNoRefcount Jun
9 09:54:00 Xen3 SM: [13115] LVMCache: will initialize now Jun  9 09:54:00
Xen3 SM: [13115]

AW: Recreating SystemVM's

2017-06-15 Thread S . Brüseke - proIO GmbH

I once did have some similar problem with my systemvms and my root cause was 
that in the global settings it referred to the wrong systemvm template. I am 
not sure if this helps you, but wanted to tell you.

Mit freundlichen Grüßen / With kind regards,

Swen

-Ursprüngliche Nachricht-
Von: Jeremy Peterson [mailto:jpeter...@acentek.net] 
Gesendet: Donnerstag, 15. Juni 2017 01:55
An: users@cloudstack.apache.org
Betreff: RE: Recreating SystemVM's

Hahaha.  The best response ever. 

I dug through these emails and someone had soft of the same log messages cannot 
attach network and blamed xenserver. Ok I'm cool with that but why oh why is it 
only system vms?

Jeremy

From: Imran Ahmed [im...@eaxiom.net]
Sent: Wednesday, June 14, 2017 6:22 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Yes,

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 9:59 PM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

Is there anyone out there reading these messages?

Am I just not seeing responses?

Jeremy


-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Wednesday, June 14, 2017 8:12 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

I opened an issue since this is still an issue.  CLOUDSTACK-9960

Jeremy

-Original Message-
From: Jeremy Peterson [mailto:jpeter...@acentek.net]
Sent: Sunday, June 11, 2017 9:10 AM
To: users@cloudstack.apache.org
Subject: Re: Recreating SystemVM's

Any other suggestions?

I am going to be scheduling to run XenServer updates.  But this all points back 
to CANNOT_ATTACH_NETWORk.

I've verified nothing is active on the Public IP space that those two VM's were 
living on.

Jeremy

From: Jeremy Peterson 
Sent: Friday, June 9, 2017 9:58 AM
To: users@cloudstack.apache.org
Subject: RE: Recreating SystemVM's

I see the vm's try to create on a host that I just removed from maintenance 
mode to install updates and here are the logs

I don't see anything that sticks out to me as a failure message.

Jun  9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'del', '169.254.0.0/16']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13068] ['ifconfig', 'xapi12', '169.254.0.1', 
'netmask', '255.255.0.0']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13068] ['ip', 'route', 'add', '169.254.0.0/16', 
'dev', 'xapi12', 'src', '169.254.0.1']
Jun  9 09:53:54 Xen3 SM: [13068]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'del', '169.254.0.0/16']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ifconfig', 'xapi12', '169.254.0.1', 
'netmask', '255.255.0.0']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS
Jun  9 09:53:54 Xen3 SM: [13071] ['ip', 'route', 'add', '169.254.0.0/16', 
'dev', 'xapi12', 'src', '169.254.0.1']
Jun  9 09:53:54 Xen3 SM: [13071]   pread SUCCESS


Jun  9 09:54:00 Xen3 SM: [13115] on-slave.multi: {'vgName':
'VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2', 'lvName1':
'VHD-68a7-6c40-4aa6-b88e-c798b6fdc04d', 'action1':
'deactivateNoRefcount', 'action2': 'cleanupLock', 'uuid2':
'68a7-6c40-4aa6-b88e-c798b6fdc04d', 'ns2':
'lvm-469b6dcd-8466-3d03-de0e-cc3983e1b6e2'}
Jun  9 09:54:00 Xen3 SM: [13115] LVMCache created for
VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2
Jun  9 09:54:00 Xen3 SM: [13115] on-slave.action 1: deactivateNoRefcount Jun
9 09:54:00 Xen3 SM: [13115] LVMCache: will initialize now Jun  9 09:54:00
Xen3 SM: [13115] LVMCache: refreshing Jun  9 09:54:00 Xen3 SM: [13115] 
['/usr/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', 
'/dev/VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2']
Jun  9 09:54:00 Xen3 SM: [13115]   pread SUCCESS
Jun  9 09:54:00 Xen3 SM: [13115] ['/usr/sbin/lvchange', '-an',
'/dev/VG_XenStorage-469b6dcd-8466-3d03-de0e-cc3983e1b6e2/VHD-68a7-6c40-4
aa6-b88e-c798b6fdc04d']
Jun  9 09:54:00 Xen3 SM: [13115]   pread SUCCESS
Jun  9 09:54:00 Xen3 SM: [13115] ['/sbin/dmsetup', 'status',
'VG_XenStorage--469b6dcd--8466--3d03--de0e--cc3983e1b6e2-VHD--68a7--6c40
--4aa6--b88e--c798b6fdc04d']
Jun  9 09:54:00 Xen3 SM: [13115]   pread SUCCESS
Jun  9 09:54:00 Xen3 SM: [13115] on-slave.action 2: cleanupLock

Jun  9 09:54:16 Xen3 SM: [13230] ['ip', 'route', 'del', '169.254.0.0/16']
Jun  9 09:54:16 Xen3 SM: [13230]   pread SUCCESS
Jun  9 09:54:16 Xen3 SM: [13230] ['ifconfig', 'xapi12', '169.254.0.1', 
'netmask', '255.255.0.0']
Jun  9 09:54:16 Xen3 SM: [13230]   pread SUCCESS
Jun  9 09:54:16 Xen3 SM: [13230] ['ip', 'route', 'add', '169.254.0.0/16', 
'dev', 'xapi12', 'src', '169.254.0.1']
Jun  9 09:54:16 Xen3 SM: [13230]   pread SUCCESS
Jun  9 09:54:19 Xen3 updatempppathd: [15446] The garbage collection routine
returned: 0 Jun  9 09:54:23 Xen3 SM: [13277] ['ip', 'route', 'del',

Re: ethNone problem with VPC

RE: Recreating SystemVM's

Re: Recreating SystemVM's

Re: Recreating SystemVM's

AW: Recreating SystemVM's

5 matches

Site Navigation

Mail list logo

Footer information