[
https://issues.apache.org/jira/browse/CLOUDSTACK-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608814#comment-13608814
]
Jayapal Reddy commented on CLOUDSTACK-1565:
-------------------------------------------
Here is my analysis from the cloud-early-config debug.
When we are configuring the default route at that time the eth2 interface is
not configured the ip address on the eth2.
configuring default route on interface with out ip address will fail.
The ip on the eth2 interface got configured after the execution of the default
route command.
There is timing issue.
Please see the below debug logs.
Note: add the below lines in script to redirect set -x output to file
set -x
exec 3>&0 4>&1 >/var/test.txt 2>&1
+ setup_interface 2 10.147.52.142 255.255.255.0 10.147.52.1
+ local intfnum=2
+ local ip=10.147.52.142
+ local mask=255.255.255.0
+ local gw=10.147.52.1
+ local force=
+ local intf=eth2
+ local bootproto=static
+ ls /run/network
ls: cannot access /run/network: No such file or directory
+ grep mountkern
+ insserv -s
S:01:S:mountkernfs.sh
+ '[' static == dhcp ']'
+ '[' 10.147.52.142 '!=' 0.0.0.0 -a 10.147.52.142 '!=' '' -o '' == force ']'
+ echo 'iface eth2 inet static'
+ '[' static == static ']'
+ echo ' address 10.147.52.142 '
+ echo ' netmask 255.255.255.0'
+ '[' 10.147.52.142 == 0.0.0.0 -o 10.147.52.142 == '' ']'
+ ls /run/network
ls: cannot access /run/network: No such file or directory
+ '[' '' == force ']'
+ ifdown eth2
ifdown: failed to open statefile /run/network/ifstate: No such file or directory
+ '[' '' '!=' 1 -o 2 '!=' 2 ']'
+ ifup eth2
ifup: failed to open statefile /run/network/ifstate: No such file or directory
+ ifconfig eth2
eth2 Link encap:Ethernet HWaddr 06:d3:78:00:00:34
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
+ cat /etc/network/interfaces
auto lo eth0 eth1 eth2
iface lo inet loopback
iface eth0 inet static
address 169.254.3.247
netmask 255.255.0.0
iface eth1 inet static
address 10.147.41.230
netmask 255.255.254.0
iface eth2 inet static
address 10.147.52.142
netmask 255.255.255.0
+ echo s-1-VM
+ echo AVAHI_DAEMON_DETECT_LOCAL=0
+ hostname s-1-VM
+ sed -i -e '/^nameserver.*$/d' /etc/resolv.conf
+ sed -i -e '/^nameserver.*$/d' /etc/dnsmasq-resolv.conf
+ '[' -n 10.103.128.16 ']'
+ echo 'nameserver 10.103.128.16'
+ echo 'nameserver 10.103.128.16'
+ '[' -n '' ']'
+ '[' -n $'10.103.128.16\r' ']'
+ echo 'nameserver 10.103.128.16
'
+ echo 'nameserver 10.103.128.16
'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n 10.146.0.0/24 -a -n 10.147.40.1 ']'
+ ip route add 10.146.0.0/24 via 10.147.40.1 dev eth1
RTNETLINK answers: No such process
+ ip route delete default
RTNETLINK answers: No such process
+ '[' '' '!=' 1 ']'
+ '[' -z eth2 ']'
+ echo 'interface is eth2 gw 10.147.52.1'
interface is eth2 gw 10.147.52.1
+ ifconfig eth2
eth2 Link encap:Ethernet HWaddr 06:d3:78:00:00:34
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
+ ip route add default via 10.147.52.1 dev eth2
RTNETLINK answers: No such process
+ sleep 5
+ sleep 3
+ ping -n -c 3 10.147.52.1
PING 10.147.52.1 (10.147.52.1): 56 data bytes
64 bytes from 10.147.52.1: icmp_seq=0 ttl=64 time=3.094 ms
64 bytes from 10.147.52.1: icmp_seq=1 ttl=64 time=1.229 ms
64 bytes from 10.147.52.1: icmp_seq=2 ttl=64 time=1.220 ms
--- 10.147.52.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.220/1.848/3.094/0.881 ms
+ pkill ping
+ '[' -n 10.146.0.0/24 -a -n 10.147.40.1 ']'
+ ifconfig eth2
eth2 Link encap:Ethernet HWaddr 06:d3:78:00:00:34
inet addr:10.147.52.142 Bcast:10.147.52.255 Mask:255.255.255.0
inet6 addr: fe80::4d3:78ff:fe00:34/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:486 (486.0 B) TX bytes:736 (736.0 B)
===========================
Fix:
Adding sleep 3 before adding route causing interface setup with the ip address
and route command is sucess.
> Used Master Branch System VM Template: Default Route on the System VMs
> (SSVM,CPVM and VR) is missing
> ----------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-1565
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1565
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: Chandan Purushothama
> Assignee: Jayapal Reddy
> Priority: Blocker
> Fix For: 4.2.0
>
>
> ================
> Steps to Reproduce:
> ================
> 1. Deploy an Advanced Zone Setup with XenServer 6.0.2 Host
> 2. Use the System Template at
> http://jenkins.cloudstack.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvmtemplate-2013-03-05-master-xen.vhd.bz2
> during the Setup deployment.
> 3. Observe that the default template download fails with "Network is
> Unreachable" Status.
> ===========
> Observations:
> ===========
> On the SSVM:
> *Observe the missing default route*
> root@s-1-NEWXENTEMPL:~# ip route
> 10.223.58.0/26 dev eth1 proto kernel scope link src 10.223.58.7
> 10.223.58.0/26 dev eth3 proto kernel scope link src 10.223.58.8
> 10.223.110.232 via 10.223.58.1 dev eth1
> 10.223.110.253 via 10.223.58.1 dev eth1
> 10.223.110.254 via 10.223.58.1 dev eth1
> 10.223.131.170 via 10.223.58.1 dev eth1
> 10.223.138.0/26 dev eth2 proto kernel scope link src 10.223.138.21
> 169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.1.101
> root@s-1-NEWXENTEMPL:~#
> root@s-1-NEWXENTEMPL:~# cat /proc/cmdline
> root=UUID=d18a6135-d041-46ea-a137-b8259269c02e ro -- quiet
> console=hvc0%template=domP%type=secstorage%host=10.223.131.170%port=8250%name=s-1-NEWXENTEMPL%zone=1%pod=1%guid=s-1-NEWXENTEMPL%resource=com.cloud.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=true%role=templateProcessor%mtu=1500%eth2ip=10.223.138.21%eth2mask=255.255.255.192%gateway=10.223.138.1%public.network.device=eth2%eth0ip=169.254.1.101%eth0mask=255.255.0.0%eth1ip=10.223.58.7%eth1mask=255.255.255.192%mgmtcidr=10.223.131.0/24%localgw=10.223.58.1%private.network.device=eth1%eth3ip=10.223.58.8%eth3mask=255.255.255.192%storageip=10.223.58.8%storagenetmask=255.255.255.192%storagegateway=10.223.58.1%internaldns1=10.223.110.254%internaldns2=10.223.110.253%dns1=8.8.8.8%dns2=8.8.4.4
> root@s-1-NEWXENTEMPL:~#
> On the management Server Log:
> 2013-03-06 17:37:01,252 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) Telling the ssvm to load the NfsSecondaryStorageResource
> 2013-03-06 17:37:01,253 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) Boot Args for VM[SecondaryStorageVm|s-1-NEWXENTEMPL]:
> template=domP type=secstorage host=10.223.131.170 port=8250
> name=s-1-NEWXENTEMPL zone=1 pod=1 guid=s-1-NEWXENTEMPL
> resource=com.cloud.storage.resource.NfsSecondaryStorageResource
> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
> eth2ip=10.223.138.21 eth2mask=255.255.255.192 gateway=10.223.138.1
> public.network.device=eth2 eth0ip=169.254.1.101 eth0mask=255.255.0.0
> eth1ip=10.223.58.7 eth1mask=255.255.255.192 mgmtcidr=10.223.131.0/24
> localgw=10.223.58.1 private.network.device=eth1 eth3ip=10.223.58.8
> eth3mask=255.255.255.192 storageip=10.223.58.8 storagenetmask=255.255.255.192
> storagegateway=10.223.58.1 internaldns1=10.223.110.254
> internaldns2=10.223.110.253 dns1=8.8.8.8 dns2=8.8.4.4
> 2013-03-06 17:37:01,275 DEBUG [agent.transport.Request] (secstorage-1:null)
> Seq 1-2123956242: Sending { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1,
> Flags: 100111,
> [{"StartCommand":{"vm":{"id":1,"name":"s-1-NEWXENTEMPL","bootloader":"PyGrub","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
> GNU/Linux 6(64-bit)","bootArgs":" template=domP type=secstorage
> host=10.223.131.170 port=8250 name=s-1-NEWXENTEMPL zone=1 pod=1
> guid=s-1-NEWXENTEMPL
> resource=com.cloud.storage.resource.NfsSecondaryStorageResource
> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
> eth2ip=10.223.138.21 eth2mask=255.255.255.192 gateway=10.223.138.1
> public.network.device=eth2 eth0ip=169.254.1.101 eth0mask=255.255.0.0
> eth1ip=10.223.58.7 eth1mask=255.255.255.192 mgmtcidr=10.223.131.0/24
> localgw=10.223.58.1 private.network.device=eth1 eth3ip=10.223.58.8
> eth3mask=255.255.255.192 storageip=10.223.58.8 storagenetmask=255.255.255.192
> storagegateway=10.223.58.1 internaldns1=10.223.110.254
> internaldns2=10.223.110.253 dns1=8.8.8.8
> dns2=8.8.4.4","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"vncPassword":"72d0e0bb4c42cc9c","params":{},"uuid":"fd705739-d518-48cf-8680-3e16f02f7253","disks":[{"id":1,"name":"ROOT-1","mountPoint":"/export/home/chandan/asfmas-131-170/primary","path":"bf5a3843-c603-4d94-bdad-fe2ebbef90bd","size":2097152000,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"b06ada48-12e1-343e-b429-6e8e516397ba","deviceId":0}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"e9026157-f1d8-4486-a89e-74b9dd21f624","ip":"10.223.138.21","netmask":"255.255.255.192","gateway":"10.223.138.1","mac":"06:4b:2c:00:00:1b","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1380","isolationUri":"vlan://1380","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"f3d9931c-395b-450d-b4e2-8968ea936642","ip":"169.254.1.101","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:65","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"05478ed5-8cd6-41fe-be55-afe0bafa3ba6","ip":"10.223.58.7","netmask":"255.255.255.192","gateway":"10.223.58.1","mac":"06:17:da:00:00:04","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false},{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"uuid":"27ff90a9-82a5-4276-a9c9-a39d396b8de1","ip":"10.223.58.8","netmask":"255.255.255.192","gateway":"10.223.58.1","mac":"06:15:e0:00:00:05","broadcastType":"Native","type":"Storage","isSecurityGroupEnabled":false}]},"hostIp":"10.223.58.2","wait":0}},{"check.CheckSshCommand":{"ip":"169.254.1.101","port":3922,"interval":6,"retries":100,"name":"s-1-NEWXENTEMPL","wait":0}}]
> }
> 2013-03-06 17:37:01,283 DEBUG [agent.transport.Request] (secstorage-1:null)
> Seq 1-2123956242: Executing: { Cmd , MgmtId: 7200344900649, via: 1, Ver: v1,
> Flags: 100111,
> [{"StartCommand":{"vm":{"id":1,"name":"s-1-NEWXENTEMPL","bootloader":"PyGrub","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
> GNU/Linux 6(64-bit)","bootArgs":" template=domP type=secstorage
> host=10.223.131.170 port=8250 name=s-1-NEWXENTEMPL zone=1 pod=1
> guid=s-1-NEWXENTEMPL
> resource=com.cloud.storage.resource.NfsSecondaryStorageResource
> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
> eth2ip=10.223.138.21 eth2mask=255.255.255.192 gateway=10.223.138.1
> public.network.device=eth2 eth0ip=169.254.1.101 eth0mask=255.255.0.0
> eth1ip=10.223.58.7 eth1mask=255.255.255.192 mgmtcidr=10.223.131.0/24
> localgw=10.223.58.1 private.network.device=eth1 eth3ip=10.223.58.8
> eth3mask=255.255.255.192 storageip=10.223.58.8 storagenetmask=255.255.255.192
> storagegateway=10.223.58.1 internaldns1=10.223.110.254
> internaldns2=10.223.110.253 dns1=8.8.8.8
> dns2=8.8.4.4","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"vncPassword":"72d0e0bb4c42cc9c","params":{},"uuid":"fd705739-d518-48cf-8680-3e16f02f7253","disks":[{"id":1,"name":"ROOT-1","mountPoint":"/export/home/chandan/asfmas-131-170/primary","path":"bf5a3843-c603-4d94-bdad-fe2ebbef90bd","size":2097152000,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"b06ada48-12e1-343e-b429-6e8e516397ba","deviceId":0}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"e9026157-f1d8-4486-a89e-74b9dd21f624","ip":"10.223.138.21","netmask":"255.255.255.192","gateway":"10.223.138.1","mac":"06:4b:2c:00:00:1b","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1380","isolationUri":"vlan://1380","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"f3d9931c-395b-450d-b4e2-8968ea936642","ip":"169.254.1.101","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:65","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"05478ed5-8cd6-41fe-be55-afe0bafa3ba6","ip":"10.223.58.7","netmask":"255.255.255.192","gateway":"10.223.58.1","mac":"06:17:da:00:00:04","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false},{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"uuid":"27ff90a9-82a5-4276-a9c9-a39d396b8de1","ip":"10.223.58.8","netmask":"255.255.255.192","gateway":"10.223.58.1","mac":"06:15:e0:00:00:05","broadcastType":"Native","type":"Storage","isSecurityGroupEnabled":false}]},"hostIp":"10.223.58.2","wait":0}},{"check.CheckSshCommand":{"ip":"169.254.1.101","port":3922,"interval":6,"retries":100,"name":"s-1-NEWXENTEMPL","wait":0}}]
> }
> 2013-03-06 17:37:01,283 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-2:null) Seq 1-2123956242: Executing request
> 2013-03-06 17:37:01,400 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-2:null) 1. The VM s-1-NEWXENTEMPL is in Starting state.
> 2013-03-06 17:37:01,563 DEBUG [xen.resource.XenServer56FP1Resource]
> (DirectAgent-2:null) Created VM e168bed7-3445-14b2-288b-fd20493c2482 for
> s-1-NEWXENTEMPL
> 2013-03-06 17:37:01,601 DEBUG [xen.resource.XenServer56FP1Resource]
> (DirectAgent-2:null) PV args are -- quiet
> console=hvc0%template=domP%type=secstorage%host=10.223.131.170%port=8250%name=s-1-NEWXENTEMPL%zone=1%pod=1%guid=s-1-NEWXENTEMPL%resource=com.cloud.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=true%role=templateProcessor%mtu=1500%eth2ip=10.223.138.21%eth2mask=255.255.255.192%gateway=10.223.138.1%public.network.device=eth2%eth0ip=169.254.1.101%eth0mask=255.255.0.0%eth1ip=10.223.58.7%eth1mask=255.255.255.192%mgmtcidr=10.223.131.0/24%localgw=10.223.58.1%private.network.device=eth1%eth3ip=10.223.58.8%eth3mask=255.255.255.192%storageip=10.223.58.8%storagenetmask=255.255.255.192%storagegateway=10.223.58.1%internaldns1=10.223.110.254%internaldns2=10.223.110.253%dns1=8.8.8.8%dns2=8.8.4.4
> 2013-03-06 17:37:01,711 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-2:null) VBD 95f97e8f-4ce8-76e6-5536-222a8eecc45c created for
> Vol[1|ROOT|bf5a3843-c603-4d94-bdad-fe2ebbef90bd|2097152000]
> 2013-03-06 17:37:01,797 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-2:null) Creating VIF for s-1-NEWXENTEMPL on nic
> [Nic:Public-10.223.138.21-vlan://1380]
> I see the gateway Information in the mgmt log and on the cmdline info on the
> SSVM. I am not sure why the default route didnt get programmed on the SSVM,
> ===========
> Work Around:
> ===========
> Manually add the missing default route on the SSVM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
