[
https://issues.apache.org/jira/browse/CLOUDSTACK-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13657227#comment-13657227
]
John Burwell commented on CLOUDSTACK-2479:
------------------------------------------
I have been able to access the SSVM via the Xen terminal console.
The following are the open ports/services running on the SSVM:
root@s-9-VM:~# netstat -antp | grep LISTEN
netstat -antp | grep LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
1507/portmap
tcp 0 0 10.0.3.204:80 0.0.0.0:* LISTEN
2862/apache2
tcp 0 0 169.254.3.110:3922 0.0.0.0:* LISTEN
2676/sshd
tcp 0 0 0.0.0.0:51384 0.0.0.0:* LISTEN
-
tcp 0 0 10.0.3.204:443 0.0.0.0:* LISTEN
2862/apache2
tcp 0 0 0.0.0.0:53094 0.0.0.0:* LISTEN
1499/rpc.statd
It is important to note that the agent running on the VM, but not listening as
indicated through the following netstat and ps commands:
root@s-9-VM:/var/cache/cloud# ps aux | grep java
ps aux | grep java
root 2327 0.2 14.2 335600 36100 ? Sl 03:09 0:08 java -mx167m
-cp
./:./conf:apache-log4j-extras-1.1.jar:aws-java-sdk-1.3.21.1.jar:backport-util-concurrent-3.1.jar:bcprov-jdk16-1.46.jar:cglib-nodep-2.2.2.jar:cloud-agent-4.1.0-SNAPSHOT.jar:cloud-api-4.1.0-SNAPSHOT.jar:cloud-console-proxy-4.1.0-SNAPSHOT.jar:cloud-core-4.1.0-SNAPSHOT.jar:cloud-utils-4.1.0-SNAPSHOT.jar:commons-codec-1.6.jar:commons-collections-3.2.1.jar:commons-configuration-1.8.jar:commons-dbcp-1.4.jar:commons-discovery-0.5.jar:commons-httpclient-3.1.jar:commons-lang-2.6.jar:commons-logging-1.0.4.jar:commons-pool-1.6.jar:ehcache-1.5.0.jar:ejb-api-3.0.jar:gson-1.7.1.jar:httpclient-4.1.jar:httpcore-4.1.jar:jackson-core-asl-1.8.9.jar:jackson-mapper-asl-1.8.9.jar:jasypt-1.9.0.jar:javax.persistence-2.0.0.jar:jsch-0.1.42.jar:jsr107cache-1.0.jar:log4j-1.2.16.jar:trilead-ssh2-build213-svnkit-1.3-patch.jar
com.cloud.agent.AgentShell root=LABEL console=tty0 xencons=ttyS0,115200
console=hvc0 console=hvc0 template=domP type=secstorage host=192.168.56.15
port=8250 name=s-9-VM zone=1 pod=1 guid=s-9-VM
resource=com.cloud.storage.resource.NfsSecondaryStorageResource
instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
eth2ip=10.0.3.204 eth2mask=255.255.255.0 gateway=10.0.3.2 eth0ip=169.254.3.110
eth0mask=255.255.0.0 eth1ip=192.168.56.112 eth1mask=255.255.255.0
mgmtcidr=192.168.56.0/24 localgw=192.168.56.1 private.network.device=eth1
eth3ip=192.168.56.145 eth3mask=255.255.255.0 storageip=192.168.56.145
storagenetmask=255.255.255.0 storagegateway=192.168.56.1
internaldns1=192.168.56.15 dns1=8.8.8.8
root 3223 0.0 0.2 3308 724 hvc0 S+ 04:03 0:00 grep java
root@s-9-VM:/var/cache/cloud#
root@s-9-VM:/var/cache/cloud# netstat -antp | grep LISTEN | grep 2327
netstat -antp | grep LISTEN | grep 2327
The following are the network interfaces defined in the VM:
root@s-9-VM:~# ifconfig
ifconfig
eth0 Link encap:Ethernet HWaddr 0e:00:a9:fe:03:6e
inet addr:169.254.3.110 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2072 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:91680 (89.5 KiB) TX bytes:656 (656.0 B)
Interrupt:8
eth1 Link encap:Ethernet HWaddr 06:49:aa:00:00:0d
inet addr:192.168.56.112 Bcast:192.168.56.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4737 errors:0 dropped:0 overruns:0 frame:0
TX packets:4783 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:776071 (757.8 KiB) TX bytes:699394 (683.0 KiB)
Interrupt:9
eth2 Link encap:Ethernet HWaddr 06:99:4e:00:00:37
inet addr:10.0.3.204 Bcast:10.0.3.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1008 errors:0 dropped:0 overruns:0 frame:0
TX packets:877 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:334683 (326.8 KiB) TX bytes:87721 (85.6 KiB)
Interrupt:11
eth3 Link encap:Ethernet HWaddr 06:fc:56:00:00:2e
inet addr:192.168.56.145 Bcast:192.168.56.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:165 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22028 (21.5 KiB) TX bytes:0 (0.0 B)
Interrupt:12
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:64 errors:0 dropped:0 overruns:0 frame:0
TX packets:64 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5292 (5.1 KiB) TX bytes:5292 (5.1 KiB)
Interestingly, the public and link-local addresses match the IP addresses
reported by the Management Server. However, the Manager Server reports that
the private IP address is 192.168.56.112, but the VM is configured with
192.168.56.145. Even with a properly functioning networking, this difference
would prevent the Management Server from connecting to the Agent on the SSVM.
The following are the contents of /var/cache/cloud/cmdline:
root=LABEL=ROOT ro console=tty0 xencons=ttyS0,115200 console=hvc0 quiet --
quiet console=hvc0 template=domP type=secstorage host=192.168.56.15 port=8250
name=s-9-VM zone=1 pod=1 guid=s-9-VM
resource=com.cloud.storage.resource.NfsSecondaryStorageResource
instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
eth2ip=10.0.3.204 eth2mask=255.255.255.0 gateway=10.0.3.2 eth0ip=169.254.3.110
eth0mask=255.255.0.0 eth1ip=192.168.56.112 eth1mask=255.255.255.0
mgmtcidr=192.168.56.0/24 localgw=192.168.56.1 private.network.device=eth1
eth3ip=192.168.56.145 eth3mask=255.255.255.0 storageip=192.168.56.145
storagenetmask=255.255.255.0 storagegateway=192.168.56.1
internaldns1=192.168.56.15 dns1=8.8.8.8
Finally, the following are the contents of /var/log/cloud.log:
root@s-9-VM:/var/log# less cloud.log
less cloud.log
Tue May 14 03:08:55 UTC 2013 Executing cloud-early-config
Tue May 14 03:08:55 UTC 2013 Detected that we are running inside xen-domU guest
Tue May 14 03:08:56 UTC 2013 Patching scripts
oldmd5=0286f69a841f2e7290067a4b948fcf4d newmd5=325eeb4cbf3466d2570d73d7a7e1af2e
Tue May 14 03:08:56 UTC 2013 Patching cloud service
Tue May 14 03:08:58 UTC 2013 Rebooting system since we patched init scripts
Tue May 14 03:09:17 UTC 2013 Executing cloud-early-config
Tue May 14 03:09:17 UTC 2013 Detected that we are running inside xen-domU guest
Tue May 14 03:09:17 UTC 2013 Patching cloud service
Tue May 14 03:09:21 UTC 2013 Setting up secondary storage system vm
Tue May 14 03:09:27 UTC 2013 Successfully setup storage network with
STORAGE_IP:192.168.56.145, STORAGE_NETMASK:255.255.255.0, STORAGE_CIDR:
Tue May 14 03:09:27 UTC 2013 Setting up apache web server
Tue May 14 03:09:27 UTC 2013 cloud: disable rp_filter
Tue May 14 03:09:27 UTC 2013 disable rpfilter
Tue May 14 03:09:27 UTC 2013 cloud: enable_fwding = 0
Tue May 14 03:09:27 UTC 2013 enable_fwding = 0
Tue May 14 03:09:27 UTC 2013 Enable service haproxy = 0
Tue May 14 03:09:27 UTC 2013 Enable service dnsmasq = 0
Tue May 14 03:09:27 UTC 2013 Enable service cloud-passwd-srvr = 0
Tue May 14 03:09:27 UTC 2013 Enable service cloud = 1
> No Connectivity Xen SSVM
> -------------------------
>
> Key: CLOUDSTACK-2479
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2479
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Storage Controller
> Affects Versions: 4.1.0
> Environment: devcloud
> Reporter: John Burwell
> Priority: Blocker
>
> On devcloud/Xen, the SSVM is being successfully created (verified through
> both CloudStack and Xen) and it responds to pings at its link-local address.
> However, the management server is unable to connect to the agent and attempts
> to SSH to the the link-local address (i.e. ssh -p 3392 -i ~/.ssh/id_rsa.cloud
> <link-local address>) fail. An nmap of the public interface reports that
> ports 80 and 443 are open while an namp reports that no ports are open on the
> private or link-local interfaces.
> The zone is configured with basic networking, and the following networks:
> * Management
> - Gateway: 192.168.56.1
> - IP Range: 192.168.56.100-192.168.56.149
> - Xen Traffic Label: xenbr0
> * Guest
> - Gateway: 10.0.3.2
> - IP Range: 10.0.3.200-10.0.3.220
> - Xen Traffic Label: xenbr1
> The following is the Xen pif list from the host depicting the mapping of the
> traffic labels to physical interfaces:
> root@zone1:~# xe pif-list
> uuid ( RO) : 207413c9-5058-7a40-6c96-2dab21057f30
> device ( RO): eth1
> currently-attached ( RO): true
> VLAN ( RO): -1
> network-uuid ( RO): 1679ddb1-5a21-b827-ab07-c16275d5ce72
> uuid ( RO) : c0274787-e768-506f-3191-f0ac17b0c72b
> device ( RO): eth0
> currently-attached ( RO): true
> VLAN ( RO): -1
> network-uuid ( RO): 8ee927b1-a35d-ac10-4471-d7a6a475839a
> The following is a listing of the Xen vifs for the SSVM:
> root@zone1:~# xe vm-vif-list vm=c72614c4-dc28-b3c4-3400-1a34db81f187
> uuid ( RO) : 741fe852-b9e4-4b49-cfa9-c7b09fc8cbf6
> vm-name-label ( RO): s-8-VM
> device ( RO): 3
> MAC ( RO): 06:74:62:00:00:2f
> network-uuid ( RO): 8ee927b1-a35d-ac10-4471-d7a6a475839a
> network-name-label ( RO): xenbr0
> uuid ( RO) : 8460116b-8f25-97c1-0178-2a25282f3359
> vm-name-label ( RO): s-8-VM
> device ( RO): 1
> MAC ( RO): 06:12:3a:00:00:07
> network-uuid ( RO): 8ee927b1-a35d-ac10-4471-d7a6a475839a
> network-name-label ( RO): xenbr0
> uuid ( RO) : 342077d6-44cc-2ee9-a733-1cac9138bf5c
> vm-name-label ( RO): s-8-VM
> device ( RO): 0
> MAC ( RO): 0e:00:a9:fe:02:a5
> network-uuid ( RO): c67782a0-ea0e-8d50-ad09-9f3fff8a4ca1
> network-name-label ( RO): cloud_link_local_network
> uuid ( RO) : 9880bf5a-a66a-1d3f-65b5-493b15a8dec6
> vm-name-label ( RO): s-8-VM
> device ( RO): 2
> MAC ( RO): 06:83:9a:00:00:35
> network-uuid ( RO): 1679ddb1-5a21-b827-ab07-c16275d5ce72
> network-name-label ( RO): xenbr1
> Finally, the following is the ifconfig for the host itself:
> root@zone1:~# ifconfig
> eth0 Link encap:Ethernet HWaddr 08:00:27:7e:74:9c
> inet6 addr: fe80::a00:27ff:fe7e:749c/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:1418 errors:0 dropped:0 overruns:0 frame:0
> TX packets:2853 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:207856 (207.8 KB) TX bytes:187744 (187.7 KB)
> eth1 Link encap:Ethernet HWaddr 08:00:27:df:00:00
> inet6 addr: fe80::a00:27ff:fedf:0/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:1081755 errors:0 dropped:0 overruns:0 frame:0
> TX packets:2012200 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:409565427 (409.5 MB) TX bytes:2742512504 (2.7 GB)
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:1021408 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1021408 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:2476556022 (2.4 GB) TX bytes:2476556022 (2.4 GB)
> vif17.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
> RX packets:14 errors:0 dropped:0 overruns:0 frame:0
> TX packets:2065 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:32
> RX bytes:764 (764.0 B) TX bytes:120310 (120.3 KB)
> vif17.1 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
> RX packets:2689 errors:0 dropped:0 overruns:0 frame:0
> TX packets:4698 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:32
> RX bytes:319908 (319.9 KB) TX bytes:547552 (547.5 KB)
> vif17.2 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
> RX packets:52 errors:0 dropped:0 overruns:0 frame:0
> TX packets:2099 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:32
> RX bytes:3592 (3.5 KB) TX bytes:124732 (124.7 KB)
> vif17.3 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:32
> RX bytes:0 (0.0 B) TX bytes:21093 (21.0 KB)
> vif4.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
> RX packets:6 errors:0 dropped:0 overruns:0 frame:0
> TX packets:312 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:32
> RX bytes:292 (292.0 B) TX bytes:18412 (18.4 KB)
> vif4.1 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
> RX packets:1895 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1347 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:32
> RX bytes:102212 (102.2 KB) TX bytes:156194 (156.1 KB)
> vif4.2 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
> UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
> RX packets:27 errors:0 dropped:0 overruns:0 frame:0
> TX packets:948 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:32
> RX bytes:756 (756.0 B) TX bytes:132252 (132.2 KB)
> xapi0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
> inet addr:169.254.0.1 Bcast:169.254.255.255 Mask:255.255.0.0
> inet6 addr: fe80::a087:27ff:feb7:9da7/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:83 errors:0 dropped:0 overruns:0 frame:0
> TX packets:4319 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:4440 (4.4 KB) TX bytes:253366 (253.3 KB)
> xenbr0 Link encap:Ethernet HWaddr 08:00:27:7e:74:9c
> inet addr:192.168.56.15 Bcast:192.168.56.255 Mask:255.255.255.0
> inet6 addr: fe80::a00:27ff:fe7e:749c/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:158084 errors:0 dropped:0 overruns:0 frame:0
> TX packets:151863 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:10473926 (10.4 MB) TX bytes:6337801241 (6.3 GB)
> xenbr1 Link encap:Ethernet HWaddr 08:00:27:df:00:00
> inet addr:10.0.3.15 Bcast:10.0.3.255 Mask:255.255.255.0
> inet6 addr: fe80::a00:27ff:fedf:0/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:529173 errors:0 dropped:0 overruns:0 frame:0
> TX packets:286709 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:368500622 (368.5 MB) TX bytes:45967489 (45.9 MB)
> I encountered this issue when I pulled a fresh copy of the 4.1 branch, and
> built it using the following command:
> mvn -P developer,systemvm clean install
> I also cleared my database, and rebuilt using the following command:
> mvn -P developer -pl developer -Ddeploydb
> Finally, I defined using a known-good Marvin configuration for which JIRA
> provides no way to attach. However, I would happy to provide it upon request.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
