Hi. I upgraded from 2.2.14 to 4.0.0 and now, the public network is not working.
2.2.14 setup Single Management Server (centos 6) 2 Hypervisors (KVM) (centos 6) Network Type: Advanced Security Groups Enabled: Yes Primary and Secondary Storage: NFS Public Network: VLAN 110 Public and Guest on: cloudbr0 Private: cloudbr1 Upgrade Instructions followed from this URL: http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.0-incubating/html-single/release-notes/index.html#upgrade-from-2.2.x-to-4.0 On step 20: "nohup cloud-sysvmadm -d 127.0.0.1 -u cloud -p password -c -r > sysvm.log 2>&1 &" I had to change this command to : "nohup cloud-sysvmadm -d 127.0.0.1 -u cloud -p password -s -r > sysvm.log 2>&1 &" In the cloud-systemadm command, there isno -c but an -s doing the same. " -s - stop then start all running SSVMs and Console Proxies " My sysvm.log output: ------------------------- nohup: ignoring input Stopping and starting 1 secondary storage vm(s)... Done stopping and starting secondary storage vm(s) Stopping and starting 1 console proxy vm(s)... Done stopping and starting console proxy vm(s). Stopping and starting 1 running routing vm(s)... Done restarting router(s). ------------------------- Different from the Document: My SystemVMs have this version on /etc/cloudstack-release: Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012 While the upgrade document says it should show: Cloudstack Release 4.0.0-incubating Mon Oct 9 15:10:04 PST 2012 Of the 2 hypervisors, all VMS are in a single hypervisor node01, and after the upgrade, this hypervisor has not been rebooted. The other hypervisor, which does not have any VM, has been rebooted. After the upgrade: CloudStack management interface is upgraded to the new ones. All VMS were running and reachable via their public IP Network Offering: Guest Network: Name, Account, VLAN and CIDR is blank, Type has shared. Network Offering: DefaultSharedNetworkOfferingWithSGService Security Groups: All 2.2.14 settings of various Security Groups are present. Zone: Network Type: Advanced Local Storage Enabled: NO Status of Virtual Router and System VMS - Running I can ssh to all the system VM's using the local link ip address Ping to private IPs of the system VM's work. ( from management server, hypervisors and from each other ) Ping to the PUBLIC IPs of the system VM's do not work from anywhere. I rebooted 2 VMs.. Their status shows running, but they are not reachable anymore( via public IP) I migrated these 2 VMS that were rebooted to the 2nd hypervisor. They migrated OK, but they are still unaccessible. >From the management interface, their status is Running. Right now , I have the following: 1. A successfully upgraded (no errors reported in any of the upgrade step given in the document) 2.2.14 to 4.0.0 platform 2. 4 running VMs that are reachable via their public IP. 3. 2 unreachable ( but running) VMS that were rebooted after the upgrade process. 4. Running System VMS 5. I can reach the outside world from the system VMs via public network 6. I can reach the system VMS via their private IP address 7. I cannot reach from the internet to the public IP to any of the system VMS. Infrastructure >> Zone >> Zone01 >> Physical Network I see zone-name as: ZONE01-pNtwk200 , State: Enabled Public: Details Page: Does not Load, spinning/spawning circle IP ranges: All blank Guest: KVM traffic label: cloudbr0 Network: Type Shared, Scope: Domain(ROOT) , Name|VlanID|CIDR: blank Management: KVM traffic label: cloudbr1 IP Ranges: POD01 - has the IPs. Storage: All default IP ranges: All Blank ---- When I ping the VMS, via tcpdump I see the hypervisor gets the ping request, but I do not see it being forwarded, or any arp requests being made. I did a iptables -Z to clear out all counters and did a ping of 5000 bytes, to check if any rules are there are any iptables rule, but I was not able to find anything. When I ping for example the public IP of the console proxy, I get the following in tcpdump log in the hypervisor: 10.0.110.230 - public IP of the console proxy 10.0.28.16 - system doing the ping request. tcpdump -i any host 10.11.110.230 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes 11:35:31.918978 P 00:1d:e6:a5:9b:e4 (oui Unknown) ethertype Unknown (0x006e), length 80: 0x0000: 0000 0800 4500 003c 35af 0000 7d01 6911 ....E..<5...}.i. 0x0010: 0a00 1c10 0a0b 6ee6 0800 4d48 0001 0013 ......n...MH.... 0x0020: 6162 6364 6566 6768 696a 6b6c 6d6e 6f70 abcdefghijklmnop 0x0030: 7172 7374 7576 7761 6263 6465 6667 6869 qrstuvwabcdefghi 11:35:31.918978 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40 11:35:31.919022 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40 11:35:31.919029 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40 11:35:31.919038 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40 11:35:31.919043 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40 11:35:36.904310 P 00:1d:e6:a5:9b:e4 (oui Unknown) ethertype Unknown (0x006e), length 80: 0x0000: 0000 0800 4500 003c 35bf 0000 7d01 6901 ....E..<5...}.i. 0x0010: 0a00 1c10 0a0b 6ee6 0800 4d47 0001 0014 ......n...MG.... 0x0020: 6162 6364 6566 6768 696a 6b6c 6d6e 6f70 abcdefghijklmnop 0x0030: 7172 7374 7576 7761 6263 6465 6667 6869 qrstuvwabcdefghi 11:35:36.904310 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40 11:35:36.904362 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40 11:35:36.904371 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40 11:35:36.904382 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40 11:35:36.904388 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40 Tcpdump from inside the console proxy at the same time does not output anything. The ICMP requests do not reach the console proxy at all. Same case for all the system VMS where its not working. Cheers, Shashi Dahal
