venkata swamybabu budumuru created CLOUDSTACK-2761:
------------------------------------------------------
Summary: [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC
Key: CLOUDSTACK-2761
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2761
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Network Controller
Affects Versions: 4.2.0
Environment: commit # 8d1189c2ae87216bc1c4a1443f75e9a8629abdc2
Reporter: venkata swamybabu budumuru
Priority: Blocker
Fix For: 4.2.0
Steps to reproduce:
1. Have latest CloudStack build with at least 1 advanced zone
2. Have at least 1 VMware cluster with 1 host.
3. Create a VPC with at least 1 Tier
4. deploy a VM
5. apply an ACL to allow all
6. acquire at least 1 IP
7. create PF/staticNAT on the above created IP to the VM created in step (4)
Observations:
(i) It failed to create staticNAT with the following error
2013-05-30 08:05:20,203 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-38:null) Seq 1-1416495168: Response Received:
2013-05-30 08:05:20,206 DEBUG [agent.transport.Request] (DirectAgent-38:null)
Seq 1-1416495168: Processing: { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 0, [{"routing.IpAssocAnswer":{"results":["10.147.44.63 -
success"],"result":true,"wait":0}}] }
2013-05-30 08:05:20,207 DEBUG [agent.transport.Request] (catalina-exec-19:null)
Seq 1-1416495168: Received: { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 0, { IpAssocAnswer } }
2013-05-30 08:05:20,211 INFO [cloud.network.NetworkManagerImpl]
(catalina-exec-19:null) Let VpcVirtualRouter handle StaticNat in network 204
2013-05-30 08:05:20,223 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-19:null)
Applying static nat rules in network Ntwk[204|Guest|11]
2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] (catalina-exec-19:null)
Seq 1-1416495169: Sending { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 100001,
[{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] (catalina-exec-19:null)
Seq 1-1416495169: Executing: { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 100001,
[{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:05:20,289 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-16:null) Seq 1-1416495169: Executing request
2013-05-30 08:05:20,290 INFO [vmware.resource.VmwareResource]
(DirectAgent-16:10.147.40.12) Executing resource SetFirewallRuleCommand:
{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
2013-05-30 08:05:20,290 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-16:10.147.40.12) Use router's private IP for SSH control. IP :
10.147.40.62
2013-05-30 08:05:21,746 ERROR [utils.ssh.SshHelper]
(DirectAgent-16:10.147.40.12) SSH execution of command /root/firewall.sh -A
-l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G has an error status code in return.
result output: Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.14: option "--set-mark" requires an argument
Try `iptables -h' or 'iptables --help' for more information.
iptables: No chain/target/match by that name.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `eth0'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
2013-05-30 08:05:21,790 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-16:10.147.40.12) Executing script on domain router 10.147.40.62:
/root/firewall.sh -A -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G
(ii) After changing the firewall.sh as mentioned below, it went fine.
get_dev_list() {
ip link show | grep -e eth[2-9] | awk -F ":" '{print $2}'
ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
}
Changed the above with the following to include eth1 device as well.
get_dev_list() {
ip link show | grep -e eth[1-9] | awk -F ":" '{print $2}'
ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
}
2013-05-30 08:32:52,492 INFO [cloud.network.NetworkManagerImpl]
(catalina-exec-3:null) Let VpcVirtualRouter handle StaticNat in network 204
2013-05-30 08:32:52,506 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-3:null)
Applying static nat rules in network Ntwk[204|Guest|11]
2013-05-30 08:32:52,523 DEBUG [agent.transport.Request] (catalina-exec-3:null)
Seq 1-1416495239: Sending { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 100001,
[{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:32:52,524 DEBUG [agent.transport.Request] (catalina-exec-3:null)
Seq 1-1416495239: Executing: { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 100001,
[{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:32:52,524 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-12:null) Seq 1-1416495239: Executing request
2013-05-30 08:32:52,525 INFO [vmware.resource.VmwareResource]
(DirectAgent-12:10.147.40.12) Executing resource SetFirewallRuleCommand:
{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
2013-05-30 08:32:52,529 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-12:10.147.40.12) Use router's private IP for SSH control. IP :
10.147.40.62
2013-05-30 08:32:53,937 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-12:10.147.40.12) Executing script on domain router 10.147.40.62:
/root/firewall.sh -A -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G
2013-05-30 08:32:53,938 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-12:null) Seq 1-1416495239: Response Received:
2013-05-30 08:32:53,938 DEBUG [agent.transport.Request] (DirectAgent-12:null)
Seq 1-1416495239: Processing: { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 0,
[{"routing.SetStaticNatRulesAnswer":{"results":[null],"result":true,"wait":0}}]
}
(iii) Tried to access the staticNat rule but, it failed because, it created a
firewall rule with DROP policy which is not required in case of staticNAt.
Ideally instead of firewall.sh, it should call "vlc_staticnat.sh" script to
configure staticnat.
Here is the snippet of iptable output.
root@r-3-VM:~# iptables -L -nv -t mangle
Chain PREROUTING (policy ACCEPT 3060 packets, 346K bytes)
pkts bytes target prot opt in out source destination
1 60 CONNMARK all -- eth1 * 0.0.0.0/0 0.0.0.0/0
state NEW CONNMARK set 0x1
2360 3256K FIREWALL_10.147.44.63 all -- * * 0.0.0.0/0
10.147.44.63
1388 77740 CONNMARK all -- eth2 * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED CONNMARK restore
27 1620 ACL_OUTBOUND_eth2 all -- eth2 * 10.0.1.0/24
!10.0.1.1 state NEW
0 0 MARK all -- eth1 * 0.0.0.0/0
10.147.44.63 state NEW MARK set 0x1
0 0 CONNMARK all -- eth1 * 0.0.0.0/0
10.147.44.63 state NEW CONNMARK save
0 0 MARK all -- eth0 * 10.0.1.188 0.0.0.0/0
state NEW MARK set 0x1
0 0 CONNMARK all -- eth0 * 10.0.1.188 0.0.0.0/0
state NEW CONNMARK save
Chain INPUT (policy ACCEPT 3048 packets, 345K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 21 packets, 10108 bytes)
pkts bytes target prot opt in out source destination
3774 3335K VPN_STATS_eth1 all -- * * 0.0.0.0/0
0.0.0.0/0
Chain OUTPUT (policy ACCEPT 2402 packets, 384K bytes)
pkts bytes target prot opt in out source destination
0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0
udp dpt:68 CHECKSUM fill
Chain POSTROUTING (policy ACCEPT 2423 packets, 395K bytes)
pkts bytes target prot opt in out source destination
Chain ACL_OUTBOUND_eth2 (1 references)
pkts bytes target prot opt in out source destination
27 1620 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FIREWALL_10.147.44.63 (1 references)
pkts bytes target prot opt in out source destination
2359 3256K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
1 60 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain VPN_STATS_eth1 (1 references)
pkts bytes target prot opt in out source destination
0 0 all -- * eth1 0.0.0.0/0 0.0.0.0/0
mark match 0x525
0 0 all -- eth1 * 0.0.0.0/0 0.0.0.0/0
mark match 0x524
(iv) Tried to configure PF rule but that as well failed but with a different
error.
here is the snippet from mgmt server log.
2013-05-30 11:30:35,264 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-495:10.147.40.12) Use router's private IP for SSH control. IP :
10.147.40.62
2013-05-30 11:30:36,548 ERROR [utils.ssh.SshHelper]
(DirectAgent-495:10.147.40.12) SSH execution of command
/opt/cloud/bin/vpc_portforwarding.sh -A -P tcp -l 10.147.44.64 -p 22-22 -r
10.0.1.188 -d 22-22 has an error status code in return. result output:
2013-05-30 11:30:36,555 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-495:null) Seq 1-1416495677: Response Received:
2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] (DirectAgent-495:null)
Seq 1-1416495677: Processing: { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1,
Flags: 0,
[{"routing.SetPortForwardingRulesAnswer":{"results":["Failed"],"result":false,"wait":0}}]
}
2013-05-30 11:30:36,556 DEBUG [agent.transport.Request]
(Job-Executor-26:job-23) Seq 1-1416495677: Received: { Ans: , MgmtId:
7280707764394, via: 1, Ver: v1, Flags: 0, { SetPortForwardingRulesAnswer } }
2013-05-30 11:30:36,556 WARN [network.rules.RulesManagerImpl]
(Job-Executor-26:job-23) Failed to apply port forwarding rules for ip due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: Unable to apply firewall rules on router
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3739)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3567)
at
com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:787)
at
com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:565)
at
com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2913)
at
com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:509)
at
com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:889)
at
com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:1072)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.user.firewall.CreatePortForwardingRuleCmd.execute(CreatePortForwardingRuleCmd.java:184)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
(v) Tried the following iptables command manually and found an error with syntax
root@r-3-VM:~# sudo iptables -t nat -A PREROUTING --proto tcp -d 10.147.44.64
--destination-port 22-22 -j DNAT --to-destination 10.0.1.188:22-22
iptables v1.4.14: invalid port/service `22-22' specified
Try `iptables -h' or 'iptables --help' for more information.
Attaching all the mgmt server logs to the bug.
Here is the system vm template version that I used for testing.
root@r-3-VM:~# cat /etc/cloudstack-release
Cloudstack Release 4.2.0 Tue Apr 16 04:09:58 UTC 2013
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
