DeepthiMachiraju created CLOUDSTACK-9940:
--------------------------------------------
Summary: Rules ( PF , Firewall )when deleted during the VR stopped
state are still persistent on the VR iptables.
Key: CLOUDSTACK-9940
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9940
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Virtual Router
Affects Versions: 4.10.0.0
Reporter: DeepthiMachiraju
Fix For: 4.10.0.0
Attachments: cloud.log
- Create a network and configure PF , FW , LB rules on the source Nat IP.
- Stop the VR and delete the above rules , and add new rules with different
port numbers.
- Start the VR and check if the above rules are configured .
Observation :
- Rules which are newly added during the VR stop state are configured properly.
- Rules which are deleted are still retained in the respective json files and
reflecting in the iptable rules.
- Rules which are deleted are cleaned up from the DB and UI , but still
persistent in iptables.
**********************************************
mysql> select * from port_forwarding_rules;
+----+-------------+-----------------+-----------------+---------------+
| id | instance_id | dest_ip_address | dest_port_start | dest_port_end |
+----+-------------+-----------------+-----------------+---------------+
| 38 | 4 | 172.16.1.227 | 22 | 22 |
| 51 | 23 | 10.1.1.18 | 888 | 888 |
+----+-------------+-----------------+-----------------+---------------+
< forwardingrules.json >
- 2000 port is the one which was deleted when router is in stopped state.
- 888 port is the newly added rule when VR in stopped state .
root@r-29-VM:/etc/cloudstack# cat forwardingrules.json
{
"10.147.52.21": [
{
"internal_ip": "10.1.1.18",
"internal_ports": "2000:2000",
"protocol": "tcp",
"public_ip": "10.147.52.21",
"public_ports": "2000:2000",
"type": "forward"
},
{
"internal_ip": "10.1.1.18",
"internal_ports": "888:888",
"protocol": "tcp",
"public_ip": "10.147.52.21",
"public_ports": "888:888",
"type": "forward"
}
],
"id": "forwardingrules"
******************************************************
Firewall Rules :
mysql> select * from firewall_rules where network_id=209 and purpose='Firewall';
+----+--------------------------------------+---------------+------------+----------+--------+----------+----------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
| id | uuid | ip_address_id | start_port |
end_port | state | protocol | purpose | account_id | domain_id | network_id |
xid | created | icmp_code |
icmp_type | related | type | vpc_id | traffic_type | display |
+----+--------------------------------------+---------------+------------+----------+--------+----------+----------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
| 50 | a41a75b3-ba8b-4126-b098-f52fa8151891 | 12 | 8888 |
8888 | Active | tcp | Firewall | 2 | 1 | 209 |
e608b208-6e27-41c4-9163-40f3f3829929 | 2017-06-05 10:29:02 | NULL |
NULL | NULL | User | NULL | Ingress | 1 |
+----+--------------------------------------+---------------+------------+----------+--------+----------+----------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
1 row in set (0.00 sec)
< firewallrules.json >
- 555 port was deleted when VR in stopped state .
- 8888 port was added when VR in stopped state
root@r-29-VM:/etc/cloudstack# cat firewallrules.json
{
"0": {
"already_added": false,
"default_egress_policy": false,
"id": 0,
"protocol": "all",
"purpose": "Firewall",
"revoked": false,
"source_cidr_list": [],
"src_ip": "",
"traffic_type": "Egress"
},
"49": {
"already_added": false,
"default_egress_policy": false,
"id": 49,
"protocol": "tcp",
"purpose": "Firewall",
"revoked": false,
"source_cidr_list": [
"1.1.1.1/32"
],
"src_ip": "10.147.52.21",
"src_port_range": [
555,
555
],
"traffic_type": "Ingress"
},
"50": {
"already_added": true,
"default_egress_policy": false,
"id": 50,
"protocol": "tcp",
"purpose": "Firewall",
"revoked": false,
"source_cidr_list": [
"2.2.2.0/24"
],
"src_ip": "10.147.52.21",
"src_port_range": [
8888,
8888
],
"traffic_type": "Ingress"
},
"id": "firewallrules"
**************************************************************
root@r-29-VM:/etc/cloudstack# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- anywhere 10.147.52.21 tcp
dpt:cisco-sccp to:10.1.1.18:2000
DNAT tcp -- anywhere 10.147.52.21 tcp
dpt:cisco-sccp to:10.1.1.18:2000
MARK tcp -- anywhere 10.147.52.21 tcp
dpt:cisco-sccp MARK set 0x2
CONNMARK tcp -- anywhere 10.147.52.21 tcp
dpt:cisco-sccp state NEW CONNMARK save
DNAT tcp -- anywhere 10.147.52.21 tcp dpt:888
to:10.1.1.18:888
DNAT tcp -- anywhere 10.147.52.21 tcp dpt:888
to:10.1.1.18:888
MARK tcp -- anywhere 10.147.52.21 tcp dpt:888 MARK
set 0x2
CONNMARK tcp -- anywhere 10.147.52.21 tcp dpt:888 state
NEW CONNMARK save
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DNAT tcp -- anywhere 10.147.52.21 tcp
dpt:cisco-sccp to:10.1.1.18:2000
DNAT tcp -- anywhere 10.147.52.21 tcp dpt:888
to:10.1.1.18:888
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- anywhere anywhere to:10.147.52.21
SNAT tcp -- 10.1.1.0/24 v22 tcp
dpt:cisco-sccp to:10.1.1.1
SNAT tcp -- 10.1.1.0/24 v22 tcp dpt:888
to:10.1.1.1
***********************************************************
2017-06-05 06:33:12,633 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-192:ctx-0deed038) (logid:e11c0ab3) VR Config file
VR-e16a066b-d50e-48d2-9ba2-e582afd185e5.cfg got created in VR, ip 169.254.1.97
with content
#Apache CloudStack Virtual Router Config File
<version>
1.0
</version>
<file>
/var/cache/cloud/ip_associations.json
{"ip_address":[{"public_ip":"10.147.52.21","source_nat":true,"add":true,"one_to_one_nat":false,"first_i_p":true,"gateway":"10.147.52.1","netmask":"255.255.255.0","vif_mac_address":"06:45:88:00:00:17","nic_dev_id":2,"new_nic":false,"nw_type":"Public"}],"type":"ips"}
</file>
<script>
/opt/cloud/bin/update_config.py ip_associations.json
</script>
<file>
/var/cache/cloud/firewall_rules.json
{"rules":[{"id":50,"src_ip":"10.147.52.21","protocol":"tcp","src_port_range":[8888,8888],"revoked":false,"already_added":true,"source_cidr_list":["2.2.2.0/24"],"purpose":"Firewall","traffic_type":"Ingress","default_egress_policy":false}],"type":"firewallrules"}
</file>
<script>
/opt/cloud/bin/update_config.py firewall_rules.json
</script>
<file>
/var/cache/cloud/forwarding_rules.json
{"rules":[{"revoke":false,"protocol":"tcp","source_ip_address":"10.147.52.21","source_port_range":"888:888","destination_ip_address":"10.1.1.18","destination_port_range":"888:888"}],"type":"forwardrules"}
</file>
<script>
/opt/cloud/bin/update_config.py forwarding_rules.json
</script>
<file>
/var/cache/cloud/load_balancer.json
{"rules":[{"configuration":["global","\tlog 127.0.0.1:3914 local0
warning","\tmaxconn 4096","\tmaxpipes 1024","\tchroot /var/lib/haproxy","\tuser
haproxy","\tgroup haproxy","\tdaemon","\t ","defaults","\tlog
global","\tmode tcp","\toption dontlognull","\tretries 3","\toption
redispatch","\toption forwardfor","\toption forceclose","\ttimeout connect
5000","\ttimeout client 50000","\ttimeout server 50000","\nlisten
stats_on_public 10.147.52.21:8081\n\tmode http\n\toption httpclose\n\tstats
enable\n\tstats uri /admin?stats\n\tstats realm Haproxy\\
Statistics\n\tstats auth admin1:AdMiN123\n","\t ","listen 10_147_52_21-666
10.147.52.21:666","\tbalance roundrobin","\tserver 10_147_52_21-666_0
10.1.1.18:666 check","\t ","\t
"],"tmp_cfg_file_path":"/etc/haproxy/","tmp_cfg_file_name":"haproxy.cfg.new.1496658791133","add_rules":["10.147.52.21:666:"],"remove_rules":[],"stat_rules":["10.147.52.21:8081:0/0:,"],"router_ip":"169.254.1.97"}],"type":"loadbalancer"}
</file>
<script>
/opt/cloud/bin/update_config.py load_balancer.json
</script>
<file>
/var/cache/cloud/monitor_service.json
{"config":"[dhcp]:processname=dnsmasq:servicename=dnsmasq:pidfile=/var/run/dnsmasq/dnsmasq.pid:,[loadbalancing]:processname=haproxy:servicename=haproxy:pidfile=/var/run/haproxy.pid:,[ssh]:processname=sshd:servicename=ssh:pidfile=/var/run/sshd.pid:,[webserver]:processname=apache2:servicename=apache2:pidfile=/var/run/apache2.pid:,","type":"monitorservice"}
</file>
<script>
/opt/cloud/bin/update_config.py monitor_service.json
</script>
<file>
/var/cache/cloud/vm_dhcp_entry.json
{"host_name":"v11","mac_address":"02:00:2b:1f:00:01","ipv4_adress":"10.1.1.101","ipv6_duid":"00:03:00:01:02:00:2b:1f:00:01","dns_adresses":"10.1.1.1","default_gateway":"10.1.1.1","default_entry":true,"type":"dhcpentry"}
</file>
<script>
/opt/cloud/bin/update_config.py vm_dhcp_entry.json
</script>
<file>
/var/cache/cloud/vm_dhcp_entry.json
{"host_name":"v22","mac_address":"02:00:5f:59:00:04","ipv4_adress":"10.1.1.18","ipv6_duid":"00:03:00:01:02:00:5f:59:00:04","dns_adresses":"10.1.1.1","default_gateway":"10.1.1.1","default_entry":true,"type":"dhcpentry"}
</file>
<script>
/opt/cloud/bin/update_config.py vm_dhcp_entry.json
</script>
<file>
/var/cache/cloud/vm_metadata.json
{"vm_ip_address":"10.1.1.101","vm_metadata":[["userdata","user-data",null],["metadata","service-offering","Small
Instance"],["metadata","availability-zone","z1"],["metadata","local-ipv4","10.1.1.101"],["metadata","local-hostname","v11"],["metadata","public-ipv4","10.147.52.21"],["metadata","public-hostname","10.147.52.21"],["metadata","instance-id","d09cf9fe-cddc-4f8a-952f-5dadede3ab91"],["metadata","vm-id","d09cf9fe-cddc-4f8a-952f-5dadede3ab91"],["metadata","public-keys",null],["metadata","cloud-identifier","CloudStack-{5a63cc81-c2c7-46ac-a959-bff4d5b12f19}"]],"type":"vmdata"}
</file>
<script>
/opt/cloud/bin/update_config.py vm_metadata.json
</script>
<file>
/var/cache/cloud/vm_metadata.json
{"vm_ip_address":"10.1.1.18","vm_metadata":[["userdata","user-data",null],["metadata","service-offering","Small
Instance"],["metadata","availability-zone","z1"],["metadata","local-ipv4","10.1.1.18"],["metadata","local-hostname","v22"],["metadata","public-ipv4","10.147.52.21"],["metadata","public-hostname","10.147.52.21"],["metadata","instance-id","7f524b97-6e2f-4b19-b40e-6da7f59f3f2e"],["metadata","vm-id","7f524b97-6e2f-4b19-b40e-6da7f59f3f2e"],["metadata","public-keys",null],["metadata","cloud-identifier","CloudStack-{5a63cc81-c2c7-46ac-a959-bff4d5b12f19}"]],"type":"vmdata"}
</file>
<script>
/opt/cloud/bin/update_config.py vm_metadata.json
</script>
2017-06-05 06:33:12,634 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-192:ctx-0deed038) (logid:e11c0ab3) Executing command in VR:
/opt/cloud/bin/router_proxy.sh vr_cfg.sh 169.254.1.97 -c
/var/cache/cloud/VR-e16a066b-d50e-48d2-9ba2-e582afd185e5.cfg
2017-06-05 06:33:13,059 DEBUG [c.c.a.ApiServlet] (catalina-exec-2:ctx-0bd038a0)
(logid:d85c5105) ===START=== 10.233.89.32 -- GET
command=queryAsyncJobResult&jobId=e11c0ab3-55fb-4a30-a0fd-dc550b1b45f5&response=json&_=1496658792996
2017-06-05 06:33:13,142 DEBUG [c.c.a.ApiServlet] (catalina-exec-2:ctx-0bd038a0
ctx-7be6be07) (logid:d85c5105) ===END=== 10.233.89.32 -- GET
command=queryAsyncJobResult&jobId=e11c0ab3-55fb-4a30-a0fd-dc550b1b45f5&response=json&_=1496658792996
2017-06-05 06:33:13,225 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-6:null) (logid:) SeqA 2-144522: Processing Seq 2-144522:
{ Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2017-06-05 06:33:13,231 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-6:null) (logid:) SeqA 2-144522: Sending Seq 2-144522: {
Ans: , MgmtId: 6760647622781, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2017-06-05 06:33:14,554 DEBUG [c.c.s.StatsCollector]
(StatsCollector-6:ctx-ca3f95b2) (logid:27ed7e15) VmStatsCollector is running...
2017-06-05 06:33:14,599 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-13:ctx-42ce21d1) (logid:ee6b66c9) Seq 1-2918895508489535017:
Executing request
*******************************************************************************
Attached cloud.log
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
