Hi Martin,
Thanks for the fix, didn’t catch you attachment first time.
Would it be possible for you to send a Pull Request? Is this patch against
master or a release branch? Generally speaking it’s best to make a PR against a
release branch, 4.7 would be fine I guess in this case. Once it’s a PR we can
test it.
Regards,
Remi
From: martin kolly <martin.ko...@senselan.ch<mailto:martin.ko...@senselan.ch>>
Reply-To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
<dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
Date: Friday 18 March 2016 at 11:58
To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
<dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
Subject: Issue: CLOUDSTACK-9255 Unable to start VM DomainRouter due to error in
finalizeStart
Hi All
We are facing the same issue as reported by Milamber (Ticket 9255)
https://issues.apache.org/jira/browse/CLOUDSTACK-9255. When deploying a couple
of VMs or Port Forwarding's the re-deployment of the router with cleanup fails.
We found that iptables configuration takes a lot of time, this eventually leads to a
timeout on the management server "Unable to start VM DomainRouter due to error in
finalizeStart, not retrying"
Environment:
- Cloudstack 4.8
- KVM (local storage)
- hosts/mgr on Ubuntu 14.04
We tested with a simple set of four forwarding rules, here the setup:
root@r-96-VM:~# cat /etc/cloudstack/forwardingrules.json
{
"185.20.146.56": [
{
"internal_ip": "10.100.1.95",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.56",
"public_ports": "22:22",
"type": "forward"
}
],
"185.20.146.79": [
{
"internal_ip": "10.100.1.42",
"internal_ports": "22:22",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "22:22",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "8443:8443",
"protocol": "tcp",
"public_ip": "185.20.146.79",
"public_ports": "8443:8443",
"type": "forward"
},
{
"internal_ip": "10.100.1.42",
"internal_ports": "53:53",
"protocol": "udp",
"public_ip": "185.20.146.79",
"public_ports": "53:53",
"type": "forward"
}
],
"id": "forwardingrules"
The definition for every port forwarding seems to take at ~1.5 seconds.
python /opt/cloud/bin/configure.py.timed /etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.42:22
time : 0.000965118408203
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.42:22
time : 0.395485162735
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination
10.100.1.42:22
time : 0.395533084869
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.16180706024
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK
--set-xmark 0x2/0xffffffff
time : 1.16329216957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.16407108307
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.53959512711
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.000781059265137
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.378201007843
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.37822508812
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 1.14627504349
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK
--set-xmark 0x2/0xffffffff
time : 1.1477329731
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14850592613
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52321791649
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT
--to-destination 10.100.1.42:53
time : 0.000754117965698
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT
--to-destination 10.100.1.42:53
time : 0.383729934692
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination
10.100.1.42:53
time : 0.383754968643
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 1.14376091957
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK
--set-xmark 0x2/0xffffffff
time : 1.14526605606
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14599299431
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52742600441
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.95:22
time : 0.000700950622559
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.95:22
time : 0.382349014282
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination
10.100.1.95:22
time : 0.382384061813
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 1.1425909996
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK
--set-xmark 0x2/0xffffffff
time : 1.14400196075
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 1.14468812943
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 1.52619600296
----------------------------------------------
Having a closer look at configure.py how the iptables rules are defined. We
think that it is not efficient to lookup these values for every policy:
def forward_vr(self, rule):
fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination
%s:%s" % \
(
rule['public_ip'],
self.getDeviceByIp(rule['public_ip']),
rule['protocol'],
rule['protocol'],
self.portsToString(rule['public_ports'], ':'),
rule['internal_ip'],
self.portsToString(rule['internal_ports'], '-')
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT --to-destination
%s:%s" % \
(
rule['public_ip'],
self.getDeviceByIp(rule['internal_ip']),
rule['protocol'],
rule['protocol'],
self.portsToString(rule['public_ports'], ':'),
rule['internal_ip'],
self.portsToString(rule['internal_ports'], '-')
.....
Defining these values once at the beginning would be much more efficient, no ?
def forward_vr(self, rule):
pub_interface = self.getDeviceByIp(rule['public_ip'])
int_interface = self.getDeviceByIp(rule['internal_ip'])
pub_ports = self.portsToString(rule['public_ports'], ':')
int_ports = self.portsToString(rule['internal_ports'], '-')
int_network = self.getNetworkByIp(rule['internal_ip'])
fw1 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
--to-destination %s:%s" % \
(
rule['public_ip'],
pub_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
fw2 = "-A PREROUTING -d %s/32 -i %s -p %s -m %s --dport %s -j DNAT
--to-destination %s:%s" % \
(
rule['public_ip'],
int_interface,
rule['protocol'],
rule['protocol'],
pub_ports,
rule['internal_ip'],
int_ports
)
.....
If we run the configure.py with these modifications we have the following:
root@r-96-VM:~# python /opt/cloud/bin/configure_modified.py
/etc/cloudstack/forwardingrules.json
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.42:22
time : 0.000349044799805
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.42:22
time : 0.000686883926392
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination
10.100.1.42:22
time : 0.000943899154663
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00131487846375
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK
--set-xmark 0x2/0xffffffff
time : 0.00161194801331
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 22 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00186896324158
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00216102600098
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.000232934951782
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.000478029251099
-A OUTPUT -d 185.20.146.79/32 -p tcp -m tcp --dport 8443 -j DNAT
--to-destination 10.100.1.42:8443
time : 0.00071907043457
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p tcp -m tcp --dport 8443
time : 0.000991106033325
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -j MARK
--set-xmark 0x2/0xffffffff
time : 0.00136613845825
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p tcp -m tcp --dport 8443 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00174498558044
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 8443 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00219202041626
----------------------------------------------
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j DNAT
--to-destination 10.100.1.42:53
time : 0.000226974487305
-A PREROUTING -d 185.20.146.79/32 -i eth0 -p udp -m udp --dport 53 -j DNAT
--to-destination 10.100.1.42:53
time : 0.000502824783325
-A OUTPUT -d 185.20.146.79/32 -p udp -m udp --dport 53 -j DNAT --to-destination
10.100.1.42:53
time : 0.000762939453125
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.42/32 -o eth0 -p udp -m udp --dport 53
time : 0.00103092193604
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -j MARK
--set-xmark 0x2/0xffffffff
time : 0.00134587287903
-A PREROUTING -d 185.20.146.79/32 -i eth2 -p udp -m udp --dport 53 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00158596038818
-A FORWARD -i eth2 -o eth0 -p udp -m udp --dport 53 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00182485580444
----------------------------------------------
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.95:22
time : 0.000264167785645
-A PREROUTING -d 185.20.146.56/32 -i eth0 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 10.100.1.95:22
time : 0.000508069992065
-A OUTPUT -d 185.20.146.56/32 -p tcp -m tcp --dport 22 -j DNAT --to-destination
10.100.1.95:22
time : 0.000750064849854
-j SNAT --to-source 10.100.1.1 -A POSTROUTING -s 10.100.1.0/24 -d
10.100.1.95/32 -o eth0 -p tcp -m tcp --dport 22
time : 0.00102114677429
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -j MARK
--set-xmark 0x2/0xffffffff
time : 0.00138115882874
-A PREROUTING -d 185.20.146.56/32 -i eth2 -p tcp -m tcp --dport 22 -m state
--state NEW -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
time : 0.00165915489197
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 22 -m state --state
NEW,ESTABLISHED -j ACCEPT
Total time for creating Policy : 0.00196814537048
----------------------------------------------
Location of configure.py:
https://github.com/apache/cloudstack/blob/master/systemvm/patches/debian/config/opt/cloud/bin/configure.py
The modified scripts are attached. Thanks for your feedback.
regards
Martin
