Hashar has uploaded a new change for review. https://gerrit.wikimedia.org/r/101192
Change subject: beta: public IP rewriting using DNAT ...................................................................... beta: public IP rewriting using DNAT MediaWiki on the beta cluster would often attempt to query some API using the DNS entry (ie: en.wikipedia.beta.wmflabs.org). The DNS server would serve the public IP address of another instance and NAT would not let the instance communicates between each other. The fix is to rewrite outgoing destination address from the public IP address to the private IP address it is bound to. The iptables rules have been applied manually until then and disappeared whenever an instance got rebooted. That caused major headhaces to the Wikidata team for a few months now. We used to maintain iptables rules using Augeas, a system that nobody understand. Nowadays we are using ferm, a tiny wrapper around iptables which make it way easier to add iptables rules. I first define a list of hardcoded mappings then call a custom define using the nice create_resources() puppet utility. I did such a thing previously for the protoproxy configuration on beta (see d0b881e6 / https://gerrit.wikimedia.org/r/#/c/63431/ ). Should be the equivalent of generating the following iptables rules: iptables -t nat -I OUTPUT --dest 208.80.153.219 \ -j DNAT --to-dest 10.4.1.133 iptables -t nat -I OUTPUT --dest 208.80.153.242 \ -j DNAT --to-dest 10.4.0.211 iptables -t nat -I OUTPUT --dest 208.80.153.243 \ -j DNAT --to-dest 10.4.0.51 iptables -t nat -I OUTPUT --dest 208.80.153.244 \ -j DNAT --to-dest 10.4.0.48 iptables -t nat -I OUTPUT --dest 208.80.153.243 \ -j DNAT --to-dest 10.4.1.82 RT #4824 bug: 45868 Change-Id: I958a6d4aa1fb701aaccb2e9622c5a71e0d3fd93b --- M manifests/misc/beta.pp M manifests/role/beta.pp 2 files changed, 51 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/92/101192/1 diff --git a/manifests/misc/beta.pp b/manifests/misc/beta.pp index 5655583..e4fde07 100644 --- a/manifests/misc/beta.pp +++ b/manifests/misc/beta.pp @@ -49,6 +49,48 @@ } +# Workaround NAT traversal issue when a beta cluster instance attempt to +# connect to a beta public IP. The NAT would get the packet loss, instead +# transparently destination IP of outgoing packets to point directly to the +# private IP instance instead of the public IP. +# +# FIXME should probably be applied by default on ALL beta cluster instances. +# +# References: +# +# RT #4824 - https://rt.wikimedia.org/Ticket/Display.html?id=4824 +# bug #45868 - https://bugzilla.wikimedia.org/show_bug.cgi?id=45868 +class misc::beta::natfixup { + + # List out the instance public IP and private IP as described in OpenStack + # manager interface + # + # FIXME ideally that should be fetched directly from OpenStack + # configuration to make sure the iptables revwrites are always in sync with + # the web interface :-D + # + $nat_mappings = { + 'deployment-cache-text1' => { public_ip => '208.80.153.219', private_ip => '10.4.1.133' }, + 'deployment-cache-upload04' => { public_ip => '208.80.153.242', private_ip => '10.4.0.211' }, + 'deployment-cache-bits03' => { public_ip => '208.80.153.243', private_ip => '10.4.0.51' }, + 'deployment-eventlogging' => { public_ip => '208.80.153.244', private_ip => '10.4.0.48' }, + 'deployment-cache-mobile01' => { public_ip => '208.80.153.143', private_ip => '10.4.1.82' }, + } + create_resources( 'misc::beta::natdestrewrite', $nat_mappings ) +} + +define misc::beta::natdestrewrite( $public_ip, $private_ip ) { + + include base::firewall + + ferm::rule { "nat rewrite for ${name}": + # iptables -t nat -I OUTPUT --dest $public_ip -j DNAT --to-dest $private_ip + rule => "table nat chain OUTPUT { dest ${public_ip} { DNAT to ${private_ip} } }", + } + +} + + class misc::beta::sync-site-resources { file { "/usr/local/bin/sync-site-resources": ensure => present, diff --git a/manifests/role/beta.pp b/manifests/role/beta.pp index 49d24ed..8291113 100644 --- a/manifests/role/beta.pp +++ b/manifests/role/beta.pp @@ -15,6 +15,15 @@ } +class role::beta::natfixup { + + system::role { 'role::beta::natfix': + description => 'Server has beta NAT fixup' + } + + include misc::beta::natfixup +} + class role::beta::maintenance { class{ 'misc::maintenance::geodata': enabled => true } } -- To view, visit https://gerrit.wikimedia.org/r/101192 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I958a6d4aa1fb701aaccb2e9622c5a71e0d3fd93b Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Hashar <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
