Hi there,
I'm pretty new to vagrant, and i like it very much.
But I have a strange networking problem, I think the VM is dropping packets
when it should not.
My host is a HW machine connected on internet in a data center via eth0
(let's say IP 1.2.3.4).
I'm trying to create a local virtual network on which I attach a couple of
VMs.
I have tried several way to achieve this but always end up with the same
problem.
Right now, i'm creating a couple of peered veth interfaces, and a vbridge.
the first veth is is named veth_host.1 and veth_host.2.
veth_host.2 has the address 192.168.50.1 on the host machine, veth_host.1
is plugged into the vbridge.
I then create a second veth pair, named veth_guest.1 and veth_guest.2,
veth_guest.1 is plugged
into the vbridge and veth_guest.2 is used by the Vagrantfile using:
vm.network :public_network, ip: "192.168.50.10", bridge: "veth_master.2"
Similarly, I plugged another VM into this bridge. (i've tried
private_network, and VBox "internal", didn't solve the problem)
Now I configure the host to act as a gateway for the local 192.168.50.0/24
network, and I forward port 80/443
from the host's internet iface to the IP address 192.168.50.10.
With this setup, i can connect to 1.2.3.4:80/443 from another machine on
the internet, I can as well
connect to 1.2.3.4:80/443 from the second VM, but I cannot connect to
1.2.3.4:80/44 from either
1.2.3.4 (the host itself) or 192.168.50.10 (the guest itself).
This has driven me nuts for the last few days.
To troubleshoot this, i'm using
- on host and guest: "tcpdump -n -v -e -i any" (no name resolution, any
interface, verbose with ethernet headers)
- "nc -v -k -l 80" in the guest
- "nc -w 2 1.2.3.4" on the client side
- when the client runs on the host, i use "-s" to choose eth0 or
veth_host.2 as the source address.
When the client is another machine accessing 1.2.3.4 or the second guest,
everything is fine, i see these packets:
- first, client->1.2.3.4, followed by 192.168.50.1->192.168.50.10
- then 192.168.50.10->192.168.50.1 followed by 1.2.3.4->client.
And the 2 "nc" communicate both ways.
For the case of the client being the host or the guest itself, I can
clearly see on the host that
veth_guest.2 receives a 192.168.50.1->192.168.50.10 SYN packet, but inside
the guest, i cannot see this packet, I only see the original
192.168.50.10->1.2.3.4 SYN packet.
Which lead me to the conclusion that , for some reasons, the incoming SYN
packet is dropped by the VM.
The guest machine has it's firewall disabled (iptables flush, inc nat,
ACCEPT everywhere), the host starts with firewall disbaled too, just to
make sure it doesn't mess up my experiments. See script at end of email to
see how i'm setting up the virtual network and the gateway.
Does anyone knows what would make VBox decide to drop packets?
Any point out, advice or general comment would be greatly appreciated.
Thanks,
Chris
Vagrantfile:
------------------------------------------------------------------
WORKER_COUNT=2
Vagrant.require_version ">= 2.0.2" # Required for private network to work
properly
Vagrant.configure(2) do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.provision :shell, path: "src/provision-vm.sh"
config.vm.provision "shell", run: "always",
inline: "ip route del default; ip route add default
via 192.168.50.1"
config.vm.provision "shell", run: "always",
inline: "echo 'nameserver 8.8.8.8' > /etc/resolv.conf"
config.vm.define "master" do |master|
master.vm.network :public_network, ip: "192.168.50.10", bridge:
"veth_master.2"
master.vm.hostname = "master.local"
end
config.vm.define "master" do |leader|
leader.vm.network :public_network, ip: "192.168.50.11", bridge:
"veth_leader.2"
leader.vm.hostname = "leader.local"
end
(1..WORKER_COUNT).each do |i|
config.vm.define "worker#{i}" do |worker|
worker.vm.network :public_network, ip: "192.168.50.#{20+i}", bridge:
"veth_worker#{i}.2"
worker.vm.hostname = "worker#{i}.local"
end
end
end
-------------------------------------------------------------------
setup_vm_networking.sh (run on host):
-------------------------------------------------------------------
WORKER_COUNT=2
function add_patch_cable()
{
name=$1
ip link add dev veth_$name.1 type veth peer name veth_$name.2
ip link set veth_$name.1 up
ip link set veth_$name.2 up
ip link set veth_$name.1 master vbridge0
# switch off checksum
off-loading
ethtool -K veth_$name.1 tx off
ethtool -K veth_$name.2 tx off
}
# Create a virtual
bridge
ip link add name vbridge0 type bridge
ip link set vbridge0 up
ethtool -K vbridge0 tx off
# Add host to the bridge, set IP address and add route to the
network
add_patch_cable host
ip addr add 192.168.50.1 dev veth_host.2
ip route add 192.168.50.0/24 dev veth_host.2
# Add all VMs to the
bridge
add_patch_cable master
add_patch_cable leader
for i in $(seq 1 $WORKER_COUNT); do
add_patch_cable worker$i
done
# Allow
routing
sysctl net.ipv4.ip_forward=1
# Forward public ports to master
VM
PUBLIC_IP="1.2.3.4"
DEST_IP="192.168.50.10"
PORTS="80 443 7999"
for port in $PORTS; do
iptables -t nat -A PREROUTING -p tcp -d $PUBLIC_IP --dport $port -j
DNAT --to-destination $DEST_IP:$port
done
# For vm initiated outbound
traffic
iptables -t nat -A POSTROUTING -j MASQUERADE
-------------------------------------------------------------------
--
This mailing list is governed under the HashiCorp Community Guidelines -
https://www.hashicorp.com/community-guidelines.html. Behavior in violation of
those guidelines may result in your removal from this mailing list.
GitHub Issues: https://github.com/mitchellh/vagrant/issues
IRC: #vagrant on Freenode
---
You received this message because you are subscribed to the Google Groups
"Vagrant" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/vagrant-up/f9c2501c-20a1-40c6-a5cc-5f088d6d3afd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
