[Bug 616064] Re: Broken networking in kvm guests with pcnet driver
Brain Fallik, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you please test for this with the latest server release of Ubuntu? ISO images are available from http://releases.ubuntu.com/raring/ . If it remains an issue, could you please run the following command in the development release from a Terminal (Applications-Accessories-Terminal), as it will automatically gather and attach updated debug information to this report: apport-collect -p linux replace-with-bug-number Also, could you please test the latest upstream kernel available following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Please do not test the daily folder, but the one all the way at the bottom. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags: kernel-fixed-upstream kernel-fixed-upstream-VERSION-NUMBER where VERSION-NUMBER is the version number of the kernel you tested. For example: kernel-fixed-upstream-v3.11-rc5 This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag: needs-upstream-testing If the mainline kernel does not fix this bug, please add the following tags: kernel-bug-exists-upstream kernel-bug-exists-upstream-VERSION-NUMBER As well, please remove the tag: needs-upstream-testing Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding. ** Tags added: bios-outdated-105 needs-upstream-testing ** Changed in: linux (Ubuntu) Status: Confirmed = Incomplete ** Description changed: Binary package hint: qemu-kvm - Some TCP operations receive unexpected reset (RST) packets when executed on the guest. I can see these RST packets via Wireshark but can't explain their source. This breaks certain (most) networking on the guest. I uncovered this when some 'git clone' operations failed with Broken Pipe in the guests. I've been able to reproduce the same failures using scp and 'svn checkout'. The failure occurs 80-90% of the time and 'svn checkout' seems to be the quickest to fail. The same operations on the host are always successful. My setup is a Lucid amd64 host with two guests (Lucid and Hardy, both i386, using kvm). Networking seems to mostly work. I can maintain persistent ssh connections to the guest and some networking operations from the guest (ie. ping) never fail. However, the git-clone/scp/svn- checkout operations fail almost all the time. I've tried this with both virtio and e1000 drivers with no difference. This may be similar to some of the behavior reported in bug 584048 but not the final patch referenced in comment 27. I built a custon deb including the upstream patch for testing and it had no effect. - $ lsb_release -rd - Description: Ubuntu 10.04.1 LTS - Release: 10.04 + BIOS: + http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5080935 $ apt-cache policy qemu-kvm qemu-kvm: - Installed: 0.12.3+noroms-0ubuntu9.2 - Candidate: 0.12.3+noroms-0ubuntu9.2 - Version table: - *** 0.12.3+noroms-0ubuntu9.2 0 - 500 http://us.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages - 100 /var/lib/dpkg/status - 0.12.3+noroms-0ubuntu9 0 - 500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages + Installed: 0.12.3+noroms-0ubuntu9.2 + Candidate: 0.12.3+noroms-0ubuntu9.2 + Version table: + *** 0.12.3+noroms-0ubuntu9.2 0 + 500 http://us.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages + 100 /var/lib/dpkg/status + 0.12.3+noroms-0ubuntu9 0 + 500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages ProblemType: Bug DistroRelease: Ubuntu 10.04 Package: kvm 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9.2 ProcVersionSignature: Ubuntu 2.6.32-24.39-server 2.6.32.15+drm33.5 Uname: Linux 2.6.32-24-server x86_64 Architecture: amd64 Date: Tue Aug 10 16:39:24 2010 InstallationMedia: Ubuntu-Server 10.04 Lucid Lynx - Beta amd64 (20100406.1) KvmCmdLine: - UIDPID PPID CSZ RSS PSR STIME TTY TIME CMD - root 17641 1 1 2174883 1087144 1 15:42 ? 00:00:59 /usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 8192 -smp 8 -name hound-lucid -uuid 0e32438c-cb4d-1eb9-a560-f25c89c1dbf6 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/hound-lucid.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/opt/hound-lucid,if=virtio,index=0,boot=on -drive if=ide,media=cdrom,index=2 -net nic,macaddr=52:54:00:61:7e:c0,vlan=0,model=virtio,name=virtio.0 -net tap,fd=59,vlan=0,name=tap.0 -chardev
[Bug 616064] Re: Broken networking in kvm guests with pcnet driver
** Changed in: bridge-utils (Ubuntu) Status: Confirmed = Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests with pcnet driver To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests with pcnet driver
** Summary changed: - Broken networking in kvm guests + Broken networking in kvm guests with pcnet driver -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests with pcnet driver To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
I have the same problem on 11.10 (old version had it too), if I scp/nfs a large file it starts, go fast but at some point the network of the machine does not work anymore. sometimes it transfers 1GB others a 100MB. I use the bridge interface to get to the machine and it still responding, it seems like the attached virtual network card is what is failing.. I remember using tun/tap with VirtualBox, is it possible to try using that with the bridge? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Following the advice on http://serverfault.com/questions/341178/downloads-stuck-from-a-vm-run-by-libvirt-kvm-on-ubuntu-10-04-lts I changed: model type='pcnet'/ To model type='rtl8139'/ And now I'm able to transfer my 37GB tar, so it seems a bug in the virtual drivers. BTW It's not only downloads that stuck, killing the proccess does not revive the interface which does not even respond to ping. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
@Daniel: yes, you can use tap devices with kvm, and attach the tap devices to the bridge. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Interesting. If multiple downloads of small files is the problem, could you retry the experiment from comment #46 but using the small wgets? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
1. $ virsh dumpxml vs-maven domain type='kvm' id='4' namevs-maven/name uuid40f97e28-afc2-1730-3dfb-292074bae1d6/uuid memory2097152/memory currentMemory2097152/currentMemory vcpu1/vcpu os type arch='x86_64' machine='pc-0.12'hvm/type boot dev='hd'/ /os features acpi/ apic/ pae/ /features clock offset='utc'/ on_poweroffdestroy/on_poweroff on_rebootrestart/on_reboot on_crashrestart/on_crash devices emulator/usr/bin/kvm/emulator disk type='file' device='disk' driver name='qemu' type='qcow2'/ source file='/var/lib/libvirt/images/vs-maven.img'/ target dev='vda' bus='virtio'/ /disk disk type='block' device='cdrom' driver name='qemu' type='raw'/ target dev='hdc' bus='ide'/ readonly/ /disk interface type='bridge' mac address='52:54:00:0f:30:ea'/ source bridge='br0'/ target dev='vnet3'/ model type='virtio'/ /interface console type='pty' tty='/dev/pts/3' source path='/dev/pts/3'/ target port='0'/ /console console type='pty' tty='/dev/pts/3' source path='/dev/pts/3'/ target port='0'/ /console input type='mouse' bus='ps2'/ graphics type='vnc' port='5903' autoport='yes' keymap='fr'/ video model type='cirrus' vram='9216' heads='1'/ /video /devices seclabel type='dynamic' model='apparmor' labellibvirt-40f97e28-afc2-1730-3dfb-292074bae1d6/label imagelabellibvirt-40f97e28-afc2-1730-3dfb-292074bae1d6/imagelabel /seclabel /domain 2. Server : admin@vs-master:~$ lsb_release -rd Description:Ubuntu 10.04.3 LTS Release:10.04 admin@vs-master:~$ uname -a Linux vs-master 2.6.32-37-server #81-Ubuntu SMP Fri Dec 2 20:49:12 UTC 2011 x86_64 GNU/Linux admin@vs-master:~$ apt-cache policy kvm kvm: Installé : (aucun) Candidat : 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9.16 Table de version : 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9.16 0 500 http://fr.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9.15 0 500 http://security.ubuntu.com/ubuntu/ lucid-security/main Packages 1:84+dfsg-0ubuntu16+0.12.3+noroms+0ubuntu9 0 500 http://fr.archive.ubuntu.com/ubuntu/ lucid/main Packages Host : admin@vs-maven:~$ lsb_release -rd Description:Ubuntu 10.04.3 LTS Release:10.04 admin@vs-maven:~$ uname -a Linux vs-maven 2.6.32-37-server #81-Ubuntu SMP Fri Dec 2 20:49:12 UTC 2011 x86_64 GNU/Linux 3. I've tried to download 5 times a big file (wget http://192.168.1.213:8081/artifactory/repo/org/hibernate/hibernate-core/4.0.0.CR2/hibernate-core-4.0.0.CR2.jar) without problem. It seems the problem happends when the server is asked to download several small files. From my client, I did : $ wget -l 2 -r http://192.168.1.213:8081/artifactory/repo/org/jboss/seam/faces/seam-faces/ And after crawling a bit, it stuck on http://192.168.1.213:8081/artifactory/repo/org/jboss/seam/faces/seam-faces/3.1.0-SNAPSHOT/; : wget is waiting for something, the little spaceship (=) keeps going left to right reverses. Later in my browser, the same page is displayed correctly. If I cancel retry the same command, it sticks somewhere else. I've tried the same command with --timeout=5 and it sometime times out to correctly get it later in another retry. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Do you want me to test another mechanism ? Thanks, but no thanks. While it's not 100% conclusive, this certainly suggests that the bug may be in the driver in qemu. To help try to reproduce, can you give: 1. output of 'virsh dumpxml vm-name' 2. the release of both your host and guest. 3. any more info you can give on the thing that is failing. Can you get it to fail with a wget (from the host to the VM) of a 1M file, or 5 repeated such wgets? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Here is what I done in my client : user@host:~$ cat /dev/urandom | hexdump | head -c 1048576 /tmp/1MB.txt user@host:~$ N=1; while [ 1 ]; do echo $N | nc 192.168.1.214 8080; cat /tmp/1MB.txt | nc 192.168.1.214 8080; ((N += 1)); sleep 0.5s; done No problem here, in terminal 1 all was displayed without blockings. Do you want me to test another mechanism ? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
To reproduce the use case, I need that nc serves big files from the server and that it would be my client which would download the 1MB file... but I don't manage to use nc correctly to do that. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
If all goes well the instructions in comment #35 will cause no problems and the VM will remain reachable. However a typo could indeed endanger the network and be less than trivial to fix. The experiment would only help us to determine whether the bug comes down to the kernel or part of the virtualization stack in userspace. So if there is a scheduled downtime for the VM during which it could be done it would be helpful, but it may not be worth the risk otherwise. ** Changed in: qemu-kvm (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Is there any ywireshark data showing arp and dns traffic at the same time as the conversation you posted? (it's not in the file itself) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Well I've tried this (after first steps with git)
# terminal 1
root@vs-master:~# ip link add type veth
# terminal 2
root@vs-master:~# /bin/ns_exec -cmn /bin/bash
about to clone with 4002
clone: Invalid argument
root@vs-master:~# echo $$
7934
# terminal 1
root@vs-master:~# ifconfig veth0 0.0.0.0 up
root@vs-master:~# brctl addif virbr0 veth0
bridge virbr0 does not exist!
root@vs-master:~# brctl addif virbr1 veth0
root@vs-master:~# ip link set veth1 netns 7934
# terminal 2
root@vs-master:~# ifconfig veth1 up
root@vs-master:~# dhclient veth1
There is already a pid file /var/run/dhclient.pid with pid 15307
killed old client process, removed PID file
Internet Systems Consortium DHCP Client V3.1.3
Copyright 2004-2009 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/veth1/1a:13:2e:39:53:2e
Sending on LPF/veth1/1a:13:2e:39:53:2e
Sending on Socket/fallback
DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 7
DHCPOFFER of 192.168.122.61 from 192.168.122.1
DHCPREQUEST of 192.168.122.61 on veth1 to 255.255.255.255 port 67
DHCPACK of 192.168.122.61 from 192.168.122.1
SIOCADDRT: No such process
bound to 192.168.122.61 -- renewal in 1771 seconds.
# terminal 1
root@vs-master:~# nc -4 -k -l
# terminal 2
root@vs-master:~# while [ 1 ]; do
echo ab | nc 192.168.1.200
sleep 0.5s
done
All had gone well, I had ab lines in terminal 1 without interruption.
I've enhanced the terminal 2 script in order to check if ab was still comming
into terminal 1 (because I've been a little lost with a big pile of ab) :
# N=1; while [ 1 ]; do echo ab$N | nc 192.168.1.200 ; ((N += 1)); sleep
0.5s; done
No problem.
I switched to sleep 0.01s and still no problem.
Last script to test content length :
# S=a; N=1; while [ 1 ]; do echo $N$S | nc 192.168.1.200 ; ((N += 1));
S=${S}a; sleep 0.01s; done
No problem.
But actually, my VM is using br0 (with static IPs in 192.168.1.0
network) rather than virbr1 so how could I test this ?
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to bridge-utils in Ubuntu.
https://bugs.launchpad.net/bugs/616064
Title:
Broken networking in kvm guests
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
It's just another bridge, so you should be able to substitude br0 for virbr1. Note that if veth0 and veth1 already exist from your last test, you can either re-use them, or else when you do another 'ip link add type veth' you'll create veth2 and veth3 and can use those. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
I've tested
# terminal 1
root@vs-master:~# brctl delif virbr1 veth0
root@vs-master:~# brctl addif br0 veth0
root@vs-master:~# ip link set veth1 netns 5039
root@vs-master:~# nc -4 -k -l 8080
# terminal 2
root@vs-master:~# /bin/ns_exec -cmn /bin/bash
about to clone with 4002
clone: Invalid argument
root@vs-master:~# echo $$
5039
root@vs-master:~# ifconfig veth1 192.168.1.214
root@vs-master:~# ifconfig veth1
veth1 Link encap:Ethernet HWaddr 1a:13:2e:39:53:2e
inet adr:192.168.1.214 Bcast:192.168.1.255 Masque:255.255.255.0
adr inet6: fe80::1813:2eff:fe39:532e/64 Scope:Lien
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Packets reçus:3202 erreurs:0 :0 overruns:0 frame:0
TX packets:193 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
Octets reçus:123117 (123.1 KB) Octets transmis:39048 (39.0 KB)
From my client :
user@host:~$ S=a; N=1; while [ 1 ]; do echo $N$S | nc 192.168.1.214 8080; ((N
+= 1)); S=${S}a; sleep 0.5s; done
The server (terminal 1) displays the strings without problems.
But I tried to stress it a bit more (with sleep 0.01s) and around 220 it
starts to lag a bit (but is not stuck anyway, still delivers the
content).
Moreover, I tried to use wireshark in order to show ARP packets, flushed
my local arp cache but without more results : the IP is resolved once
and then no more, and the TCP connection still sticks later.
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to bridge-utils in Ubuntu.
https://bugs.launchpad.net/bugs/616064
Title:
Broken networking in kvm guests
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Could you try wget of large files instead of the echo? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Unfortunately i don't see anything telling in the wireshark log. Could you try out the instructions in comment #33 (taking into account the corrections in comment #35)? When a VM is running, what do ifconfig -a netstat -nr show? ** Changed in: qemu-kvm (Ubuntu) Assignee: Serge Hallyn (serge-hallyn) = (unassigned) ** Changed in: qemu-kvm (Ubuntu) Status: Confirmed = Incomplete -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Thanks for the info, Anthony. The wireshark info failed to download for me. Could you attach it here? My only guess ATM is that it is a bad interaction with the physical switch or router to which eth3 is attached. I'd ask you to try with different hardware, but I see you're running production VMs on there. Can you tell us exactly (names and models) what hardware is physically involved in the network leading up to eth3? Perhaps we can insert a logging layer 2 firewall between the switch and eth3 to see what is going on there. Do you have a spare machine with two NICs which could be used for that? ** Changed in: qemu-kvm (Ubuntu) Status: Incomplete = Confirmed ** Also affects: bridge-utils (Ubuntu) Importance: Undecided Status: New ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
** Changed in: linux (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Yes, here is the Wireshark capture. Actually, the eth3 is leading to a NetAsq U70-A firewall (8.0.3 firmware) on a DMZ subnetwork. The client I'm using is located on another subnetwork attached to that firewall. I can make some test if you want and send me directives... Moreover, the server has 4 eth, I can attach eth2 for instance directly in the same network as my client machine ? ** Attachment added: kvm net 2.pcap https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/616064/+attachment/2634585/+files/kvm%20net%202.pcap -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
marking 'confirmed' in bridge-utils just to get this off new bug lists. ** Changed in: bridge-utils (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to bridge-utils in Ubuntu. https://bugs.launchpad.net/bugs/616064 Title: Broken networking in kvm guests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/616064/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Note I made two mistakes in comment 33: 1. only do the 'ip link add type veth' once, not twice, and 2. the nc sender should send to port , not 999, to match the receiver port, of course. -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Serge, Apologies for the delay. This bug occurs on a production system for us so it's challenging to find time to run experiments. Now is a good time so I should have some new results for you shortly. -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Serge, I've completed (I hope) your suggestions. Unfortunately neither the new KVM nor the new kernel seemed to resolve the issue. More details below but I'm not sure where to take this from here. As you point out, it might be an issue with our environment. I doubt it's bad software installed on the host since the host is running bare minimum stock Lucid - just enough to act as a VM host. The bad networking switch/equipment is also a possibility, although I'm a bit confused since I would expect the NAT configuration to be indistinguishable from any basic machine on the network since the purpose of NAT is to hide the virtual LAN inside the host. Perhaps some information actually leaks from the NAT abstraction in packets sent out from the host? For my experiments today, I: 1. switched networking back to NAT and was able to reproduce the issue (svn co depot_tools) again 2. apt-add-repo ppa:serge-hallyn/lucid-kvm-test 3. apt-get update apt-get upgrade; this pulled in new versions of kvm, qemu-common, qemu-kvm, as well as a new kernel 2.6.32-24-server as part of the normal Lucid updates 4. rebooted the machine 5. reproduced the issue again 6. apt-add-repository ppa:kernel-ppa/pre-proposed 7. apt-get install linux-headers-server linux-image-server linux-server 8. rebooted the machine 9. reproduced the issue again I did not test new kernel + stock KVM due to time constraints. Please let me know if there are any other tests you'd like me to try and I'll try to schedule more maintenance time. Thanks, brian -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 616064] Re: Broken networking in kvm guests
Quoting Brian Fallik (616...@bugs.launchpad.net): 6. apt-add-repository ppa:kernel-ppa/pre-proposed Did you do an apt-get update here? If not, then you didn't get the updated kernels. 7. apt-get install linux-headers-server linux-image-server linux-server 8. rebooted the machine 9. reproduced the issue again I did not test new kernel + stock KVM due to time constraints. Yeah that wouldn't be worth it. Please let me know if there are any other tests you'd like me to try and I'll try to schedule more maintenance time. Ok, let's see if using just the bridge without kvm can reproduce the problem. If you can, please try the following: 1. Get a usable ns_exec: git clone git://git.sr71.net/~hallyn/cr_tests.git cd cr_tests git checkout ns_exec make ns_exec cp ns_exec /bin/ 2. Create a veth tunnel sudo ip link add type veth 3. Open two root terminals to configure a network namespace for our test terminal 1: ip link add type veth terminal 2: /bin/ns_exec -cmn /bin/bash echo $$ # call this $pid henceforth terminal 1: ifconfig veth0 0.0.0.0 up brctl addif virbr0 veth0 ip link set veth1 netns $pid # use pid from above terminal 2: ifconfig veth1 up dhclient veth1 Now, on either the host or on an external host, do nc -4 -k -l And from terminal 2 start dumping data while [ 1 ]; do echo ab | nc ipaddr 999 sleep 0.5s done where ipaddr is the host or external host's ip address. Does that eventually lock up too? -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 616064] Re: Broken networking in kvm guests
Serge, On Fri, Sep 10, 2010 at 12:56 PM, Serge Hallyn 616...@bugs.launchpad.net wrote: Quoting Brian Fallik (616...@bugs.launchpad.net): 6. apt-add-repository ppa:kernel-ppa/pre-proposed Did you do an apt-get update here? If not, then you didn't get the updated kernels. Yes, I just missed adding it to my notes. I manually verified that new versions of the packages with versions matching the PPAs were actually installed. Ok, let's see if using just the bridge without kvm can reproduce the problem. If you can, please try the following: 1. Get a usable ns_exec: git clone git://git.sr71.net/~hallyn/cr_tests.git cd cr_tests git checkout ns_exec make ns_exec cp ns_exec /bin/ 2. Create a veth tunnel sudo ip link add type veth 3. Open two root terminals to configure a network namespace for our test terminal 1: ip link add type veth terminal 2: /bin/ns_exec -cmn /bin/bash echo $$ # call this $pid henceforth terminal 1: ifconfig veth0 0.0.0.0 up brctl addif virbr0 veth0 ip link set veth1 netns $pid # use pid from above terminal 2: ifconfig veth1 up dhclient veth1 Now, on either the host or on an external host, do nc -4 -k -l And from terminal 2 start dumping data while [ 1 ]; do echo ab | nc ipaddr 999 sleep 0.5s done where ipaddr is the host or external host's ip address. Does that eventually lock up too? I'll try and test this soon but it may fall to next week. Thanks, brian -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Addendum to the above - it's been pointed out to me that scp DOES trigger the bug for you, so I was completely unable to reproduce your bug. My setup however was using the 0.12.5 kvm listed above, as well as the lucid pre-proposed kernels from https://launchpad.net/~kernel-ppa/+archive/pre-proposed. Can you please re-test with those? If the bug is fixed with those, then please let us know which fixed it - kernel drivers or updated qemu. If the bug persists, then I'd hazard the culprit is something else in your environment (a misbehaving switch, or even some ne'erdowell software on the host, though that is unlikely) because I think I've completely reproduced your settings. ** Changed in: qemu-kvm (Ubuntu) Status: Confirmed = Incomplete -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Alas, the unfirewalled machine I was going to try on has now been taken away. I may be able to reproduce this using user-space qemu on a VM on a cluster, and will try that next week if I have to, but in the meantime: I'm planning to try to upload qemu-kvm 0.12.5 to the lucid-backports ppa in the next few days. In the meantime, I've placed it in my own ppa - it should be done building in a few hours. Could you try the kvm out of: https://launchpad.net/~serge-hallyn/+archive/lucid-kvm-test (when 0.12.5 shows itself as built) and see if you still reproduce the problem? TBH I suspect it's actually a host kernel problem, but then if 0.12.5 still has the problem, we can look into that. -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
(Note to self - it may be worth-retrying with an i386 guest on previous setup) -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Serge, I switched networking back to NAT and repeated the svn co to verify the problem still exists. It does. Then I tried the wget operation 5 times and all of them succeeded. I also verified that we're using the same default.xml configuration. They're almost identical except for one interesting (maybe related?) observation. On my host, /etc/libvirt/qemu/networks/default contains: forward/ but your example shows: forward mode=nat/ However, if I run virsh net-dumpxml default default.xml and view the output file, I also see: forward mode=nat/ It may just be that, since http://libvirt.org/formatnetwork.html describes nat mode as the default, I suspect dumpxml outputs it but it isn't actually necessary in the xml. Side question: every time I switch networking times, the guest bumps ethernet device number (eth0, eth1, ...). This makes it annoying to reset networking to switch static/dhcp on the next reboot. Is there a method to ensure that the guest always uses eth0? brian -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
I switched networking back to NAT and repeated the svn co to verify the problem still exists. It does. Then I tried the wget operation 5 times and all of them succeeded. Ok, then I'll have to re-test on a machine where I'm not firewalled. Will post the results as soon as I have a chance to set such a machine up. Side question: every time I switch networking times, the guest bumps ethernet device number (eth0, eth1, ...). This makes it annoying to reset networking to switch static/dhcp on the next reboot. Is there a method to ensure that the guest always uses eth0? I'm nto sure what you mean by 'switch networking times' - but I assume you are talking about the guest's ethN? If so, libvirt is probably assigning it new MAC addresses, and udev in the guest is trying to be smart and helpful. Edit /etc/udev/rules.d/70-persistent-net.rules and put the latest mac address in place of the one for eth0, and delete the eth1..ethN entries. -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 616064] Re: Broken networking in kvm guests
On Mon, Aug 30, 2010 at 2:04 PM, Serge Hallyn 616...@bugs.launchpad.net wrote: Side question: every time I switch networking times, the guest bumps ethernet device number (eth0, eth1, ...). This makes it annoying to reset networking to switch static/dhcp on the next reboot. Is there a method to ensure that the guest always uses eth0? I'm nto sure what you mean by 'switch networking times' - but I assume you are talking about the guest's ethN? If so, libvirt is probably assigning it new MAC addresses, and udev in the guest is trying to be smart and helpful. Edit /etc/udev/rules.d/70-persistent-net.rules and put the latest mac address in place of the one for eth0, and delete the eth1..ethN entries. Yes, I meant every time I switch the networking the guests ethN keeps increasing. I'll take a look at /etc/udev/rules.d/70-persistent-net.rules. brian -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Brian, do you get the bug also when you just wget a large file (say wget http://archive.ubuntu.com/ubuntu/pool/restricted/b/bcmwl/bcmwl_5.60.48.36+bdcom.orig.tar.gz )? -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
I set up a NAT config using the following /etc/libvirt/qemu/networks/default.xml: network namedefault/name bridge name=virbr%d / forward mode=nat/ ip address=192.168.122.1 netmask=255.255.255.0 dhcp range start=192.168.122.2 end=192.168.122.254 / /dhcp /ip /network but with repeated wgets of ~1M was not able to get any hang. I'm really not sure where to go with this one, other than to look over your wireshark output a little more for new clues. -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Hi Brian, I think the two easiest things to test next are: 1. in my setup, stp was disabled on br0, in yours I think it was enabled. You might try your setup with stp disabled (which really should not be what you want, but it's a test). 2. you said In the scenario you suggested which seemed to work, tap1 was bridged directly to eth0 and used an IP address from the host's actual subnet. This is similar to: http://wiki.libvirt.org/page/Networking#Bridge Exactly, that was the guide I originally got the config from. So it's a good idea to test the setup in http://wiki.libvirt.org/page/Networking#Bridge on your machine. -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 616064] Re: Broken networking in kvm guests
On Thu, Aug 26, 2010 at 9:01 AM, Serge Hallyn 616...@bugs.launchpad.net wrote: 2. you said In the scenario you suggested which seemed to work, tap1 was bridged directly to eth0 and used an IP address from the host's actual subnet. This is similar to: http://wiki.libvirt.org/page/Networking#Bridge Exactly, that was the guide I originally got the config from. So it's a good idea to test the setup in http://wiki.libvirt.org/page/Networking#Bridge on your machine. Makes sense. I'm going to work on that right now. brian -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Serge, I've gotten the bridge configuration set up and everything seems stable and working after a few hours of testing. Obviously we'll keep watching it over the next few days. Thanks for the suggestion for the workaround. It'd be nice to understand the issue with the original configuration but it looks like bridging guests with the host's physical Ethernet device gives us a path forward. brian -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 616064] Re: Broken networking in kvm guests
Quoting Brian Fallik (616...@bugs.launchpad.net):
Serge,
I've gotten the bridge configuration set up and everything seems stable
and working after a few hours of testing. Obviously we'll keep watching
it over the next few days. Thanks for the suggestion for the
workaround. It'd be nice to understand the issue with the original
configuration but it looks like bridging guests with the host's physical
Ethernet device gives us a path forward.
In your bridging setup, do you have STP on or off? ('brctl show' should
tell you for sure)
Without a doubt, if using bridging is a workaround, we still need to
figure out why NATing is breaking for you.
Actually, hm. Can you also paste the output of
iptables -L
on the host?
--
Broken networking in kvm guests
https://bugs.launchpad.net/bugs/616064
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to qemu-kvm in ubuntu.
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Serge, $ brctl show bridge name bridge id STP enabled interfaces br0 8000.001a64b33b10 no eth0 vnet0 vnet1 virbr0 8000. yes In case it's not obvious, br0 is the bridge I manually created today. virbr0 is (i think) the bridge created by libvirt which I had used previously. virbr0 was never bound to eth0 - it was used for the 'virtual network' scenario. Attached it the output of iptables -L as of right now, but I *think* this output may not reflect the same configuration when libvirt is managing the virtual network. In the case I think libvirt manipulates the network configuration, including iptables, to support the virtual LAN. We're in the middle of a test build on the guest which I'd prefer not to interrupt, but if you'd like me to run another test with the networking switched back to virtual LAN, I can do that later today. I can also recapture iptables -L at that point. brian ** Attachment added: iptables -L https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/616064/+attachment/1521218/+files/iptables_L.txt -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
can you try 'brctl stp br0 on' and see if all still works? -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Thanks, Brian - no, next it's up to me to try to reproduce with the NAT config. I need to rebuilt my test system first though. Hoping to have results in the afternoon. ** Changed in: qemu-kvm (Ubuntu) Assignee: (unassigned) = Serge Hallyn (serge-hallyn) -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
I failed to reproduce this. Here is how I tried. I have a 10.04.1 lucid host. My /etc/network/interfaces looks like: # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet manual auto br0 iface br0 inet dhcp bridge_ports eth0 bridge_stp off bridge_fd 0 bridge_maxwait 0 I started two 10.04.1 lucid guests, with the following command lines: kvm -hda m1.img -net nic,macaddr=52:54:8b:3d:1a:82,model=virtio -net tap,ifname=tap1,script=no,downscript=no -vnc :1 kvm -hda m2.img -net nic,macaddr=52:54:8b:3d:1a:83,model=virtio -net tap,ifname=tap2,script=no,downscript=no -vnc :2 Then, on the host, I added the tap devices to br0: /sbin/ifconfig tap1 0.0.0.0 up brctl addif br0 tap1 /sbin/ifconfig tap2 0.0.0.0 up brctl addif br0 tap2 and ran 'dhclient eth0' on each guest. Then on each guest I did: apt-get install subversion svn co http://src.chromium.org/svn/trunk/tools/depot_tools It succeeded on both. My impression from the report is that this should fail very quickly. If I should keep trying in a loop please let me know. I wonder whether vlan tagging (which you are doing) is involved in the problem... -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
Re: [Bug 616064] Re: Broken networking in kvm guests
Quoting Brian Fallik (616...@bugs.launchpad.net): Serge - my uploads in comment #5 and comment #6 should have been while guest #1 (Lucid) was running. This problem occurs regardless of 1 or 2 guests are running. I've stopped launching the second guest (Hardy) since the number of guests seemed irrelevant. But I don't see a tap0 or tap1 in the output. I for some reason thought Id seen you say you were using 10.04.1 as guest as well. Let me try a hardy guest! -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Tried to reproduce with a hardy guest, but that failed as well. Did this use to work before a particular upgrade, or is this the first you've tried? -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Serge, I think I'm starting to understand this issue and I'm fairly certain the problem is in libvirt. First off, it seems like I can repeatedly checkout depot_tools in a KVM using a setup that I think matches your example except for a new (and different) error. My first step was shutdown libvirt-bin. Then I modified interfaces on the host to look like: # The primary network interface auto eth0 iface eth0 inet manual auto br0 iface br0 inet static address 10.10.224.21 netmask 255.255.255.0 gateway 10.10.224.254 bridge_ports eth0 bridge_stp off bridge_fd 0 bridge_maxwait 0 and restarted networking. Next I: ifconfig tap1 0.0.0.0 up brctl addif br0 tap1 and invoked KVM like so: /usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 8192 -smp 8 -name hound-lucid -uuid 0e32438c-cb4d-1eb9-a560-f25c89c1dbf6 -boot c -drive file=/opt/hound-lucid,if=virtio,index=0,boot=on -drive if=ide,media=cdrom,index=2 -net nic,macaddr=52:54:8b:3d:1a:82,model=virtio -net tap,ifname=tap1,script=no,downscript=no -vnc :1 Finally, I used vnc on the guest to statically configure networking. With this set up I tried the svn co of depot tools 3 times. However, the guest now seems to lose all networking connectivity after a few short minutes. eth0 on the host loses its IP configuration. The workaround is to /etc/init/d/networking restart. This symptom is different from the error originally described in this bug but may be related to some manual step in my testing. Back to the original issue (failed 'svn co'), the most notable difference between the working setup and the broken one is the network architecture. In my original posts, tap0 and tap1 formed a private LAN hidden behind the host's eth0 via NAT and IP forwarding and using a separate IP address space. This was managed by libvirt and is described here: http://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29 In the scenario you suggested which seemed to work, tap1 was bridged directly to eth0 and used an IP address from the host's actual subnet. This is similar to: http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_physical_device.22.29 I'm not sure if this problem is the networking architecture itself or libvirt's management of it. I'm not 100% sure what my next steps should be. Is there a different mechanism to manage KVM virtual networks without using libvirt? Or should I give up on that approach and use shared physical device, in which case I need to allocate additional IPs for the guests and ensure that they're properly secured from the LAN. brian -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Brian, Can you please include the info from: brctl show ifconfig -a Both those commands should be executed on the host. Thanks! ** Changed in: qemu-kvm (Ubuntu) Status: Confirmed = Incomplete -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
** Attachment added: brctl show https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/616064/+attachment/1516198/+files/brctl_show.log -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
** Attachment added: ifconfig -a https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/616064/+attachment/1516203/+files/ifconfig_a.log -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
** Changed in: qemu-kvm (Ubuntu) Status: Incomplete = Confirmed -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Re: I can see these RST packets via Wireshark but can't explain their source. Where do those RST packets appear to come from ? ** Changed in: qemu-kvm (Ubuntu) Importance: Undecided = Medium ** Changed in: qemu-kvm (Ubuntu) Status: New = Incomplete -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
Here's example failure output from running the command inside the Lucid guest: $ svn co http://src.chromium.org/svn/trunk/tools/depot_tools svn: PROPFIND of '/svn/trunk/tools/depot_tools': Could not read status line: Connection reset by peer (http://src.chromium.org) The same command is always successful in the Lucid host. Here is some tshark output from the event. 10.10.224.21 is eth0 in the host and 192.168.122.218 is eth0 in the guest. bfal...@hound:~$ sudo tshark -i any host src.chromium.org Running as user root and group root. This could be dangerous. Capturing on Pseudo-device that captures on all interfaces 0.00 72.14.204.113 - 192.168.122.218 TCP http 60574 [ACK] Seq=1 Ack=1 Win=240 Len=0 TSV=1180982073 TSER=100666 0.02 72.14.204.113 - 192.168.122.218 TCP [TCP Dup ACK 1#1] http 60574 [ACK] Seq=1 Ack=1 Win=240 Len=0 TSV=1180982073 TSER=100666 0.017332 72.14.204.113 - 10.10.224.21 HTTP/XML HTTP/1.1 207 Multi-Status 0.017338 72.14.204.113 - 192.168.122.218 HTTP/XML HTTP/1.1 207 Multi-Status 0.017340 72.14.204.113 - 192.168.122.218 HTTP/XML [TCP Out-Of-Order] HTTP/1.1 207 Multi-Status 0.017856 192.168.122.218 - 72.14.204.113 TCP [TCP segment of a reassembled PDU] 0.017866 192.168.122.218 - 72.14.204.113 TCP [TCP Out-Of-Order] [TCP segment of a reassembled PDU] 0.017874 10.10.224.21 - 72.14.204.113 TCP [TCP segment of a reassembled PDU] 0.017884 192.168.122.218 - 72.14.204.113 HTTP/XML PROPFIND /svn/!svn/bln/55717 HTTP/1.1 0.017894 192.168.122.218 - 72.14.204.113 TCP [TCP Out-Of-Order] [TCP segment of a reassembled PDU] 0.017898 10.10.224.21 - 72.14.204.113 HTTP/XML PROPFIND /svn/!svn/bln/55717 HTTP/1.1 0.044971 72.14.204.113 - 10.10.224.21 TCP http 60574 [ACK] Seq=537 Ack=565 Win=273 Len=0 TSV=1180982119 TSER=100677 0.044978 72.14.204.113 - 192.168.122.218 TCP http 60574 [ACK] Seq=537 Ack=565 Win=273 Len=0 TSV=1180982119 TSER=100677 0.044980 72.14.204.113 - 192.168.122.218 TCP [TCP Dup ACK 13#1] http 60574 [ACK] Seq=537 Ack=565 Win=273 Len=0 TSV=1180982119 TSER=100677 0.086739 72.14.204.113 - 10.10.224.21 HTTP/XML HTTP/1.1 207 Multi-Status 0.086745 72.14.204.113 - 192.168.122.218 HTTP/XML HTTP/1.1 207 Multi-Status 0.086747 72.14.204.113 - 192.168.122.218 HTTP/XML [TCP Out-Of-Order] HTTP/1.1 207 Multi-Status 0.087307 192.168.122.218 - 72.14.204.113 TCP [TCP segment of a reassembled PDU] 0.087317 192.168.122.218 - 72.14.204.113 TCP [TCP Out-Of-Order] [TCP segment of a reassembled PDU] 0.087324 10.10.224.21 - 72.14.204.113 TCP [TCP segment of a reassembled PDU] 0.087334 192.168.122.218 - 72.14.204.113 HTTP/XML PROPFIND /svn/trunk/tools/depot_tools HTTP/1.1 0.087341 192.168.122.218 - 72.14.204.113 TCP [TCP Out-Of-Order] [TCP segment of a reassembled PDU] 0.087344 10.10.224.21 - 72.14.204.113 HTTP/XML PROPFIND /svn/trunk/tools/depot_tools HTTP/1.1 0.114646 72.14.204.113 - 10.10.224.21 TCP http 60574 [ACK] Seq=1132 Ack=1290 Win=307 Len=0 TSV=1180982189 TSER=100694 0.114653 72.14.204.113 - 192.168.122.218 TCP http 60574 [ACK] Seq=1132 Ack=1290 Win=307 Len=0 TSV=1180982189 TSER=100694 0.114655 72.14.204.113 - 192.168.122.218 TCP [TCP Dup ACK 25#1] http 60574 [ACK] Seq=1132 Ack=1290 Win=307 Len=0 TSV=1180982189 TSER=100694 0.132462 72.14.204.113 - 10.10.224.21 HTTP/XML HTTP/1.1 207 Multi-Status 0.132469 72.14.204.113 - 192.168.122.218 HTTP/XML HTTP/1.1 207 Multi-Status 0.132470 72.14.204.113 - 192.168.122.218 HTTP/XML [TCP Out-Of-Order] HTTP/1.1 207 Multi-Status 0.133009 192.168.122.218 - 72.14.204.113 TCP [TCP segment of a reassembled PDU] 0.133019 192.168.122.218 - 72.14.204.113 TCP [TCP Out-Of-Order] [TCP segment of a reassembled PDU] 0.133026 10.10.224.21 - 72.14.204.113 TCP [TCP segment of a reassembled PDU] 0.133036 192.168.122.218 - 72.14.204.113 HTTP/XML PROPFIND /svn/!svn/vcc/default HTTP/1.1 0.133039 192.168.122.218 - 72.14.204.113 TCP [TCP Out-Of-Order] [TCP segment of a reassembled PDU] 0.133043 10.10.224.21 - 72.14.204.113 HTTP/XML PROPFIND /svn/!svn/vcc/default HTTP/1.1 0.158530 72.14.204.113 - 10.10.224.21 TCP http 60574 [ACK] Seq=1988 Ack=1870 Win=340 Len=0 TSV=1180982233 TSER=100706 0.158538 72.14.204.113 - 192.168.122.218 TCP http 60574 [ACK] Seq=1988 Ack=1870 Win=340 Len=0 TSV=1180982233 TSER=100706 0.158540 72.14.204.113 - 192.168.122.218 TCP [TCP Dup ACK 37#1] http 60574 [ACK] Seq=1988 Ack=1870 Win=340 Len=0 TSV=1180982233 TSER=100706 0.178445 72.14.204.113 - 10.10.224.21 HTTP/XML HTTP/1.1 207 Multi-Status 0.178451 72.14.204.113 - 192.168.122.218 HTTP/XML HTTP/1.1 207 Multi-Status 0.178453 72.14.204.113 - 192.168.122.218 HTTP/XML [TCP Out-Of-Order] HTTP/1.1 207 Multi-Status 0.178968 192.168.122.218 - 72.14.204.113 TCP [TCP segment of a reassembled PDU] 0.178978 192.168.122.218 - 72.14.204.113 TCP [TCP Out-Of-Order] [TCP segment of a reassembled PDU] 0.178984 10.10.224.21 - 72.14.204.113 TCP [TCP segment of a reassembled PDU]
[Bug 616064] Re: Broken networking in kvm guests
** Changed in: qemu-kvm (Ubuntu) Status: Incomplete = Confirmed -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 616064] Re: Broken networking in kvm guests
** Attachment added: BootDmesg.txt http://launchpadlibrarian.net/53444870/BootDmesg.txt ** Attachment added: CurrentDmesg.txt http://launchpadlibrarian.net/53444871/CurrentDmesg.txt ** Attachment added: Dependencies.txt http://launchpadlibrarian.net/53444872/Dependencies.txt ** Attachment added: Lspci.txt http://launchpadlibrarian.net/53444873/Lspci.txt ** Attachment added: Lsusb.txt http://launchpadlibrarian.net/53444874/Lsusb.txt ** Attachment added: ProcCpuinfo.txt http://launchpadlibrarian.net/53444875/ProcCpuinfo.txt ** Attachment added: ProcInterrupts.txt http://launchpadlibrarian.net/53444876/ProcInterrupts.txt ** Attachment added: ProcModules.txt http://launchpadlibrarian.net/53444877/ProcModules.txt ** Attachment added: RelatedPackageVersions.txt http://launchpadlibrarian.net/53444878/RelatedPackageVersions.txt ** Attachment added: UdevDb.txt http://launchpadlibrarian.net/53444879/UdevDb.txt ** Attachment added: UdevLog.txt http://launchpadlibrarian.net/53444880/UdevLog.txt -- Broken networking in kvm guests https://bugs.launchpad.net/bugs/616064 You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to qemu-kvm in ubuntu. -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
