On Tue, Feb 12, 2013 at 08:06:04AM +0100, Peter Lieven wrote: > On 23.01.2013 11:03, Michael S. Tsirkin wrote: > >On Tue, Jan 22, 2013 at 10:04:07AM +0100, Peter Lieven wrote: > >>On 23.11.2012 12:01, Michael S. Tsirkin wrote: > >>>On Fri, Nov 23, 2012 at 10:41:21AM +0100, Peter Lieven wrote: > >>>> > >>>>Am 23.11.2012 um 08:02 schrieb Stefan Hajnoczi: > >>>> > >>>>>On Thu, Nov 22, 2012 at 03:29:52PM +0100, Peter Lieven wrote: > >>>>>>is anyone aware of a problem with the linux network bridge that in very > >>>>>>rare circumstances stops > >>>>>>a bridge from sending pakets to a tap device? > >>>>>> > >>>>>>My problem occurs in conjunction with vanilla qemu-kvm-1.2.0 and Ubuntu > >>>>>>Kernel 3.2.0-34.53 > >>>>>>which is based on Linux 3.2.33. > >>>>>> > >>>>>>I was not yet able to reproduce the issue, it happens in really rare > >>>>>>cases. The symptom is that > >>>>>>the tap does not have any TX packets. RX is working fine. I see the > >>>>>>packets coming in at > >>>>>>the physical interface on the host, but they are not forwarded to the > >>>>>>tap interface. > >>>>>>The bridge itself has learnt the mac address of the vServer that is > >>>>>>connected to the tap interface. > >>>>>>It does not help to toggle the bridge link status, the tap interface > >>>>>>status or the interface in the vServer. > >>>>>>It seems that problem occurs if a tap interface that has previously > >>>>>>been used, but set to nonpersistent > >>>>>>is set persistent again and then is by chance assigned to the same > >>>>>>vServer (=same mac address on same > >>>>>>bridge) again. Unfortunately it seems not to be reproducible. > >>>>> > >>>>>Not sure but this patch from Michael Tsirkin may help - it solves an > >>>>>issue with persistent tap devices: > >>>>> > >>>>>http://patchwork.ozlabs.org/patch/198598/ > >>>> > >>>>Hi Stefan, > >>>> > >>>>thanks for the pointer. I have seen this patch, but I have neglected it > >>>>because it was dealing > >>>>with persistent taps. But maybe the taps in the kernel are not deleted > >>>>directly. > >>>>Can you remember what the syptomps of the above issue have been? Sorry for > >>>>being vague, but I currently have no clue whats going on. > >>>> > >>>>Can someone who has more internal knowledge of the bridging/tap code say > >>>>if qemu can > >>>>be responsible at all if the tap device is not receiving packets from the > >>>>bridge. > >>>> > >>>>If I have the following config. Lets say packets coming in via physical > >>>>interface eth1.123, > >>>>and a bridge called br123.I further have a virtual machine with tap0. > >>>>Both eth1.123 > >>>>and tap0 are member of br123. > >>>> > >>>>If the issue occurs the vServer has no network connectivity inbound. If I > >>>>sent a ping > >>>>from the vServer I see it on tap0 and leaving on eth1.123. I see further > >>>>the arp reply coming > >>>>in via eth1.123, but the reply can't be seen on tap0. > >>>> > >>>>Peter > >>> > >>>If guest is not consuming packets, a TX queue in tap device > >>>will with time overrun (there's space for 1000 packets there). > >>>This is code from tun: > >>> > >>> if (skb_queue_len(&tfile->socket.sk->sk_receive_queue) > >>> >= dev->tx_queue_len / tun->numqueues){ > >>> if (!(tun->flags & TUN_ONE_QUEUE)) { > >>> /* Normal queueing mode. */ > >>> /* Packet scheduler handles dropping of further > >>> * packets. */ > >>> netif_stop_subqueue(dev, txq); > >>> > >>> /* We won't see all dropped packets > >>> * individually, so overrun > >>> * error is more appropriate. */ > >>> dev->stats.tx_fifo_errors++; > >>> > >>> > >>>So you can detect that this triggered by looking at fifo errors counter in > >>>device. > >>> > >>>Once this happens TX queue is stopped, then you hit this path: > >>> > >>> if (!netif_xmit_stopped(txq)) { > >>> __this_cpu_inc(xmit_recursion); > >>> rc = dev_hard_start_xmit(skb, dev, txq); > >>> __this_cpu_dec(xmit_recursion); > >>> if (dev_xmit_complete(rc)) { > >>> HARD_TX_UNLOCK(dev, txq); > >>> goto out; > >>> } > >>> } > >>> > >>>so packets are not passed to device anymore. > >>>It will stay this way until guest consumes some packets and > >>>queue is restarted. > >> > >>After some time I again have a vServer in this state. It seems not like > >>there > >>are no TX errors. > >> > >># ifconfig tap10 > >>tap10 Link encap:Ethernet HWaddr 7a:59:20:6f:e7:e5 > >> inet6 addr: fe80::7859:20ff:fe6f:e7e5/64 Scope:Link > >> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 > >> RX packets:197431 errors:0 dropped:0 overruns:0 frame:0 > >> TX packets:264309 errors:0 dropped:0 overruns:2 carrier:0 > >> collisions:0 txqueuelen:500 > >> RX bytes:13842063 (13.8 MB) TX bytes:35092821 (35.0 MB) > >> > >>It seems like the bridge is not forwarding any packets to the tap device > >>anymore altough it has learnt > >>the MAC-Adresses and there are also broadcast packets coming in. > >> > >>Any more ideas where I could debug? > >> > >>Peter > >> > >>> > >>>>> > >>>>>Stefan > > > >Hmm. So there are two overrun errors that triggered, so > >it's possible after the second one the queue got stuck in an xoff state. > >You'd have to use something like systemtap or kdb to poke at the > >queue state to see whether xoff flag is set and/or look > >at the receive queue length. > > > >For future, we can try to set TUN_ONE_QUEUE flag on the interface, > >or try applying this patch > >5d097109257c03a71845729f8db6b5770c4bbedc > >in kernel see if this helps. > > > > If have set this option for 2 weeks now and not seen this problem again. > How does this flag work with the recently added tap multiqueue support? > > Peter
This will be the only option in 3.8. -- MST
