Am 12.02.2013 um 10:08 schrieb "Michael S. Tsirkin" <m...@redhat.com>:
> On Tue, Feb 12, 2013 at 08:06:04AM +0100, Peter Lieven wrote: >> On 23.01.2013 11:03, Michael S. Tsirkin wrote: >>> On Tue, Jan 22, 2013 at 10:04:07AM +0100, Peter Lieven wrote: >>>> On 23.11.2012 12:01, Michael S. Tsirkin wrote: >>>>> On Fri, Nov 23, 2012 at 10:41:21AM +0100, Peter Lieven wrote: >>>>>> >>>>>> Am 23.11.2012 um 08:02 schrieb Stefan Hajnoczi: >>>>>> >>>>>>> On Thu, Nov 22, 2012 at 03:29:52PM +0100, Peter Lieven wrote: >>>>>>>> is anyone aware of a problem with the linux network bridge that in >>>>>>>> very rare circumstances stops >>>>>>>> a bridge from sending pakets to a tap device? >>>>>>>> >>>>>>>> My problem occurs in conjunction with vanilla qemu-kvm-1.2.0 and >>>>>>>> Ubuntu Kernel 3.2.0-34.53 >>>>>>>> which is based on Linux 3.2.33. >>>>>>>> >>>>>>>> I was not yet able to reproduce the issue, it happens in really rare >>>>>>>> cases. The symptom is that >>>>>>>> the tap does not have any TX packets. RX is working fine. I see the >>>>>>>> packets coming in at >>>>>>>> the physical interface on the host, but they are not forwarded to the >>>>>>>> tap interface. >>>>>>>> The bridge itself has learnt the mac address of the vServer that is >>>>>>>> connected to the tap interface. >>>>>>>> It does not help to toggle the bridge link status, the tap interface >>>>>>>> status or the interface in the vServer. >>>>>>>> It seems that problem occurs if a tap interface that has previously >>>>>>>> been used, but set to nonpersistent >>>>>>>> is set persistent again and then is by chance assigned to the same >>>>>>>> vServer (=same mac address on same >>>>>>>> bridge) again. Unfortunately it seems not to be reproducible. >>>>>>> >>>>>>> Not sure but this patch from Michael Tsirkin may help - it solves an >>>>>>> issue with persistent tap devices: >>>>>>> >>>>>>> http://patchwork.ozlabs.org/patch/198598/ >>>>>> >>>>>> Hi Stefan, >>>>>> >>>>>> thanks for the pointer. I have seen this patch, but I have neglected it >>>>>> because it was dealing >>>>>> with persistent taps. But maybe the taps in the kernel are not deleted >>>>>> directly. >>>>>> Can you remember what the syptomps of the above issue have been? Sorry >>>>>> for >>>>>> being vague, but I currently have no clue whats going on. >>>>>> >>>>>> Can someone who has more internal knowledge of the bridging/tap code say >>>>>> if qemu can >>>>>> be responsible at all if the tap device is not receiving packets from >>>>>> the bridge. >>>>>> >>>>>> If I have the following config. Lets say packets coming in via physical >>>>>> interface eth1.123, >>>>>> and a bridge called br123.I further have a virtual machine with tap0. >>>>>> Both eth1.123 >>>>>> and tap0 are member of br123. >>>>>> >>>>>> If the issue occurs the vServer has no network connectivity inbound. If >>>>>> I sent a ping >>>>>> from the vServer I see it on tap0 and leaving on eth1.123. I see further >>>>>> the arp reply coming >>>>>> in via eth1.123, but the reply can't be seen on tap0. >>>>>> >>>>>> Peter >>>>> >>>>> If guest is not consuming packets, a TX queue in tap device >>>>> will with time overrun (there's space for 1000 packets there). >>>>> This is code from tun: >>>>> >>>>> if (skb_queue_len(&tfile->socket.sk->sk_receive_queue) >>>>>> = dev->tx_queue_len / tun->numqueues){ >>>>> if (!(tun->flags & TUN_ONE_QUEUE)) { >>>>> /* Normal queueing mode. */ >>>>> /* Packet scheduler handles dropping of further >>>>> * packets. */ >>>>> netif_stop_subqueue(dev, txq); >>>>> >>>>> /* We won't see all dropped packets >>>>> * individually, so overrun >>>>> * error is more appropriate. */ >>>>> dev->stats.tx_fifo_errors++; >>>>> >>>>> >>>>> So you can detect that this triggered by looking at fifo errors counter >>>>> in device. >>>>> >>>>> Once this happens TX queue is stopped, then you hit this path: >>>>> >>>>> if (!netif_xmit_stopped(txq)) { >>>>> __this_cpu_inc(xmit_recursion); >>>>> rc = dev_hard_start_xmit(skb, dev, txq); >>>>> __this_cpu_dec(xmit_recursion); >>>>> if (dev_xmit_complete(rc)) { >>>>> HARD_TX_UNLOCK(dev, txq); >>>>> goto out; >>>>> } >>>>> } >>>>> >>>>> so packets are not passed to device anymore. >>>>> It will stay this way until guest consumes some packets and >>>>> queue is restarted. >>>> >>>> After some time I again have a vServer in this state. It seems not like >>>> there >>>> are no TX errors. >>>> >>>> # ifconfig tap10 >>>> tap10 Link encap:Ethernet HWaddr 7a:59:20:6f:e7:e5 >>>> inet6 addr: fe80::7859:20ff:fe6f:e7e5/64 Scope:Link >>>> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 >>>> RX packets:197431 errors:0 dropped:0 overruns:0 frame:0 >>>> TX packets:264309 errors:0 dropped:0 overruns:2 carrier:0 >>>> collisions:0 txqueuelen:500 >>>> RX bytes:13842063 (13.8 MB) TX bytes:35092821 (35.0 MB) >>>> >>>> It seems like the bridge is not forwarding any packets to the tap device >>>> anymore altough it has learnt >>>> the MAC-Adresses and there are also broadcast packets coming in. >>>> >>>> Any more ideas where I could debug? >>>> >>>> Peter >>>> >>>>> >>>>>>> >>>>>>> Stefan >>> >>> Hmm. So there are two overrun errors that triggered, so >>> it's possible after the second one the queue got stuck in an xoff state. >>> You'd have to use something like systemtap or kdb to poke at the >>> queue state to see whether xoff flag is set and/or look >>> at the receive queue length. >>> >>> For future, we can try to set TUN_ONE_QUEUE flag on the interface, >>> or try applying this patch >>> 5d097109257c03a71845729f8db6b5770c4bbedc >>> in kernel see if this helps. >>> >> >> If have set this option for 2 weeks now and not seen this problem again. >> How does this flag work with the recently added tap multiqueue support? >> >> Peter > > This will be the only option in 3.8. Ok, but wouldn`t it be good to set it in qemu for kernels <3.8? Peter > > -- > MST
