Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 03/03/2014 04:13 AM, Christian Borntraeger wrote: > On 02/03/14 02:21, Vlad Yasevich wrote: >> On 03/01/2014 02:27 PM, Vlad Yasevich wrote: >>> On 03/01/2014 06:15 AM, Christian Borntraeger wrote: On 28/02/14 23:14, Vlad Yasevich wrote: > On 02/27/2014 03:52 PM, Christian Borntraeger wrote: >> Vlad, >> >> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 >> macvtap: Add support of packet capture on macvtap device. >> >> causes a performance regression for iperf traffic between two KVM guests >> on my s390 system. Both guests are connected via two macvtaps on the >> same OSA >> network card. >> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get >> ~4Gbit/sec >> >> Latency seems to be unchanges (uperf 1byte ping pong). >> >> According to ifconfig in the guest, I have ~ 1500 bytes per packet with >> this >> patch and ~ 4 bytes without. So for some reason this patch causes >> the >> network stack to do segmentation. (the guest kernel stays the same, only >> host >> kernel is changed). >> >> Any ideas? > > I am looking. It shouldn't cause addition segmentations and when I ran > netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. >>> >>> Could you post 'ethtool -k' output for both lower-level device and the >>> macvtap device? >>> >>> Thanks >>> -vlad >>> >> >> Ok. I think I see what's happening. Since you turn off offloads on >> lower device, that's propagated to macvlan device. As a result, when >> when we call dev_queue_xmit on the vlan->dev, we end up segmenting since >> lower level says it does support segmentation. >> >> One way to fix this is to never disable offloads on macvlan. macvlan >> will always try to use __dev_queue_xmit() with it's lower device, so any >> segmentation can happen there. > > If you have anything that I should test, let me know. Hi Christian Just sent out a patch to fix this. I tried it with namespaces and kvm guests and it seems to restore performance for me. Please give it a try. Thanks -vlad > > Christian > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 02/03/14 02:21, Vlad Yasevich wrote: > On 03/01/2014 02:27 PM, Vlad Yasevich wrote: >> On 03/01/2014 06:15 AM, Christian Borntraeger wrote: >>> On 28/02/14 23:14, Vlad Yasevich wrote: On 02/27/2014 03:52 PM, Christian Borntraeger wrote: > Vlad, > > commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 > macvtap: Add support of packet capture on macvtap device. > > causes a performance regression for iperf traffic between two KVM guests > on my s390 system. Both guests are connected via two macvtaps on the same > OSA > network card. > Before that patch I get ~20 Gbit/sec between two guests, afterwards I get > ~4Gbit/sec > > Latency seems to be unchanges (uperf 1byte ping pong). > > According to ifconfig in the guest, I have ~ 1500 bytes per packet with > this > patch and ~ 4 bytes without. So for some reason this patch causes the > network stack to do segmentation. (the guest kernel stays the same, only > host > kernel is changed). > > Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. >>> >>> Dont know if the different bytes/packets ratio is really the reason or >>> just a side effect. As a hint: the underlying network device does not >>> support >>> segmentation, but this should not matter for traffic between to guests. >> >> Could you post 'ethtool -k' output for both lower-level device and the >> macvtap device? >> >> Thanks >> -vlad >> > > Ok. I think I see what's happening. Since you turn off offloads on > lower device, that's propagated to macvlan device. As a result, when > when we call dev_queue_xmit on the vlan->dev, we end up segmenting since > lower level says it does support segmentation. > > One way to fix this is to never disable offloads on macvlan. macvlan > will always try to use __dev_queue_xmit() with it's lower device, so any > segmentation can happen there. If you have anything that I should test, let me know. Christian -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 01/03/14 20:27, Vlad Yasevich wrote: > On 03/01/2014 06:15 AM, Christian Borntraeger wrote: >> On 28/02/14 23:14, Vlad Yasevich wrote: >>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? >>> >>> I am looking. It shouldn't cause addition segmentations and when I ran >>> netperf on the code I didn't see any difference in the throughput. >> >> Dont know if the different bytes/packets ratio is really the reason or >> just a side effect. As a hint: the underlying network device does not support >> segmentation, but this should not matter for traffic between to guests. > > Could you post 'ethtool -k' output for both lower-level device and the > macvtap device? Features for eth0: rx-checksumming: off [fixed] tx-checksumming: off tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off [fixed] tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] Features for macvtap1: rx-checksumming: off [fixed] tx-checksumming: off tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off [fixed] tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: on [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 01/03/14 20:27, Vlad Yasevich wrote: On 03/01/2014 06:15 AM, Christian Borntraeger wrote: On 28/02/14 23:14, Vlad Yasevich wrote: On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. Could you post 'ethtool -k' output for both lower-level device and the macvtap device? Features for eth0: rx-checksumming: off [fixed] tx-checksumming: off tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off [fixed] tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] Features for macvtap1: rx-checksumming: off [fixed] tx-checksumming: off tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off [fixed] tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off [fixed] tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: on [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 02/03/14 02:21, Vlad Yasevich wrote: On 03/01/2014 02:27 PM, Vlad Yasevich wrote: On 03/01/2014 06:15 AM, Christian Borntraeger wrote: On 28/02/14 23:14, Vlad Yasevich wrote: On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. Could you post 'ethtool -k' output for both lower-level device and the macvtap device? Thanks -vlad Ok. I think I see what's happening. Since you turn off offloads on lower device, that's propagated to macvlan device. As a result, when when we call dev_queue_xmit on the vlan-dev, we end up segmenting since lower level says it does support segmentation. One way to fix this is to never disable offloads on macvlan. macvlan will always try to use __dev_queue_xmit() with it's lower device, so any segmentation can happen there. If you have anything that I should test, let me know. Christian -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 03/03/2014 04:13 AM, Christian Borntraeger wrote: On 02/03/14 02:21, Vlad Yasevich wrote: On 03/01/2014 02:27 PM, Vlad Yasevich wrote: On 03/01/2014 06:15 AM, Christian Borntraeger wrote: On 28/02/14 23:14, Vlad Yasevich wrote: On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. Could you post 'ethtool -k' output for both lower-level device and the macvtap device? Thanks -vlad Ok. I think I see what's happening. Since you turn off offloads on lower device, that's propagated to macvlan device. As a result, when when we call dev_queue_xmit on the vlan-dev, we end up segmenting since lower level says it does support segmentation. One way to fix this is to never disable offloads on macvlan. macvlan will always try to use __dev_queue_xmit() with it's lower device, so any segmentation can happen there. If you have anything that I should test, let me know. Hi Christian Just sent out a patch to fix this. I tried it with namespaces and kvm guests and it seems to restore performance for me. Please give it a try. Thanks -vlad Christian -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 03/01/2014 02:27 PM, Vlad Yasevich wrote: > On 03/01/2014 06:15 AM, Christian Borntraeger wrote: >> On 28/02/14 23:14, Vlad Yasevich wrote: >>> On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? >>> >>> I am looking. It shouldn't cause addition segmentations and when I ran >>> netperf on the code I didn't see any difference in the throughput. >> >> Dont know if the different bytes/packets ratio is really the reason or >> just a side effect. As a hint: the underlying network device does not support >> segmentation, but this should not matter for traffic between to guests. > > Could you post 'ethtool -k' output for both lower-level device and the > macvtap device? > > Thanks > -vlad > Ok. I think I see what's happening. Since you turn off offloads on lower device, that's propagated to macvlan device. As a result, when when we call dev_queue_xmit on the vlan->dev, we end up segmenting since lower level says it does support segmentation. One way to fix this is to never disable offloads on macvlan. macvlan will always try to use __dev_queue_xmit() with it's lower device, so any segmentation can happen there. -vlad >> >> Maybe you remember, we had a similar situation with commit >> 3e4f8b787370978733ca6cae452720a4f0c296b8 >> (macvtap: Perform GSO on forwarding path), the setup is basically the same. >> >> >> Christian >> >> -- >> To unsubscribe from this list: send the line "unsubscribe netdev" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 03/01/2014 06:15 AM, Christian Borntraeger wrote: > On 28/02/14 23:14, Vlad Yasevich wrote: >> On 02/27/2014 03:52 PM, Christian Borntraeger wrote: >>> Vlad, >>> >>> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 >>> macvtap: Add support of packet capture on macvtap device. >>> >>> causes a performance regression for iperf traffic between two KVM guests >>> on my s390 system. Both guests are connected via two macvtaps on the same >>> OSA >>> network card. >>> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get >>> ~4Gbit/sec >>> >>> Latency seems to be unchanges (uperf 1byte ping pong). >>> >>> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this >>> patch and ~ 4 bytes without. So for some reason this patch causes the >>> network stack to do segmentation. (the guest kernel stays the same, only >>> host >>> kernel is changed). >>> >>> Any ideas? >> >> I am looking. It shouldn't cause addition segmentations and when I ran >> netperf on the code I didn't see any difference in the throughput. > > Dont know if the different bytes/packets ratio is really the reason or > just a side effect. As a hint: the underlying network device does not support > segmentation, but this should not matter for traffic between to guests. Could you post 'ethtool -k' output for both lower-level device and the macvtap device? Thanks -vlad > > Maybe you remember, we had a similar situation with commit > 3e4f8b787370978733ca6cae452720a4f0c296b8 > (macvtap: Perform GSO on forwarding path), the setup is basically the same. > > > Christian > > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 28/02/14 23:14, Vlad Yasevich wrote: > On 02/27/2014 03:52 PM, Christian Borntraeger wrote: >> Vlad, >> >> commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 >> macvtap: Add support of packet capture on macvtap device. >> >> causes a performance regression for iperf traffic between two KVM guests >> on my s390 system. Both guests are connected via two macvtaps on the same OSA >> network card. >> Before that patch I get ~20 Gbit/sec between two guests, afterwards I get >> ~4Gbit/sec >> >> Latency seems to be unchanges (uperf 1byte ping pong). >> >> According to ifconfig in the guest, I have ~ 1500 bytes per packet with this >> patch and ~ 4 bytes without. So for some reason this patch causes the >> network stack to do segmentation. (the guest kernel stays the same, only >> host >> kernel is changed). >> >> Any ideas? > > I am looking. It shouldn't cause addition segmentations and when I ran > netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8 (macvtap: Perform GSO on forwarding path), the setup is basically the same. Christian -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 28/02/14 23:14, Vlad Yasevich wrote: On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8 (macvtap: Perform GSO on forwarding path), the setup is basically the same. Christian -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 03/01/2014 06:15 AM, Christian Borntraeger wrote: On 28/02/14 23:14, Vlad Yasevich wrote: On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. Could you post 'ethtool -k' output for both lower-level device and the macvtap device? Thanks -vlad Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8 (macvtap: Perform GSO on forwarding path), the setup is basically the same. Christian -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 03/01/2014 02:27 PM, Vlad Yasevich wrote: On 03/01/2014 06:15 AM, Christian Borntraeger wrote: On 28/02/14 23:14, Vlad Yasevich wrote: On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests. Could you post 'ethtool -k' output for both lower-level device and the macvtap device? Thanks -vlad Ok. I think I see what's happening. Since you turn off offloads on lower device, that's propagated to macvlan device. As a result, when when we call dev_queue_xmit on the vlan-dev, we end up segmenting since lower level says it does support segmentation. One way to fix this is to never disable offloads on macvlan. macvlan will always try to use __dev_queue_xmit() with it's lower device, so any segmentation can happen there. -vlad Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8 (macvtap: Perform GSO on forwarding path), the setup is basically the same. Christian -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 02/27/2014 03:52 PM, Christian Borntraeger wrote: > Vlad, > > commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 > macvtap: Add support of packet capture on macvtap device. > > causes a performance regression for iperf traffic between two KVM guests > on my s390 system. Both guests are connected via two macvtaps on the same OSA > network card. > Before that patch I get ~20 Gbit/sec between two guests, afterwards I get > ~4Gbit/sec > > Latency seems to be unchanges (uperf 1byte ping pong). > > According to ifconfig in the guest, I have ~ 1500 bytes per packet with this > patch and ~ 4 bytes without. So for some reason this patch causes the > network stack to do segmentation. (the guest kernel stays the same, only host > kernel is changed). > > Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. -vlad > > > Christian > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: macvtap performance regression (bisected) between 3.13 and 3.14-rc1
On 02/27/2014 03:52 PM, Christian Borntraeger wrote: Vlad, commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523 macvtap: Add support of packet capture on macvtap device. causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card. Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec Latency seems to be unchanges (uperf 1byte ping pong). According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 4 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed). Any ideas? I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput. -vlad Christian -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
