Ok, This is it:
(neutron-openvswitch-agent)[root@hercules-23 /]# grep -i mtu /etc/neutron/plugins/ml2/openvswitch_agent.ini # MTU size of veth interfaces (integer value) #veth_mtu = 9000 Thank you Manuel From: Remo Mattei [mailto:[email protected]] Sent: Tuesday, December 19, 2017 3:09 PM To: Manuel Sopena Ballesteros Cc: [email protected] Subject: Re: [Openstack] need help understanding neutron/OVS I think you are looking at the wrong file.. if you are using ovs you need to look at that… example: [root@os-poc-controller-0 ml2]# grep -i mtu openvswitch_agent.ini # MTU size of veth interfaces (integer value) #veth_mtu = 9000 On Dec 18, 2017, at 20:06, Manuel Sopena Ballesteros <[email protected]<mailto:[email protected]>> wrote: Yes, This is my neutron ml2 config on the compute node [ml2] type_drivers = flat,vlan,vxlan tenant_network_types = vxlan path_mtu = 9000 mechanism_drivers = openvswitch,l2population extension_drivers = port_security [ml2_type_vlan] network_vlan_ranges = [ml2_type_flat] flat_networks = physnet1 [ml2_type_vxlan] vni_ranges = 1:1000 vxlan_group = 239.1.1.1 [securitygroup] firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver [agent] tunnel_types = vxlan l2_population = true arp_responder = true [ovs] ovsdb_connection = tcp:127.0.0.1:6640 local_ip = 10.0.32.23 thank you Manuel From: Remo Mattei [mailto:[email protected]] Sent: Tuesday, December 19, 2017 12:56 PM To: Manuel Sopena Ballesteros Cc: [email protected]<mailto:[email protected]> Subject: Re: [Openstack] need help understanding neutron/OVS Did you change the mtu on neutron config? Sent from my iPad On Dec 18, 2017, at 5:38 PM, Manuel Sopena Ballesteros <[email protected]<mailto:[email protected]>> wrote: Dear Openstack community, Openstack environment: 2 compute nodes each has: • 1x AMD Opteron(tm) Processor 6282 SE (56 cpus) • 512GB RAM • 1x dual port 25Gbps Mellanox connect-4 XOR bond Physical network: • Compute nodes connected through a mellanox 2700 switch (100Gbps) • MTU 9000 Openstack network: • OVS + VxLan • Vxlan offloading setup on physical nics • MTU on vms 8950 Compute nodes nics: [root@hercules-23 ~]# ip a … 6: p2p1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff 7: p2p2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP qlen 1000 link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff 8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP qlen 1000 link/ether 7c:fe:90:12:23:c4 brd ff:ff:ff:ff:ff:ff inet 10.0.32.23/16 brd 10.0.255.255 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::3826:9daa:ef1b:43b0/64 scope link valid_lft forever preferred_lft forever … VM nics: [centos@centos2 ~]$ ip a … 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc pfifo_fast state UP qlen 1000 link/ether fa:16:3e:a3:d4:85 brd ff:ff:ff:ff:ff:ff inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic eth0 valid_lft 57885sec preferred_lft 57885sec inet6 fe80::f816:3eff:fea3:d485/64 scope link valid_lft forever preferred_lft forever host to host bandwidth using muttcp (1 thread): [root@hercules-22 ~]# nuttcp -l65536 -fparse 10.0.32.23 megabytes=19844.1759 real_seconds=10.00 rate_Mbps=16643.4189 tx_cpu=61 rx_cpu=99 retrans=0 rtt_ms=0.28 VMTP Output: ================== Total Scenarios: 29 Passed Scenarios: 17 [100.00%] Failed Scenarios: 0 [0.00%] Skipped Scenarios: 12 +----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+ | Scenario | Scenario Name | Functional Status | Data | +----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+ | 1.1 | Same Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '7613130', 'rtt_ms': '0.463333'} | | 1.2 | Same Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 59954, 'loss_rate': 0.02}, 1024: {'tp_kbps': 444264, | | | | | 'loss_rate': 0.05}, 8192: {'tp_kbps': 2877674, 'loss_rate': 0.0}} | | 1.3 | Same Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.376/0.285/0.439/0.047', | | | | | '64-byte': '0.617/0.361/1.513/0.388', '1500-byte': '0.369/0.322/0.452/0.048'}} | | 1.4 | Same Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} | | 2.1 | Same Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6864266', 'rtt_ms': '0.653333'} | | 2.2 | Same Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 80955, 'loss_rate': 0.46}, 1024: {'tp_kbps': 166186, | | | | | 'loss_rate': 36.2}, 8192: {'tp_kbps': 2189389, 'loss_rate': 0.05}} | | 2.3 | Same Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.484/0.315/0.561/0.068', | | | | | '64-byte': '0.719/0.421/2.714/0.666', '1500-byte': '0.465/0.358/0.553/0.055'}} | | 2.4 | Same Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} | | 3.1 | Different Network, Fixed IP, Intra-node, TCP | PASSED | {'tp_kbps': '5207719', 'rtt_ms': '0.896667'} | | 3.2 | Different Network, Fixed IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 76569, 'loss_rate': 0.9}, 1024: {'tp_kbps': 571847, | | | | | 'loss_rate': 2.75}, 8192: {'tp_kbps': 1855166, 'loss_rate': 0.0}} | | 3.3 | Different Network, Fixed IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.859/0.658/1.103/0.133', | | | | | '64-byte': '0.976/0.576/1.946/0.361', '1500-byte': '0.824/0.644/1.082/0.136'}} | | 3.4 | Different Network, Fixed IP, Intra-node, Multicast | SKIPPED | {} | | 4.1 | Different Network, Fixed IP, Inter-node, TCP | PASSED | {'tp_kbps': '6730928', 'rtt_ms': '1.37667'} | | 4.2 | Different Network, Fixed IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 31131, 'loss_rate': 4.84}, 1024: {'tp_kbps': 579859, | | | | | 'loss_rate': 4.35}, 8192: {'tp_kbps': 1601092, 'loss_rate': 0.18}} | | 4.3 | Different Network, Fixed IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.915/0.740/1.138/0.144', | | | | | '64-byte': '1.134/0.713/2.210/0.539', '1500-byte': '0.980/0.719/1.188/0.131'}} | | 4.4 | Different Network, Fixed IP, Inter-node, Multicast | SKIPPED | {} | | 5.1 | Different Network, Floating IP, Intra-node, TCP | PASSED | {'tp_kbps': '5406554', 'rtt_ms': '0.836667'} | | 5.2 | Different Network, Floating IP, Intra-node, UDP | PASSED | {128: {'tp_kbps': 71084, 'loss_rate': 4.26}, 1024: {'tp_kbps': 507900, | | | | | 'loss_rate': 15.96}, 8192: {'tp_kbps': 2011414, 'loss_rate': 3.72}} | | 5.3 | Different Network, Floating IP, Intra-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '0.902/0.760/1.007/0.073', | | | | | '64-byte': '0.875/0.661/0.967/0.104', '1500-byte': '0.889/0.754/0.988/0.082'}} | | 5.4 | Different Network, Floating IP, Intra-node, Multicast | SKIPPED | {} | | 6.1 | Different Network, Floating IP, Inter-node, TCP | PASSED | {'tp_kbps': '6770995', 'rtt_ms': '0.973333'} | | 6.2 | Different Network, Floating IP, Inter-node, UDP | PASSED | {128: {'tp_kbps': 81812, 'loss_rate': 2.25}, 1024: {'tp_kbps': 617485, | | | | | 'loss_rate': 3.27}, 8192: {'tp_kbps': 1295351, 'loss_rate': 0.02}} | | 6.3 | Different Network, Floating IP, Inter-node, ICMP | PASSED | {'rtt avg/min/max/stddev msec': {'391-byte': '1.028/0.920/1.254/0.105', | | | | | '64-byte': '1.043/0.807/1.227/0.137', '1500-byte': '0.988/0.759/1.194/0.146'}} | | 6.4 | Different Network, Floating IP, Inter-node, Multicast | SKIPPED | {} | | 7.1 | Native Throughput, TCP | SKIPPED | {} | | 7.2 | Native Throughput, UDP | SKIPPED | {} | | 7.3 | Native Throughput, ICMP | SKIPPED | {} | | 7.4 | Native Throughput, Multicast | SKIPPED | {} | | 8.1 | VM to Host Uploading | SKIPPED | {} | | 8.2 | VM to Host Downloading | SKIPPED | {} | +----------+-------------------------------------------------------+-------------------+----------------------------------------------------------------------------------+ My concern is about east-west bandwidth: Inter-host and intra-subnet (vmtp Scenario 2.1): I am getting ~6.8Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts intra-host and intra-subnet (vmtp Scenario 1.1): In this example traffic remains inside the compute node and no tunneling is required. I am getting ~7.6Gbps which I found quite poor compared to 16.0 Gbps I am getting from the hosts NOTE: I am not running anything except these tests on the machines so cpu usage is quite low. NOTE2: I know vmtp uses I would like to understand what is stopping my vms to get full network bandwidth. Is this related with an OVS configuration? ML2? Other part of neutron? Something else? Thank you very much Manuel Sopena Ballesteros | Big data Engineer Garvan Institute of Medical Research The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: [email protected]<mailto:[email protected]> NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected]<mailto:[email protected]> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. [https://master.mailbutler.io/tracking/70F490E0-E398-4120-A92A-6CF45ECC075B] NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
