Reviewed: https://review.opendev.org/711048 Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=41b8019c7b4b3921a077a032463b9d8c74957b4b Submitter: Zuul Branch: master
commit 41b8019c7b4b3921a077a032463b9d8c74957b4b Author: Maciej Józefczyk <[email protected]> Date: Tue Mar 3 17:10:57 2020 +0100 QoS - Change the way we measure bw limits This patch introduces new way of fetching the data. Instead creating the file, it reads /dev/zero. /dev/zero is always very fast, so we also break the previous hard disk limitations. The test time is limited to 5 seconds. After that we calculate avg bytes per second value and compare it to expected one. Sometimes it is visible that first kilobytes of the test file are downloaded a little bit faster than the actual bw limit claims, especially while testing OVN as a backend. When it happens the avg bytes per second value that is measured in the test could be higher than required limit. It is pretty easy to show the case while testing QoS with iperf3: Accepted connection from 172.24.5.1, port 59690 [ 5] local 10.1.0.35 port 5201 connected to 172.24.5.1 port 59692 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 5] 0.00-1.00 sec 1.32 MBytes 11.0 Mbits/sec 139 2.62 KBytes [ 5] 1.00-2.00 sec 628 KBytes 5.15 Mbits/sec 96 10.5 KBytes [ 5] 2.00-3.00 sec 502 KBytes 4.12 Mbits/sec 84 7.85 KBytes [ 5] 3.00-4.00 sec 649 KBytes 5.32 Mbits/sec 83 10.5 KBytes [ 5] 4.00-5.00 sec 643 KBytes 5.26 Mbits/sec 84 3.93 KBytes [ 5] 5.00-6.00 sec 529 KBytes 4.33 Mbits/sec 73 5.23 KBytes [ 5] 6.00-7.00 sec 628 KBytes 5.15 Mbits/sec 92 20.9 KBytes [ 5] 7.00-8.00 sec 534 KBytes 4.37 Mbits/sec 82 18.3 KBytes [ 5] 8.00-9.00 sec 667 KBytes 5.47 Mbits/sec 110 7.85 KBytes [ 5] 9.00-10.00 sec 635 KBytes 5.20 Mbits/sec 90 11.8 KBytes [ 5] 10.00-10.02 sec 0.00 Bytes 0.00 bits/sec 0 11.8 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-10.02 sec 6.61 MBytes 5.53 Mbits/sec 933 sender [ 5] 0.00-10.02 sec 6.41 MBytes 5.36 Mbits/sec receiver ----------------------------------------------------------- We can find out that during first second of the test the bw limit is exceeded, but after that the traffic is shaped. In our case when we run the tempest QoS test the avg bytes per second measured value that we compare with bw limit is impacted. Closes-Bug: 1866039 Change-Id: I0964464e709baf9958548384933bd000fdee979b ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1866039 Title: [OVN] QoS gives different bandwidth limit measures than ml2/ovs Status in neutron: Fix Released Bug description: There is a difference in QoS tempest tests results between ml2/ovs and ml2/ovn. In the change [1] that enables QoS tempest tests for OVN the test neutron_tempest_plugin.scenario.test_qos.QoSTest.test_qos_basic_and_update fails on the last check [2], after the policy is updated to be configured with values: max_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3 max_burst_kbps=constants.LIMIT_KILO_BITS_PER_SECOND * 3, Which means: max_kbps = 3000 max_burst_kbps = 3000 Previous QoS validations in this test passes with values (max_kbps, max_burst_kbps): (1000, 1000) and (2000, 2000). I added some more debug log to the tempest test here [3], so that we can compare test expected and measured values. Those are taken from test runs from gates. ----------------------------------------------------------------------- Expected is calculated as: TOLERANCE_FACTOR = 1.5 constants.LIMIT_KILO_BITS_PER_SECOND = 1000 MULTIPLEXING_FACTOR = 1 or 2 or 3 depends on stage of the test LIMIT_BYTES_SEC = (constants.LIMIT_KILO_BITS_PER_SECOND * 1024 * TOLERANCE_FACTOR / 8.0) * MULTIPLEXING_FACTOR ----------------------------------------------------------------------- Results: If expected <= measured, the test passes. |max_kbps/max_burst_kbps|expected(bps)|ovs(bps)|ovn(bps)|linux_bridge(bps)| |(1000, 1000)|192000|112613|141250|129124| |(2000, 2000)|384000|311978|408886, 411005, 385152, 422114, 352903|300163| |(3000, 3000)|576000|523677|820522,..... failed|459569| As we see only for (3000, 3000) OVN test failed. For (2000, 2000) it passed after 5 retries. ----------------------------------------------------------------------- So lets see how the QoS is configured on OVN nowadays: stack@mjozefcz-devstack-qos-2:~/logs$ neutron qos-bandwidth-limit-rule-list 047f7a8c-e143-471f-979c-4a4d95cefa5e neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +-----------+--------------------------------------+----------------+----------+ | direction | id | max_burst_kbps | max_kbps | +-----------+--------------------------------------+----------------+----------+ | egress | 9dd84dc7-f216-432f-b1aa-ec17eb488720 | 3000 | 3000 | +-----------+--------------------------------------+----------------+----------+ Configured OVN NBDB: stack@mjozefcz-devstack-qos-2:~/logs$ ovn-nbctl list qos _uuid : 1176fe8f-695d-4f79-a99f-f0df8a7b8652 action : {} bandwidth : {burst=3000, rate=3000} direction : from-lport external_ids : {} match : "inport == \"4521ef05-d139-4d84-a100-efb83fde2b47\"" priority : 2002 Configured meter on bridge: stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13 dump-meters br-int OFPST_METER_CONFIG reply (OF1.3) (xid=0x2): meter=1 kbps burst stats bands= type=drop rate=3000 burst_size=3000 Flow in bridge: stack@mjozefcz-devstack-qos-2:~/logs$ sudo ovs-ofctl -O OpenFlow13 dump-flows br-int | grep meter cookie=0x398f0e17, duration=71156.273s, table=16, n_packets=136127, n_bytes=41572857, priority=2002,reg14=0x4,metadata=0x1 actions=meter:1,resubmit(,17) -------------------------------------------------------------------------- Questions: * Why the test results are different compared to ml2/OVS? * Maybe burst values should be configured differently? [1] https://review.opendev.org/#/c/704833/ [2] https://github.com/openstack/neutron-tempest-plugin/blob/328edc882a3debf4f1b39687dfb559d7c5c385f3/neutron_tempest_plugin/scenario/test_qos.py#L271 [3] https://review.opendev.org/#/c/711048/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1866039/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
