On 6/3/22 16:47, Jon Kohler wrote:
>
>
>> On Jun 2, 2022, at 5:50 PM, Ilya Maximets <i.maxim...@ovn.org> wrote:
>>
>> On 6/2/22 23:13, Jon Kohler wrote:
>>> Any takers? I’m hoping I’ve got the right mailing list, as I did see
>>> the thread get generated on the mailing list website?
>>
>> Hey, Jon.
>>
>> Yes, it's the right mailing list, though ovs-dev and dev are
>> the same thing, so you don't need to send to both.
>>
>> For the subject, you sent a patch just about a week ago.
>> Unfortunately, patches are not moving that fast. People
>> are busy with working on issues and doing reviews of other
>> patches, so please, be patient.
>>
>> Just from the patches received last month we still have
>> about 60 under review right now. You can see them in the
>> patchwork here:
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_project_openvswitch_list_&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=NGPRGGo37mQiSXgHKm5rCQ&m=8oqFH3XK_nAzOxdHmUuHbWgSEPemwCFQKUdGExfEPEx3NfR3ioDpK5CZHwMdEyf_&s=Osr7hGzYKATNpHa9IvtWygokIFsGsmT4GrnDuUQ3FzY&e=
>>
>>
>> And as you can see we have much older patches too. So, it
>> will take some time until someone will get to yours.
>>
>> Best regards, Ilya Maximets.
>
> Thanks for the feedback and pointers. My apologies, I wasn’t trying to be
> ansty :) this is my first patch to this mailing list and wanted to make sure
> it
> was going to the right place.
No problem. :)
>
> Ok no problems, I can fully empathize with patch review backlogs :) No
> specific rush on this one. Nice CPU win whenever someone can get
> to it, but would definitely rather have quality over speed to make sure
> we haven’t missed anything! :)
Thanks for working on improvements! I hope, someone will get
to this patch in the near future as it looks interesting indeed.
>
> Cheers,
> Jon
>
>>
>>>
>>>> On May 31, 2022, at 10:03 AM, Jon Kohler <j...@nutanix.com> wrote:
>>>>
>>>>
>>>>
>>>>> On May 26, 2022, at 9:11 PM, Jon Kohler <j...@nutanix.com> wrote:
>>>>>
>>>>> For netdev_linux_update_via_netlink(), hint to the kernel that
>>>>> we do not need it to gather netlink internal stats when we want
>>>>> to update the netlink flags, as those stats are not rendered
>>>>> within OVS.
>>>>>
>>>>> Background:
>>>>> ovs-vswitchd can spend quite a bit of time blocked by the kernel
>>>>> during netlink calls, especially systems with many cores. This
>>>>> time is dominated by the kernel-side internal stats gathering
>>>>> mechanism in netlink, specifically:
>>>>> inet6_fill_link_af
>>>>> inet6_fill_ifla6_attrs
>>>>> __snmp6_fill_stats64
>>>>>
>>>>> In Linux 4.4+, there exists a hint for netlink requests to not
>>>>> trigger the ipv6 stats gathering mechanism, which greatly reduces
>>>>> the amount of time that ovs-vswitchd is on CPU.
>>>>>
>>>>> Testing and Results:
>>>>> Tested booting 320 VM's and measuring OVS utilization with perf
>>>>> record, then visualized into a flamegraph using a patched version
>>>>> of ovs 2.14.2. Calls under bridge_run() seem to get hit the worst
>>>>> by this issue.
>>>>>
>>>>> Before bridge_run() == 11.3% of samples
>>>>> After bridge_run() == 3.4% of samples
>>>>>
>>>>> Note that there are at least two observed netlink calls under
>>>>> bridge_run that are still kernel stats heavy after this patch:
>>>>>
>>>>> Call 1:
>>>>> bridge_run -> netdev_run -> route_table_run -> route_table_reset ->
>>>>> ovs_router_insert -> ovs_router_insert__ -> get_src_addr ->
>>>>> netdev_ger_addr_list -> netdev_linux_get_addr_list -> getifaddrs
>>>>>
>>>>> Since the actual netlink call is coming from getifaddrs() in glibc,
>>>>> fixing would likely involve either duplicating glibc code in ovs
>>>>> source or patch glibc.
>>>>>
>>>>> Call 2:
>>>>> bridge_run -> iface_refresh_stats -> netdev_get_stats ->
>>>>> netdev_linux_get_stats -> get_stats_via_netlink
>>>>>
>>>>> This does use netlink based stats; however, it isn't immediately
>>>>> clear if just dropping the stats from inet6_fill_link_af would
>>>>> impact anything or not. Given this call is more intermittent, its
>>>>> of lesser concern.
>>>>>
>>>>> Signed-off-by: Jon Kohler <j...@nutanix.com>
>>>>> Acked-by: Greg Smith <gasm...@nutanix.com>
>>>>
>>>> Gentle bump
>>>
>>>
>>>
>>>>
>>>>> ---
>>>>> lib/netdev-linux.c | 9 +++++++++
>>>>> 1 file changed, 9 insertions(+)
>>>>>
>>>>> diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c
>>>>> index 2766b3f2bf..f0246d3b2b 100644
>>>>> --- a/lib/netdev-linux.c
>>>>> +++ b/lib/netdev-linux.c
>>>>> @@ -247,6 +247,12 @@ enum {
>>>>> VALID_NUMA_ID = 1 << 8,
>>>>> };
>>>>>
>>>>> +/* Linux 4.4 introduced the ability to skip the internal stats gathering
>>>>> + * that netlink does via an external filter mask that can be passed into
>>>>> + * a netlink request.
>>>>> + */
>>>>> +#define RTEXT_FILTER_SKIP_STATS (1 << 3)
>>>>> +
>>>>> /* Use one for the packet buffer and another for the aux buffer to receive
>>>>> * TSO packets. */
>>>>> #define IOV_STD_SIZE 1
>>>>> @@ -6418,6 +6424,9 @@ netdev_linux_update_via_netlink(struct netdev_linux
>>>>> *netdev)
>>>>> if (netdev_linux_netnsid_is_remote(netdev)) {
>>>>> nl_msg_put_u32(&request, IFLA_IF_NETNSID, netdev->netnsid);
>>>>> }
>>>>> +
>>>>> + nl_msg_put_u32(&request, IFLA_EXT_MASK, RTEXT_FILTER_SKIP_STATS);
>>>>> +
>>>>> error = nl_transact(NETLINK_ROUTE, &request, &reply);
>>>>> ofpbuf_uninit(&request);
>>>>> if (error) {
>>>>> --
>>>>> 2.30.1 (Apple Git-130)
>>>
>>> _______________________________________________
>>> dev mailing list
>>> d...@openvswitch.org
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-2Ddev&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=NGPRGGo37mQiSXgHKm5rCQ&m=8oqFH3XK_nAzOxdHmUuHbWgSEPemwCFQKUdGExfEPEx3NfR3ioDpK5CZHwMdEyf_&s=h-hQ3onb9ZZoJwMrpOdPVN0uZVoTPJtZ8ZHFKTCknZ0&e=
>
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
