On 7/10/23 16:54, Kevin Traynor wrote:
> Extend 'pmd-sleep-max' so that individual PMD thread cores
> may have a specified max sleep request value.
>
> Any PMD thread core without a value will use the datapath default
> (no sleep request) or datapath global value set by the user.
>
> To set PMD thread cores 8 and 9 to never request a load based sleep
> and all other PMD thread cores to be able to request a max sleep of
> 50 usecs:
>
> $ ovs-vsctl set open_vswitch . other_config:pmd-sleep-max=50,8:0,9:0
>
> To set PMD thread cores 10 and 11 to request a max sleep of 100 usecs
> and all other PMD thread cores to never request a sleep:
>
> $ ovs-vsctl set open_vswitch . other_config:pmd-sleep-max=10:100,11:100
>
> 'pmd-sleep-show' can be used to dump the global and individual PMD thread
> core max sleep request values.
>
> Signed-off-by: Kevin Traynor <[email protected]>
> Reviewed-by: David Marchand <[email protected]>
> ---
> Documentation/topics/dpdk/pmd.rst | 23 +++
> lib/dpif-netdev-private-thread.h | 3 +
> lib/dpif-netdev.c | 226 +++++++++++++++++++++++++++---
> tests/pmd.at | 32 ++---
> 4 files changed, 252 insertions(+), 32 deletions(-)
Notes on this patch are a bit chaotic as they are mostly about the looks
and perception. Hope they make some sense. :)
>
> diff --git a/Documentation/topics/dpdk/pmd.rst
> b/Documentation/topics/dpdk/pmd.rst
> index 40e6b7843..7eb4eb7b3 100644
> --- a/Documentation/topics/dpdk/pmd.rst
> +++ b/Documentation/topics/dpdk/pmd.rst
> @@ -375,4 +375,27 @@ system configuration (e.g. enabling processor C-states)
> and workloads.
> rate.
>
> +Max sleep request values can be set for individual PMDs using key:value
> pairs.
'max sleep request' doesn't sound like a human term.
Maybe 'Maximum sleep values' or 'maximum sleep intervals' ?
> +Any PMD that has been assigned a specified value will use that. Any PMD that
> +does not have a specified value will use the current global default.
> +
> +Specified values for individual PMDs can be added or removed at any time.
What do you think about:
s/PMD/PMD thread/
s/PMDs/PMD threads/
averywhere in the text above?
> +
> +For example, to set PMD threads on cores 8 and 9 to never request a load
> based
> +sleep and all others PMD threads to be able to request a max sleep of 50
> usecs::
s/sleep of 50 usecs/maximum sleep of 50 microseconds (us)/
> +
> + $ ovs-vsctl set open_vswitch . other_config:pmd-sleep-max=50,8:0,9:0
> +
> +The max sleep request for each PMD can be checked in the logs or with::
Should we remove the previous mentioning of that command?
> +
> + $ ovs-appctl dpif-netdev/pmd-sleep-show
> + PMD max sleep request is 50 usecs by default.
> + PMD load based sleeps are enabled by default.
> + PMD thread core 8 NUMA 0: Max sleep request set to 0 usecs.
> + PMD thread core 9 NUMA 1: Max sleep request set to 0 usecs.
> + PMD thread core 10 NUMA 0: Max sleep request set to 50 usecs.
> + PMD thread core 11 NUMA 1: Max sleep request set to 50 usecs.
> + PMD thread core 12 NUMA 0: Max sleep request set to 50 usecs.
> + PMD thread core 13 NUMA 1: Max sleep request set to 50 usecs.
A few note about the output:
- Too much 'PMD', IMO. :)
- The word 'request' has an unclear meaning and seems redundant.
- 'by default' is a bit confusing.
- We do not use 'usecs' in other places. We use 'us' in pmd stats.
What do you think about something like this:
$ ovs-appctl dpif-netdev/pmd-sleep-show
Default max sleep: 50 us
pmd thread numa_id 0 core_id 8:
max sleep: 0 us
pmd thread numa_id 1 core_id 9:
max sleep: 0 us
pmd thread numa_id 0 core_id 10:
max sleep: 50 us
pmd thread numa_id 1 core_id 11:
max sleep: 50 us
pmd thread numa_id 0 core_id 12:
max sleep: 50 us
pmd thread numa_id 1 core_id 13:
max sleep: 50 us
---
?
Less words, the same header for numa/core as in other pmd-*-show
commands. This format also allows extension in the future, if
needed.
Alternatively, we may degrade the output to a table:
$ ovs-appctl dpif-netdev/pmd-sleep-show
Default max sleep: 50 us
core_id numa_id max sleep
------- ------- ---------
8 0 0 us
9 1 0 us
10 0 50 us
11 1 50 us
12 0 50 us
13 1 50 us
---
It's the most readable option, but least extensible.
> +
> .. _ovs-vswitchd(8):
> http://openvswitch.org/support/dist-docs/ovs-vswitchd.8.html
> diff --git a/lib/dpif-netdev-private-thread.h
> b/lib/dpif-netdev-private-thread.h
> index 1ec3cd794..5c72ce5d9 100644
> --- a/lib/dpif-netdev-private-thread.h
> +++ b/lib/dpif-netdev-private-thread.h
> @@ -181,4 +181,7 @@ struct dp_netdev_pmd_thread {
> bool isolated;
>
> + /* Max sleep request. UINT64_MAX indicates dp default should be used.*/
Can we just store a value here directly after parsing without
fallback logic?
> + atomic_uint64_t max_sleep;
> +
> /* Queue id used by this pmd thread to send packets on all netdevs if
> * XPS disabled for this netdev. All static_tx_qid's are unique and less
> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> index 522253c82..0e79705d8 100644
> --- a/lib/dpif-netdev.c
> +++ b/lib/dpif-netdev.c
> @@ -180,4 +180,9 @@ static struct odp_support dp_netdev_support = {
> #define PMD_SLEEP_INC_US 1
>
> +struct pmd_sleep {
> + unsigned core_id;
> + uint64_t max_sleep;
> +};
> +
> struct dpcls {
> struct cmap_node node; /* Within dp_netdev_pmd_thread.classifiers */
> @@ -290,4 +295,6 @@ struct dp_netdev {
> /* Max load based sleep request. */
> atomic_uint64_t pmd_max_sleep;
> + /* Max load based sleep request user string. */
> + char *max_sleep_list;
Can we move it somewhere else in the structure?
This part is on a hot path in the datapath. Might be better for
performance to keep important variables together.
> /* Enable the SMC cache from ovsdb config */
> atomic_bool smc_enable_db;
> @@ -1013,9 +1020,15 @@ pmd_max_sleep_show(struct ds *reply, struct
> dp_netdev_pmd_thread *pmd,
> {
> if (pmd->core_id != NON_PMD_CORE_ID) {
> + uint64_t pmd_max_sleep;
> +
> + atomic_read_relaxed(&pmd->max_sleep, &pmd_max_sleep);
> ds_put_format(reply,
> "PMD thread core %3u NUMA %2d: "
> "Max sleep request set to",
> pmd->core_id, pmd->numa_id);
> - ds_put_format(reply, " %4"PRIu64" usecs.", default_max_sleep);
> + ds_put_format(reply, " %4"PRIu64" usecs.",
> + pmd_max_sleep == UINT64_MAX
> + ? default_max_sleep
> + : pmd_max_sleep);
Not sure why we need 2 different calls to ds_put_format().
And the ? can be moved on a previous line with : re-indented accordingly.
> ds_put_cstr(reply, "\n");
> }
> @@ -1528,7 +1541,8 @@ dpif_netdev_pmd_info(struct unixctl_conn *conn, int
> argc, const char *argv[],
> atomic_read_relaxed(&dp->pmd_max_sleep, &default_max_sleep);
> ds_put_format(&reply, "PMD max sleep request is %"PRIu64" "
> - "usecs.", default_max_sleep);
> + "usecs by default.", default_max_sleep);
> ds_put_cstr(&reply, "\n");
> - ds_put_format(&reply, "PMD load based sleeps are %s.",
> + ds_put_format(&reply, "PMD load based sleeps are %s "
> + "by default.",
> default_max_sleep ? "enabled" : "disabled");
> ds_put_cstr(&reply, "\n");
> @@ -1924,4 +1938,6 @@ create_dp_netdev(const char *name, const struct
> dpif_class *class,
> }
>
> + dp->max_sleep_list = NULL;
> +
> dp->last_tnl_conf_seq = seq_read(tnl_conf_seq);
> *dpp = dp;
> @@ -2033,4 +2049,5 @@ dp_netdev_free(struct dp_netdev *dp)
> dp_netdev_meter_destroy(dp);
>
> + free(dp->max_sleep_list);
> free(dp->pmd_cmask);
> free(CONST_CAST(char *, dp->name));
> @@ -4847,4 +4864,8 @@ set_pmd_auto_lb(struct dp_netdev *dp, bool state, bool
> always_log)
> }
>
> +static void
> +set_all_pmd_max_sleeps(struct dp_netdev *dp, const struct smap *config,
> + bool always_log);
Name of a prototype should be on a same line with a return type.
> +
> /* Applies datapath configuration from the database. Some of the changes are
> * actually applied in dpif_netdev_run(). */
> @@ -4864,5 +4885,4 @@ dpif_netdev_set_config(struct dpif *dpif, const struct
> smap *other_config)
> uint8_t cur_rebalance_load;
> uint32_t rebalance_load, rebalance_improve;
> - uint64_t pmd_max_sleep, cur_pmd_max_sleep;
> bool log_autolb = false;
> enum sched_assignment_type pmd_rxq_assign_type;
> @@ -5019,16 +5039,7 @@ dpif_netdev_set_config(struct dpif *dpif, const struct
> smap *other_config)
> "Please use pmd-sleep-max instead.");
> }
> + set_all_pmd_max_sleeps(dp, other_config, first_set_config);
>
> - pmd_max_sleep = smap_get_ullong(other_config, "pmd-sleep-max", 0);
> - pmd_max_sleep = MIN(PMD_RCU_QUIESCE_INTERVAL, pmd_max_sleep);
> - atomic_read_relaxed(&dp->pmd_max_sleep, &cur_pmd_max_sleep);
> - if (first_set_config || pmd_max_sleep != cur_pmd_max_sleep) {
> - atomic_store_relaxed(&dp->pmd_max_sleep, pmd_max_sleep);
> - VLOG_INFO("PMD max sleep request is %"PRIu64" usecs.",
> pmd_max_sleep);
> - VLOG_INFO("PMD load based sleeps are %s.",
> - pmd_max_sleep ? "enabled" : "disabled" );
> - }
> -
> - first_set_config = false;
> + first_set_config = false;
> return 0;
> }
> @@ -5070,4 +5081,182 @@ parse_affinity_list(const char *affinity_list,
> unsigned *core_ids, int n_rxq)
> }
>
> +static int
> +parse_pmd_sleep_list(const char *max_sleep_list,
> + struct pmd_sleep **pmd_sleeps)
> +{
> + char *list, *copy, *key, *value;
> + int num_vals = 0;
> +
> + if (!max_sleep_list) {
> + return num_vals;
> + }
> +
> + list = copy = xstrdup(max_sleep_list);
> +
> + while (ofputil_parse_key_value(&list, &key, &value)) {
> + char *error = NULL;
> + unsigned core;
> + uint64_t temp, pmd_max_sleep;
> + int i;
> +
> + error = str_to_u64(key, &temp);
> + if (error) {
> + free(error);
> + continue;
> + }
> +
> + error = str_to_u64(value, &pmd_max_sleep);
> + if (error) {
> + /* No value specified. key is dp default. */
> + core = UINT_MAX;
> + pmd_max_sleep = temp;
> + free(error);
> + } else {
> + /* Value specified. key is pmd core id.*/
> + if (temp >= UINT_MAX) {
> + continue;
> + }
> + core = (unsigned) temp;
> + }
> +
> + /* Detect duplicate max sleep values for default or a specific core.
> */
> + for (i = 0; i < num_vals; i++) {
> + if ((*pmd_sleeps)[i].core_id == core) {
> + break;
> + }
> + }
> + if (i == num_vals) {
> + /* Not duplicate, add a new entry. */
> + *pmd_sleeps = xrealloc(*pmd_sleeps,
> + (num_vals + 1) * sizeof **pmd_sleeps);
> + num_vals++;
> + }
> +
> + pmd_max_sleep = MIN(PMD_RCU_QUIESCE_INTERVAL, pmd_max_sleep);
> +
> + (*pmd_sleeps)[i].core_id = core;
> + (*pmd_sleeps)[i].max_sleep = pmd_max_sleep;
Might be cleaner to maintain a locally allocated array and assign the
output parameter on exit.
> + }
> +
> + free(copy);
> + return num_vals;
> +}
> +
> +static void log_pmd_sleep(unsigned core_id, int numa_id,
> + uint64_t pmd_max_sleep, uint64_t default_max_sleep)
> +{
> + VLOG_INFO("PMD thread core %3u NUMA %2d: Max sleep request set to "
> + "%4"PRIu64" usecs.", core_id, numa_id,
s/usecs/microseconds/ or s/usecs/us/
And we usually print cores as signed integers to get -1 for non-pmd threads.
OTOH, we should not log for non-pmd at all here, because sleep makes no sense
for them. But I see this in the log:
dpif_netdev|INFO|PMD thread core 4294967295 NUMA 2147483647: Max sleep request
set to 0 usecs.
Also, would be nice to get more unified logging with the general thread
creation:
dpif_netdev|INFO|PMD thread core 4 NUMA 0: Max sleep request set to 0
usecs.
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 4 created.
dpif_netdev|INFO|PMD thread core 2 NUMA 0: Max sleep request set to 0
usecs.
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 2 created.
Maybe something like:
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 4 max sleep changed to 0
us.
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 4 created.
And we probably shouldn't log per-pmd if it didn't actually change.
It makes sense to log the first time the default.
> + pmd_max_sleep == UINT64_MAX
> + ? default_max_sleep
> + : pmd_max_sleep);
This should be:
pmd_max_sleep == UINT64_MAX ? default_max_sleep
: pmd_max_sleep);
> +}
> +
> +static void
> +set_pmd_max_sleep(struct dp_netdev *dp, struct dp_netdev_pmd_thread *pmd)
This function should be called something like pmd_init_max_sleep.
It is using the atomic_init call, so it should only be used for init.
> +{
> + struct pmd_sleep *pmd_sleeps = NULL;
> + uint64_t max_sleep = UINT64_MAX;
Can we use a defualt here?
> + int num_vals;
> +
> + num_vals = parse_pmd_sleep_list(dp->max_sleep_list, &pmd_sleeps);
> +
> + /* Check if the user has set a specific value for this pmd. */
> + for (int i = 0; i < num_vals; i++) {
> + if (pmd_sleeps[i].core_id == pmd->core_id) {
> + max_sleep = pmd_sleeps[i].max_sleep;
> + break;
> + }
> + }
> + atomic_init(&pmd->max_sleep, max_sleep);
> + log_pmd_sleep(pmd->core_id, pmd->numa_id, max_sleep, dp->pmd_max_sleep);
> + free(pmd_sleeps);
> +}
> +
> +static void
> +set_all_pmd_max_sleeps(struct dp_netdev *dp, const struct smap *config,
> + bool always_log)
> +{
> + const char *max_sleep_list = smap_get(config, "pmd-sleep-max");
> + struct pmd_sleep *pmd_sleeps = NULL;
> + struct dp_netdev_pmd_thread **pmd_list = NULL;
> + struct dp_netdev_pmd_thread *pmd;
> + int num_vals = 0;
> + uint64_t default_max_sleep = 0;
> + uint64_t cur_default_max_sleep;
> + size_t n;
Reverse x-mass tree.
> +
> + if (nullable_string_is_equal(max_sleep_list, dp->max_sleep_list)
> + && !always_log) {
> + return;
> + }
> +
> + /* Free existing string and copy new one. */
> + free(dp->max_sleep_list);
> + dp->max_sleep_list = nullable_xstrdup(max_sleep_list);
> +
> + num_vals = parse_pmd_sleep_list(max_sleep_list, &pmd_sleeps);
> +
> + /* Check if the user has set a dp default. */
> + for (int i = 0; i < num_vals; i++) {
> + if (pmd_sleeps[i].core_id == UINT_MAX) {
> + default_max_sleep = pmd_sleeps[i].max_sleep;
> + break;
> + }
> + }
> + atomic_read_relaxed(&dp->pmd_max_sleep, &cur_default_max_sleep);
> + if (default_max_sleep != cur_default_max_sleep) {
> + atomic_store_relaxed(&dp->pmd_max_sleep, default_max_sleep);
> + always_log = true;
> + }
> +
> + CMAP_FOR_EACH (pmd, node, &dp->poll_threads) {
> + uint64_t new_max_sleep, cur_pmd_max_sleep;
> +
> + if (pmd->core_id == NON_PMD_CORE_ID) {
> + continue;
> + }
> +
> + /* Default to dp value. */
> + new_max_sleep = UINT64_MAX;
Can we use a default here?
> +
> + /* Check for pmd specific value. */
> + for (int i = 0; i < num_vals; i++) {
> + if (pmd->core_id == pmd_sleeps[i].core_id) {
> + new_max_sleep = pmd_sleeps[i].max_sleep;
> + break;
> + }
> + }
> + atomic_read_relaxed(&pmd->max_sleep, &cur_pmd_max_sleep);
> + if (new_max_sleep != cur_pmd_max_sleep) {
> + atomic_store_relaxed(&pmd->max_sleep, new_max_sleep);
> + always_log = true;
> + }
> + }
> +
> + if (always_log) {
> + VLOG_INFO("PMD max sleep request is %"PRIu64" "
> + "usecs by default.", default_max_sleep);
Same comment about usecs.
And 'by default' is a bit confusing, but I'm not sure how to re-phrase.
> + VLOG_INFO("PMD load based sleeps are %s by default.",
> + default_max_sleep ? "enabled" : "disabled" );
The "enabled/disabled by default" log is confusing, because we do not
say the same for each thread. We only list numbers for them. It's
probbaly better to just list a number here as well.
> +
> + sorted_poll_thread_list(dp, &pmd_list, &n);
> +
> + for (size_t i = 0; i < n; i++) {
> + uint64_t cur_pmd_max_sleep;
> +
> + pmd = pmd_list[i];
> + if (pmd->core_id == NON_PMD_CORE_ID) {
> + continue;
> + }
> + atomic_read_relaxed(&pmd->max_sleep, &cur_pmd_max_sleep);
> + log_pmd_sleep(pmd->core_id, pmd->numa_id, cur_pmd_max_sleep,
> + default_max_sleep);
> + }
> + free(pmd_list);
Would be better if we only logged changes on per-thread basis, otherwise
there might be a lot of noise in the log file.
It's probably fine to log a default value though on a first start
and then on changes. If we will store the actual default value in the
pmd's local value, changes in the default value will trigger changes
in per-pmd ones and we will log them.
> + }
> + free(pmd_sleeps);
> +}
> +
> /* Parses 'affinity_list' and applies configuration if it is valid. */
> static int
> @@ -7059,5 +7248,8 @@ reload:
>
> atomic_read_relaxed(&pmd->dp->smc_enable_db,
> &pmd->ctx.smc_enable_db);
> - atomic_read_relaxed(&pmd->dp->pmd_max_sleep, &max_sleep);
> + atomic_read_relaxed(&pmd->max_sleep, &max_sleep);
> + if (max_sleep == UINT64_MAX) {
> + atomic_read_relaxed(&pmd->dp->pmd_max_sleep, &max_sleep);
> + }
This will not be needed if done during the config parsing.
>
> for (i = 0; i < poll_cnt; i++) {
> @@ -7646,4 +7838,6 @@ dp_netdev_configure_pmd(struct dp_netdev_pmd_thread
> *pmd, struct dp_netdev *dp,
> cmap_init(&pmd->tx_bonds);
>
> + set_pmd_max_sleep(dp, pmd);
init
> +
> /* Initialize DPIF function pointer to the default configured version. */
> atomic_init(&pmd->netdev_input_func, dp_netdev_impl_get_default());
> diff --git a/tests/pmd.at b/tests/pmd.at
> index df578608c..6040d558c 100644
> --- a/tests/pmd.at
> +++ b/tests/pmd.at
> @@ -66,6 +66,6 @@ dnl Checks correct pmd load based sleep is set for the
> datapath.
> dnl Checking starts from line number 'line' in ovs-vswithd.log .
> m4_define([CHECK_DP_SLEEP_MAX], [
> - SLEEP_TIME="PMD max sleep request is $1 usecs."
> - SLEEP_STATE="PMD load based sleeps are $2."
> + SLEEP_TIME="PMD max sleep request is $1 usecs by default."
> + SLEEP_STATE="PMD load based sleeps are $2 by default."
> line_st=$3
> if [[ -z "$line_st" ]]
> @@ -1279,6 +1279,6 @@ CHECK_DP_SLEEP_MAX([0], [disabled], [])
>
> AT_CHECK([ovs-appctl dpif-netdev/pmd-sleep-show], [0], [dnl
> -PMD max sleep request is 0 usecs.
> -PMD load based sleeps are disabled.
> +PMD max sleep request is 0 usecs by default.
> +PMD load based sleeps are disabled by default.
> ])
>
> @@ -1288,6 +1288,6 @@ AT_CHECK([ovs-vsctl set open_vswitch .
> other_config:pmd-sleep-max="1"])
> CHECK_DP_SLEEP_MAX([1], [enabled], [+$LINENUM])
> AT_CHECK([ovs-appctl dpif-netdev/pmd-sleep-show], [0], [dnl
> -PMD max sleep request is 1 usecs.
> -PMD load based sleeps are enabled.
> +PMD max sleep request is 1 usecs by default.
> +PMD load based sleeps are enabled by default.
> ])
>
> @@ -1297,6 +1297,6 @@ AT_CHECK([ovs-vsctl set open_vswitch .
> other_config:pmd-sleep-max="10000"])
> CHECK_DP_SLEEP_MAX([10000], [enabled], [+$LINENUM])
> AT_CHECK([ovs-appctl dpif-netdev/pmd-sleep-show], [0], [dnl
> -PMD max sleep request is 10000 usecs.
> -PMD load based sleeps are enabled.
> +PMD max sleep request is 10000 usecs by default.
> +PMD load based sleeps are enabled by default.
> ])
>
> @@ -1306,6 +1306,6 @@ AT_CHECK([ovs-vsctl set open_vswitch .
> other_config:pmd-sleep-max="0"])
> CHECK_DP_SLEEP_MAX([0], [disabled], [+$LINENUM])
> AT_CHECK([ovs-appctl dpif-netdev/pmd-sleep-show], [0], [dnl
> -PMD max sleep request is 0 usecs.
> -PMD load based sleeps are disabled.
> +PMD max sleep request is 0 usecs by default.
> +PMD load based sleeps are disabled by default.
> ])
>
> @@ -1315,6 +1315,6 @@ AT_CHECK([ovs-vsctl set open_vswitch .
> other_config:pmd-sleep-max="10001"])
> CHECK_DP_SLEEP_MAX([10000], [enabled], [+$LINENUM])
> AT_CHECK([ovs-appctl dpif-netdev/pmd-sleep-show], [0], [dnl
> -PMD max sleep request is 10000 usecs.
> -PMD load based sleeps are enabled.
> +PMD max sleep request is 10000 usecs by default.
> +PMD load based sleeps are enabled by default.
> ])
>
> @@ -1324,6 +1324,6 @@ AT_CHECK([ovs-vsctl set open_vswitch .
> other_config:pmd-sleep-max="490"])
> CHECK_DP_SLEEP_MAX([490], [enabled], [+$LINENUM])
> AT_CHECK([ovs-appctl dpif-netdev/pmd-sleep-show], [0], [dnl
> -PMD max sleep request is 490 usecs.
> -PMD load based sleeps are enabled.
> +PMD max sleep request is 490 usecs by default.
> +PMD load based sleeps are enabled by default.
> ])
>
> @@ -1333,6 +1333,6 @@ AT_CHECK([ovs-vsctl set open_vswitch .
> other_config:pmd-sleep-max="499"])
> CHECK_DP_SLEEP_MAX([499], [enabled], [+$LINENUM])
> AT_CHECK([ovs-appctl dpif-netdev/pmd-sleep-show], [0], [dnl
> -PMD max sleep request is 499 usecs.
> -PMD load based sleeps are enabled.
> +PMD max sleep request is 499 usecs by default.
> +PMD load based sleeps are enabled by default.
> ])
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
