Hi, I also found that if once theres are lots of flows, the memory (RSS) usage of OVS process would be quite high, 2~3GB. Even then the flows disappear later, the memory still keeps.
I am not sure how many people notices this, but if indeed OVS has such defect, I guess this should be critical blocker. Would like to get more info whether others meet such issue, and any clue if possible. Br, Wang Zhike ---------------------------------------------------------------------- Message: 1 Date: Sat, 18 Nov 2017 10:07:34 -0800 From: Ben Pfaff <b...@ovn.org> To: ovs-dev@openvswitch.org,William Tu <u9012...@gmail.com> Subject: Re: [ovs-dev] [PATCH] memory: kill ovs-vswitchd under super high memory usage. Message-ID: <a0efea4b-40c1-4a0f-886d-b8e39deca...@ovn.org> Content-Type: text/plain; charset=utf-8 On November 16, 2017 10:49:16 PM PST, William Tu <u9012...@gmail.com> wrote: >When deploying OVS on a large scale testbed, we occationally see OVS >gets killed by the oom (out-of-memory) killer, after installing 100k >rules and seeing ovs-vswitchd consumes more than 4GB of memory. >Unfortunately, there is no better way to debug and root cause the >memory >leak. The patch tries to add heuristic about the memory consumption >of numbers of rules and the memory usage (typically 1-2 kB per rule) >and set an upper bound for the memory usage of ovs-vswitchd. If the >memory usage, rss (resident set size), is larger than x16 num_rules, >we kill the ovs-vswitchd with SIGSEGV, hoping to generate coredump >file to help debugging. > >Signed-off-by: William Tu <u9012...@gmail.com> >--- > lib/memory.c | 26 ++++++++++++++++++++++++++ > 1 file changed, 26 insertions(+) > >diff --git a/lib/memory.c b/lib/memory.c >index da97476c6a45..75cce6e5dcc3 100644 >--- a/lib/memory.c >+++ b/lib/memory.c >@@ -25,6 +25,7 @@ > #include "timeval.h" > #include "unixctl.h" > #include "openvswitch/vlog.h" >+#include <signal.h> > > VLOG_DEFINE_THIS_MODULE(memory); > >@@ -110,6 +111,27 @@ memory_should_report(void) > } > > static void >+check_memory_usage(unsigned int num_rules) >+{ >+ struct rusage usage; >+ unsigned long int rss; >+ >+ getrusage(RUSAGE_SELF, &usage); >+ rss = (unsigned long int) usage.ru_maxrss; /* in kilobytes */ >+ >+ /* Typically a rule takes about 1-2 kilobytes of memory. If the >rss >+ * (resident set size) is larger than 1GB and x16 of num_rules, we >+ * might have a memory leak. Thus, kill it with SIGSEGV to >generate a >+ * coredump. >+ */ >+ if (rss > 1024 * 1024 && rss > num_rules * 16) { >+ VLOG_ERR("Unexpected high memory usage of %lu kB," >+ " rules %u killed with SIGSEGV", rss, num_rules); >+ raise(SIGSEGV); >+ } >+} >+ >+static void > compose_report(const struct simap *usage, struct ds *s) > { > const struct simap_node **nodes = simap_sort(usage); >@@ -120,6 +142,10 @@ compose_report(const struct simap *usage, struct >ds *s) > const struct simap_node *node = nodes[i]; > > ds_put_format(s, "%s:%u ", node->name, node->data); >+ >+ if (!strcmp(node->name, "rules")) { >+ check_memory_usage(node->data); >+ } > } > ds_chomp(s, ' '); > free(nodes); >-- >2.7.4 > >_______________________________________________ >dev mailing list >d...@openvswitch.org >https://mail.openvswitch.org/mailman/listinfo/ovs-dev I know I suggested this but I didn't mean it as something that we'd carry in the tree but only as a temporary patch while we're trying to track down the leak. ------------------------------ _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev