Re: [ovs-discuss] ovs-ofctl memory consumption is large compared to flow bundle size [formatting correction]

2018-03-22 Thread Armando M. 
On 20 March 2018 at 13:48, Ben Pfaff  wrote:

> On Mon, Mar 19, 2018 at 10:04:00PM -0700, Ben Pfaff wrote:
> > On Mon, Mar 12, 2018 at 03:47:16PM -0400, Michael Ben-Ami via discuss
> wrote:
> > > We found that when we add a flow bundle of about 25MB of textual flows,
> > > ovs-ofctl ballooned in resident memory to around 563MB. Similarly for a
> > > bundle about half the size at 12.4MB, ovs-ofctl hit 285MB.
> >
> > I have a branch in my "reviews" repository that should fix this:
> > https://github.com/blp/ovs-reviews/tree/memory
> >
> > It's not quite ready to post for formal review (the commit messages need
> > work and it probably leaks some memory), but if you have a minute to
> > test it out, please do consider it.
>
> I polished it up and sent it out for review:
> https://patchwork.ozlabs.org/project/openvswitch/list/?serie
> s=34920


I tried this out and the preliminary results are impressive [1]: on a 1GB
machine I couldn't even get ovs-ofctl to run (not that I did expect it to
-- getting `ovs-ofctl: virtual memory exhausted` error). Switching to the
revised version I was able to push all flows in under a second and half.
This also work correctly against 2.7.3.

I'll give the actual patches a more thorough look over the next couple of
days, but in the meantime kudos!!

Cheers,
Armando

[1] http://paste.openstack.org/show/709439/
>
>
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-ofctl memory consumption is large compared to flow bundle size [formatting correction]

2018-03-21 Thread Michael Ben-Ami via discuss 
Thank you so much Ben and team! Assuming it is merged, we will deploy as
part of our next OVS upgrade. In the meantime, we have found conjunctive
matchers to be a solid workaround to help with limiting the size of the
flow bundle, and in turn the consumed memory.

On Tue, Mar 20, 2018 at 4:48 PM, Ben Pfaff  wrote:

> On Mon, Mar 19, 2018 at 10:04:00PM -0700, Ben Pfaff wrote:
> > On Mon, Mar 12, 2018 at 03:47:16PM -0400, Michael Ben-Ami via discuss
> wrote:
> > > We found that when we add a flow bundle of about 25MB of textual flows,
> > > ovs-ofctl ballooned in resident memory to around 563MB. Similarly for a
> > > bundle about half the size at 12.4MB, ovs-ofctl hit 285MB.
> >
> > I have a branch in my "reviews" repository that should fix this:
> > https://github.com/blp/ovs-reviews/tree/memory
> >
> > It's not quite ready to post for formal review (the commit messages need
> > work and it probably leaks some memory), but if you have a minute to
> > test it out, please do consider it.
>
> I polished it up and sent it out for review:
> https://patchwork.ozlabs.org/project/openvswitch/list/?
> series=34920
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-ofctl memory consumption is large compared to flow bundle size [formatting correction]

2018-03-19 Thread Ben Pfaff 
On Mon, Mar 12, 2018 at 03:47:16PM -0400, Michael Ben-Ami via discuss wrote:
> We found that when we add a flow bundle of about 25MB of textual flows,
> ovs-ofctl ballooned in resident memory to around 563MB. Similarly for a
> bundle about half the size at 12.4MB, ovs-ofctl hit 285MB.

I have a branch in my "reviews" repository that should fix this:
https://github.com/blp/ovs-reviews/tree/memory

It's not quite ready to post for formal review (the commit messages need
work and it probably leaks some memory), but if you have a minute to
test it out, please do consider it.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] ovs-ofctl memory consumption is large compared to flow bundle size [formatting correction]

2018-03-12 Thread Michael Ben-Ami via discuss 
Hello all,

We found that when we add a flow bundle of about 25MB of textual flows,
ovs-ofctl ballooned in resident memory to around 563MB. Similarly for a
bundle about half the size at 12.4MB, ovs-ofctl hit 285MB.

We've attached environment and reproduction details below, including a link
to the actual flows we used to test with in the 25MB case.

While we did notice that the area of flow handling had some improvements in
memory management (e.g. [1]), we wonder whether there is more work in
progress or potential for regression in this area.

On the other end, we wonder if there are other ways in which flows can be
summarized from the user perspective so that calling ovs-ofctl does not
result in such a heavyweight memory footprint (albeit temporary).

We’ve also done some profiling and included the results below. Any advice
on other profiling tools or techniques would also be appreciated.

Thanks for your time,
Michael


-- Environment --

# ovs-ofctl --version
ovs-ofctl (Open vSwitch) 2.7.3
OpenFlow versions 0x1:0x4

# ovsdb-server --version
ovsdb-server (Open vSwitch) 2.7.3

# uname -a
Linux  4.9.77-1-generic #4+1516907002do~691dc34 SMP Thu Jan 25
19:04:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

-- Reproduction --

You can download the sample file we tested with directly through wget.
Opening in a browser might make the browser hang.

# wget
https://gist.githubusercontent.com/mdlayher/4a837a501a898f17615bf3841667e772/raw/90b17b40621db5589f32c7fd3f6e16ceffb41bbb/tunnel_flows.txt

# ls -al tunnel_flows.txt
-rw-r--r-- 1 root root 24870584 Mar 12 18:23 tunnel_flows.txt

# cat tunnel_flows.txt | /usr/bin/time -v ovs-ofctl --bundle add-flow
--flow-format=OXM-OpenFlow14 --protocols=OpenFlow14 br0 -
Command being timed: "ovs-ofctl --bundle add-flow
--flow-format=OXM-OpenFlow14 --protocols=OpenFlow14 br0 -"
User time (seconds): 1.44
System time (seconds): 0.33
Percent of CPU this job got: 35%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.03
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 563412
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 14236
Voluntary context switches: 511
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

-- Profiling --

#cat tunnel_flows.txt | valgrind --tool=exp-dhat ovs-ofctl --bundle
add-flow --flow-format=OXM-OpenFlow14 --protocols=OpenFlow14 br0 -
2>valgrind.out


==27766==  1 of 10 
==27766== max-live:872,415,232 in 1 blocks
==27766== tot-alloc:   872,415,232 in 1 blocks (avg size 872415232.00)
==27766== deaths:  1, at avg age 6,224,041,622 (99.98% of prog lifetime)
==27766== acc-ratios:  0.67 rd, 0.62 wr  (591,370,685 b-read, 546,028,291
b-written)
==27766==at 0x4C28EF0: malloc (in
/usr/lib/valgrind/vgpreload_exp-dhat-amd64-linux.so)
==27766==by 0x4C2B28F: realloc (in
/usr/lib/valgrind/vgpreload_exp-dhat-amd64-linux.so)
==27766==by 0x51C8F44: xrealloc (in
/usr/lib/libopenvswitch-2.7.so.1.0.3)
==27766==by 0x5177CB4: parse_ofp_flow_mod_file (in
/usr/lib/libopenvswitch-2.7.so.1.0.3)
==27766==by 0x40D1AB: ??? (in /usr/bin/ovs-ofctl)
==27766==by 0x40D2DA: ??? (in /usr/bin/ovs-ofctl)
==27766==by 0x5112CD6: ??? (in /usr/lib/libopenvswitch-2.7.so.1.0.3)
==27766==by 0x406C6B: ??? (in /usr/bin/ovs-ofctl)
==27766==by 0x56EEF44: (below main) (libc-start.c:287)

[1]
https://github.com/openvswitch/ovs/commit/5bacd5cdd6318306fb34dc42d1bd848eba19f551
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss