Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez]

> On 9 May 2018, at 23:32, Ben Pfaff  wrote:
> 
> I did the backport a week ago.
> 
Thanks a lot!

> Now the logic for compaction is in storage.c, in
> ovsdb_storage_should_snapshot().  If you notice any significant
> behavioral changes in practice, let me know; I don't think that there
> should be any.
Yes, I found them and looks the same indeed. Thanks once again, this is helpful 
:)
> 
>> On Fri, Apr 27, 2018 at 07:21:32PM +0200, Daniel Alvarez Sanchez wrote:
>> Hi Ben,
>> After [0] got merged the compaction code is not there anymore. How is this
>> being done
>> now? Also, can we get the backport of [1][2] to 2.9 branch?
>> 
>> [0]
>> https://github.com/openvswitch/ovs/commit/1b1d2e6daa563cc91f974ffdc082fb3a8b424801
>> [1]
>> https://github.com/openvswitch/ovs/commit/1cfdc175ab1ecbc8f5d22f78d8e5f4344d55c5dc#diff-62fba9ea73e44f70aa9f56228bd4658c
>> [2]
>> https://github.com/openvswitch/ovs/commit/69f453713459c60e5619174186f94a0975891580
>> 
>> On Thu, Mar 8, 2018 at 11:21 PM, Daniel Alvarez Sanchez >> wrote:
>> 
>>> Ok, I've just sent a patch and if you're not convinced we can
>>> just do the 2x change. Thanks a lot!
>>> Daniel
>>> 
 On Thu, Mar 8, 2018 at 10:19 PM, Ben Pfaff  wrote:
 
 I guess I wouldn't object.
 
> On Thu, Mar 08, 2018 at 10:11:11PM +0100, Daniel Alvarez Sanchez wrote:
> Thanks Ben and Mark. I'd be okay with 2x.
> Don't you think that apart from that it can still be good to compact
 after
> a
> certain amount of time (like 1 day) if the number of transactions is > 0
> regardless of the size?
> 
>> On Thu, Mar 8, 2018 at 10:00 PM, Ben Pfaff  wrote:
>> 
>> It would be trivial to change 4x to 2x.  4x was just the suggestion in
>> the Raft thesis.  If 2x would make everyone a little more comfortable,
>> let's make that change.
>> 
 
>>> 
>>> 
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Ben Pfaff]

I did the backport a week ago.

Now the logic for compaction is in storage.c, in
ovsdb_storage_should_snapshot().  If you notice any significant
behavioral changes in practice, let me know; I don't think that there
should be any.

On Fri, Apr 27, 2018 at 07:21:32PM +0200, Daniel Alvarez Sanchez wrote:
> Hi Ben,
> After [0] got merged the compaction code is not there anymore. How is this
> being done
> now? Also, can we get the backport of [1][2] to 2.9 branch?
> 
> [0]
> https://github.com/openvswitch/ovs/commit/1b1d2e6daa563cc91f974ffdc082fb3a8b424801
> [1]
> https://github.com/openvswitch/ovs/commit/1cfdc175ab1ecbc8f5d22f78d8e5f4344d55c5dc#diff-62fba9ea73e44f70aa9f56228bd4658c
> [2]
> https://github.com/openvswitch/ovs/commit/69f453713459c60e5619174186f94a0975891580
> 
> On Thu, Mar 8, 2018 at 11:21 PM, Daniel Alvarez Sanchez  > wrote:
> 
> > Ok, I've just sent a patch and if you're not convinced we can
> > just do the 2x change. Thanks a lot!
> > Daniel
> >
> > On Thu, Mar 8, 2018 at 10:19 PM, Ben Pfaff  wrote:
> >
> >> I guess I wouldn't object.
> >>
> >> On Thu, Mar 08, 2018 at 10:11:11PM +0100, Daniel Alvarez Sanchez wrote:
> >> > Thanks Ben and Mark. I'd be okay with 2x.
> >> > Don't you think that apart from that it can still be good to compact
> >> after
> >> > a
> >> > certain amount of time (like 1 day) if the number of transactions is > 0
> >> > regardless of the size?
> >> >
> >> > On Thu, Mar 8, 2018 at 10:00 PM, Ben Pfaff  wrote:
> >> >
> >> > > It would be trivial to change 4x to 2x.  4x was just the suggestion in
> >> > > the Raft thesis.  If 2x would make everyone a little more comfortable,
> >> > > let's make that change.
> >> > >
> >>
> >
> >
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

Hi Ben,
After [0] got merged the compaction code is not there anymore. How is this
being done
now? Also, can we get the backport of [1][2] to 2.9 branch?

[0]
https://github.com/openvswitch/ovs/commit/1b1d2e6daa563cc91f974ffdc082fb3a8b424801
[1]
https://github.com/openvswitch/ovs/commit/1cfdc175ab1ecbc8f5d22f78d8e5f4344d55c5dc#diff-62fba9ea73e44f70aa9f56228bd4658c
[2]
https://github.com/openvswitch/ovs/commit/69f453713459c60e5619174186f94a0975891580

On Thu, Mar 8, 2018 at 11:21 PM, Daniel Alvarez Sanchez  wrote:

> Ok, I've just sent a patch and if you're not convinced we can
> just do the 2x change. Thanks a lot!
> Daniel
>
> On Thu, Mar 8, 2018 at 10:19 PM, Ben Pfaff  wrote:
>
>> I guess I wouldn't object.
>>
>> On Thu, Mar 08, 2018 at 10:11:11PM +0100, Daniel Alvarez Sanchez wrote:
>> > Thanks Ben and Mark. I'd be okay with 2x.
>> > Don't you think that apart from that it can still be good to compact
>> after
>> > a
>> > certain amount of time (like 1 day) if the number of transactions is > 0
>> > regardless of the size?
>> >
>> > On Thu, Mar 8, 2018 at 10:00 PM, Ben Pfaff  wrote:
>> >
>> > > It would be trivial to change 4x to 2x.  4x was just the suggestion in
>> > > the Raft thesis.  If 2x would make everyone a little more comfortable,
>> > > let's make that change.
>> > >
>>
>
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

Ok, I've just sent a patch and if you're not convinced we can
just do the 2x change. Thanks a lot!
Daniel

On Thu, Mar 8, 2018 at 10:19 PM, Ben Pfaff  wrote:

> I guess I wouldn't object.
>
> On Thu, Mar 08, 2018 at 10:11:11PM +0100, Daniel Alvarez Sanchez wrote:
> > Thanks Ben and Mark. I'd be okay with 2x.
> > Don't you think that apart from that it can still be good to compact
> after
> > a
> > certain amount of time (like 1 day) if the number of transactions is > 0
> > regardless of the size?
> >
> > On Thu, Mar 8, 2018 at 10:00 PM, Ben Pfaff  wrote:
> >
> > > It would be trivial to change 4x to 2x.  4x was just the suggestion in
> > > the Raft thesis.  If 2x would make everyone a little more comfortable,
> > > let's make that change.
> > >
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Ben Pfaff]

I guess I wouldn't object.

On Thu, Mar 08, 2018 at 10:11:11PM +0100, Daniel Alvarez Sanchez wrote:
> Thanks Ben and Mark. I'd be okay with 2x.
> Don't you think that apart from that it can still be good to compact after
> a
> certain amount of time (like 1 day) if the number of transactions is > 0
> regardless of the size?
> 
> On Thu, Mar 8, 2018 at 10:00 PM, Ben Pfaff  wrote:
> 
> > It would be trivial to change 4x to 2x.  4x was just the suggestion in
> > the Raft thesis.  If 2x would make everyone a little more comfortable,
> > let's make that change.
> >
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

Thanks Ben and Mark. I'd be okay with 2x.
Don't you think that apart from that it can still be good to compact after
a
certain amount of time (like 1 day) if the number of transactions is > 0
regardless of the size?

On Thu, Mar 8, 2018 at 10:00 PM, Ben Pfaff  wrote:

> It would be trivial to change 4x to 2x.  4x was just the suggestion in
> the Raft thesis.  If 2x would make everyone a little more comfortable,
> let's make that change.
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

I agree with you Mark. I tried to check how much it would shrink with 1800
ports in the system:

[stack@ovn ovs]$ sudo ovn-nbctl list Logical_Switch_Port | grep uuid | wc -l
1809
[stack@ovn ovs]$ sudo ovn-sbctl list Logical_Flow | grep uuid | wc -l


50780
[stack@ovn ovs]$ ls -alh ovn*.db


-rw-r--r--. 1 stack stack 15M Mar  8 15:56 ovnnb_db.db
-rw-r--r--. 1 stack stack 61M Mar  8 15:56 ovnsb_db.db
[stack@ovn ovs]$ sudo ovs-appctl -t
/usr/local/var/run/openvswitch/ovnsb_db.ctl ovsdb-server/compact
[stack@ovn ovs]$ sudo ovs-appctl -t
/usr/local/var/run/openvswitch/ovnnb_db.ctl ovsdb-server/compact


[stack@ovn ovs]$ ls -alh ovn*.db
-rw-r--r--. 1 stack stack 5.8M Mar  8 20:45 ovnnb_db.db
-rw-r--r--. 1 stack stack  23M Mar  8 20:45 ovnsb_db.db

As you can see, with ~50K lflows, the database min size would be ~23M while
the NB database
is much smaller. Still I think we need to do something to not allow delay
the compact task to
kick in this much unnecessarily. Or maybe we want some sort of
configuration (ie. normal, aggressive,...)
for this since in some situations it may help to have the full log of the
DB (although this can be
achieved through periodic backups :?). That said, I'm not a big fan of such
configs but...



On Thu, Mar 8, 2018 at 9:31 PM, Mark Michelson  wrote:

> Most of the data in this thread has been pretty easily explainable based
> on what I've seen in the code compared with the nature of the data in the
> southbound database.
>
> The southbound database tends to have more data in it than other databases
> in OVS, due especially to the Logical_Flow table. The result is that auto
> shrinking of the database does not shrink it down by as much as other
> databases. You can see in Daniel's graphs that each time the southbound
> database is shrunk, its "base" size ends up noticeably larger than it
> previously was.
>
> Couple that with the fact that the database has to increase to 4x its
> previous snapshot size in order to be shrunk, and you can end up with a
> situation after a while where the "shrunk" southbound database is 750MB,
> and it won't shrink again until it exceeds 3GB.
>
> To fix this, I think there are a few things that can be done:
>
> * Somehow make the southbound database have less data in it. I don't have
> any real good ideas for how to do this, and doing this in a
> backwards-compatible way will be difficult.
>
> * Ease the requirements for shrinking a database. For instance, once the
> database reaches a certain size, maybe it doesn't need to grow by 4x in
> order to be a candidate for shrinking. Maybe it only needs to double in
> size. Or, there could be some time cutoff where the database always will be
> shrunk. So for instance, every hour, always shrink the database, no matter
> how much activity has occurred in it (okay, maybe not if there have been 0
> transactions).


Maybe we can just do the the shrink if the last compact took place >24h ago
regardless of the other conditions.
I can send a patch for this if you guys like the idea. It's some sort of
"cleanup task" just in case and seems harmless.
What do you say?

>
>
> On 03/07/2018 02:50 PM, Ben Pfaff wrote:
>
>> OK.
>>
>> I guess we need to investigate this issue from the basics.
>>
>> On Wed, Mar 07, 2018 at 09:02:02PM +0100, Daniel Alvarez Sanchez wrote:
>>
>>> With OVS 2.8 branch it never shrank when I started to delete the ports
>>> since
>>> the DB sizes didn't grow, which makes sense to me. The conditions weren't
>>> met for further compaction.
>>> See attached image.
>>>
>>> NB:
>>> 2018-03-07T18:25:49.269Z|9|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnnb_db.db:
>>> compacting database online (647.317 seconds old, 436 transactions,
>>> 10505382
>>> bytes)
>>> 2018-03-07T18:35:51.414Z|00012|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnnb_db.db:
>>> compacting database online (602.089 seconds old, 431 transactions,
>>> 29551917
>>> bytes)
>>> 2018-03-07T18:45:52.263Z|00015|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnnb_db.db:
>>> compacting database online (600.563 seconds old, 463 transactions,
>>> 52843231
>>> bytes)
>>> 2018-03-07T18:55:53.810Z|00016|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnnb_db.db:
>>> compacting database online (601.128 seconds old, 365 transactions,
>>> 57618931
>>> bytes)
>>>
>>>
>>> SB:
>>> 2018-03-07T18:33:24.927Z|9|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnsb_db.db:
>>> compacting database online (1102.840 seconds old, 775 transactions,
>>> 10505486 bytes)
>>> 2018-03-07T18:43:27.569Z|00012|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnsb_db.db:
>>> compacting database online (602.394 seconds old, 445 transactions,
>>> 15293972
>>> bytes)
>>> 2018-03-07T18:53:31.664Z|00015|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnsb_db.db:
>>> compacting database online (603.605 seconds old, 385 transactions,
>>> 19282371
>>> bytes)
>>> 2018-03-07T19:03:42.116Z|00031|ovsdb_file|INFO|/opt/stack/
>>> data/ovs/ovnsb_db.db:
>>> compacting database online (607.542 

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Ben Pfaff]

OK.

I guess we need to investigate this issue from the basics.

On Wed, Mar 07, 2018 at 09:02:02PM +0100, Daniel Alvarez Sanchez wrote:
> With OVS 2.8 branch it never shrank when I started to delete the ports since
> the DB sizes didn't grow, which makes sense to me. The conditions weren't
> met for further compaction.
> See attached image.
> 
> NB:
> 2018-03-07T18:25:49.269Z|9|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
> compacting database online (647.317 seconds old, 436 transactions, 10505382
> bytes)
> 2018-03-07T18:35:51.414Z|00012|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
> compacting database online (602.089 seconds old, 431 transactions, 29551917
> bytes)
> 2018-03-07T18:45:52.263Z|00015|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
> compacting database online (600.563 seconds old, 463 transactions, 52843231
> bytes)
> 2018-03-07T18:55:53.810Z|00016|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
> compacting database online (601.128 seconds old, 365 transactions, 57618931
> bytes)
> 
> 
> SB:
> 2018-03-07T18:33:24.927Z|9|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
> compacting database online (1102.840 seconds old, 775 transactions,
> 10505486 bytes)
> 2018-03-07T18:43:27.569Z|00012|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
> compacting database online (602.394 seconds old, 445 transactions, 15293972
> bytes)
> 2018-03-07T18:53:31.664Z|00015|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
> compacting database online (603.605 seconds old, 385 transactions, 19282371
> bytes)
> 2018-03-07T19:03:42.116Z|00031|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
> compacting database online (607.542 seconds old, 371 transactions, 23538784
> bytes)
> 
> 
> 
> 
> On Wed, Mar 7, 2018 at 7:18 PM, Daniel Alvarez Sanchez 
> wrote:
> 
> > No worries, I just triggered the test now running OVS compiled out of
> > 2.8 branch (2.8.3). I'll post the results and investigate too.
> >
> > I have just sent a patch to fix the timing issue we can see in the traces I
> > posted. I applied it and it works, I believe it's good to fix as it gives
> > us
> > an idea of how frequent the compact is, and also to backport if you
> > agree with it.
> >
> > Thanks!
> >
> > On Wed, Mar 7, 2018 at 7:13 PM, Ben Pfaff  wrote:
> >
> >> OK, thanks.
> >>
> >> If this is a lot of trouble, let me know and I'll investigate directly
> >> instead of on the basis of a suspected regression.
> >>
> >> On Wed, Mar 07, 2018 at 07:06:50PM +0100, Daniel Alvarez Sanchez wrote:
> >> > All right, I'll repeat it with code in branch-2.8.
> >> > Will post the results once the test finishes.
> >> > Daniel
> >> >
> >> > On Wed, Mar 7, 2018 at 7:03 PM, Ben Pfaff  wrote:
> >> >
> >> > > On Wed, Mar 07, 2018 at 05:53:15PM +0100, Daniel Alvarez Sanchez
> >> wrote:
> >> > > > Repeated the test with 1000 ports this time. See attached image.
> >> > > > For some reason, the sizes grow while deleting the ports (the
> >> > > > deletion task starts at around x=2500). The weird thing is why
> >> > > > they keep growing and the online compact doesn't work as when
> >> > > > I do it through ovs-appctl tool.
> >> > > >
> >> > > > I suspect this is a bug and eventually it will grow and grow unless
> >> > > > we manually compact the db.
> >> > >
> >> > > Would you mind trying out an older ovsdb-server, for example the one
> >> > > from OVS 2.8?  Some of the logic in ovsdb-server around compaction
> >> > > changed in OVS 2.9, so it would be nice to know whether this was a
> >> > > regression or an existing bug.
> >> > >
> >>
> >
> >


___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

With OVS 2.8 branch it never shrank when I started to delete the ports since
the DB sizes didn't grow, which makes sense to me. The conditions weren't
met for further compaction.
See attached image.

NB:
2018-03-07T18:25:49.269Z|9|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (647.317 seconds old, 436 transactions, 10505382
bytes)
2018-03-07T18:35:51.414Z|00012|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (602.089 seconds old, 431 transactions, 29551917
bytes)
2018-03-07T18:45:52.263Z|00015|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (600.563 seconds old, 463 transactions, 52843231
bytes)
2018-03-07T18:55:53.810Z|00016|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (601.128 seconds old, 365 transactions, 57618931
bytes)


SB:
2018-03-07T18:33:24.927Z|9|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (1102.840 seconds old, 775 transactions,
10505486 bytes)
2018-03-07T18:43:27.569Z|00012|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (602.394 seconds old, 445 transactions, 15293972
bytes)
2018-03-07T18:53:31.664Z|00015|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (603.605 seconds old, 385 transactions, 19282371
bytes)
2018-03-07T19:03:42.116Z|00031|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (607.542 seconds old, 371 transactions, 23538784
bytes)




On Wed, Mar 7, 2018 at 7:18 PM, Daniel Alvarez Sanchez 
wrote:

> No worries, I just triggered the test now running OVS compiled out of
> 2.8 branch (2.8.3). I'll post the results and investigate too.
>
> I have just sent a patch to fix the timing issue we can see in the traces I
> posted. I applied it and it works, I believe it's good to fix as it gives
> us
> an idea of how frequent the compact is, and also to backport if you
> agree with it.
>
> Thanks!
>
> On Wed, Mar 7, 2018 at 7:13 PM, Ben Pfaff  wrote:
>
>> OK, thanks.
>>
>> If this is a lot of trouble, let me know and I'll investigate directly
>> instead of on the basis of a suspected regression.
>>
>> On Wed, Mar 07, 2018 at 07:06:50PM +0100, Daniel Alvarez Sanchez wrote:
>> > All right, I'll repeat it with code in branch-2.8.
>> > Will post the results once the test finishes.
>> > Daniel
>> >
>> > On Wed, Mar 7, 2018 at 7:03 PM, Ben Pfaff  wrote:
>> >
>> > > On Wed, Mar 07, 2018 at 05:53:15PM +0100, Daniel Alvarez Sanchez
>> wrote:
>> > > > Repeated the test with 1000 ports this time. See attached image.
>> > > > For some reason, the sizes grow while deleting the ports (the
>> > > > deletion task starts at around x=2500). The weird thing is why
>> > > > they keep growing and the online compact doesn't work as when
>> > > > I do it through ovs-appctl tool.
>> > > >
>> > > > I suspect this is a bug and eventually it will grow and grow unless
>> > > > we manually compact the db.
>> > >
>> > > Would you mind trying out an older ovsdb-server, for example the one
>> > > from OVS 2.8?  Some of the logic in ovsdb-server around compaction
>> > > changed in OVS 2.9, so it would be nice to know whether this was a
>> > > regression or an existing bug.
>> > >
>>
>
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

No worries, I just triggered the test now running OVS compiled out of
2.8 branch (2.8.3). I'll post the results and investigate too.

I have just sent a patch to fix the timing issue we can see in the traces I
posted. I applied it and it works, I believe it's good to fix as it gives us
an idea of how frequent the compact is, and also to backport if you
agree with it.

Thanks!

On Wed, Mar 7, 2018 at 7:13 PM, Ben Pfaff  wrote:

> OK, thanks.
>
> If this is a lot of trouble, let me know and I'll investigate directly
> instead of on the basis of a suspected regression.
>
> On Wed, Mar 07, 2018 at 07:06:50PM +0100, Daniel Alvarez Sanchez wrote:
> > All right, I'll repeat it with code in branch-2.8.
> > Will post the results once the test finishes.
> > Daniel
> >
> > On Wed, Mar 7, 2018 at 7:03 PM, Ben Pfaff  wrote:
> >
> > > On Wed, Mar 07, 2018 at 05:53:15PM +0100, Daniel Alvarez Sanchez wrote:
> > > > Repeated the test with 1000 ports this time. See attached image.
> > > > For some reason, the sizes grow while deleting the ports (the
> > > > deletion task starts at around x=2500). The weird thing is why
> > > > they keep growing and the online compact doesn't work as when
> > > > I do it through ovs-appctl tool.
> > > >
> > > > I suspect this is a bug and eventually it will grow and grow unless
> > > > we manually compact the db.
> > >
> > > Would you mind trying out an older ovsdb-server, for example the one
> > > from OVS 2.8?  Some of the logic in ovsdb-server around compaction
> > > changed in OVS 2.9, so it would be nice to know whether this was a
> > > regression or an existing bug.
> > >
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Ben Pfaff]

OK, thanks.

If this is a lot of trouble, let me know and I'll investigate directly
instead of on the basis of a suspected regression.

On Wed, Mar 07, 2018 at 07:06:50PM +0100, Daniel Alvarez Sanchez wrote:
> All right, I'll repeat it with code in branch-2.8.
> Will post the results once the test finishes.
> Daniel
> 
> On Wed, Mar 7, 2018 at 7:03 PM, Ben Pfaff  wrote:
> 
> > On Wed, Mar 07, 2018 at 05:53:15PM +0100, Daniel Alvarez Sanchez wrote:
> > > Repeated the test with 1000 ports this time. See attached image.
> > > For some reason, the sizes grow while deleting the ports (the
> > > deletion task starts at around x=2500). The weird thing is why
> > > they keep growing and the online compact doesn't work as when
> > > I do it through ovs-appctl tool.
> > >
> > > I suspect this is a bug and eventually it will grow and grow unless
> > > we manually compact the db.
> >
> > Would you mind trying out an older ovsdb-server, for example the one
> > from OVS 2.8?  Some of the logic in ovsdb-server around compaction
> > changed in OVS 2.9, so it would be nice to know whether this was a
> > regression or an existing bug.
> >
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

All right, I'll repeat it with code in branch-2.8.
Will post the results once the test finishes.
Daniel

On Wed, Mar 7, 2018 at 7:03 PM, Ben Pfaff  wrote:

> On Wed, Mar 07, 2018 at 05:53:15PM +0100, Daniel Alvarez Sanchez wrote:
> > Repeated the test with 1000 ports this time. See attached image.
> > For some reason, the sizes grow while deleting the ports (the
> > deletion task starts at around x=2500). The weird thing is why
> > they keep growing and the online compact doesn't work as when
> > I do it through ovs-appctl tool.
> >
> > I suspect this is a bug and eventually it will grow and grow unless
> > we manually compact the db.
>
> Would you mind trying out an older ovsdb-server, for example the one
> from OVS 2.8?  Some of the logic in ovsdb-server around compaction
> changed in OVS 2.9, so it would be nice to know whether this was a
> regression or an existing bug.
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Ben Pfaff]

On Wed, Mar 07, 2018 at 05:53:15PM +0100, Daniel Alvarez Sanchez wrote:
> Repeated the test with 1000 ports this time. See attached image.
> For some reason, the sizes grow while deleting the ports (the
> deletion task starts at around x=2500). The weird thing is why
> they keep growing and the online compact doesn't work as when
> I do it through ovs-appctl tool.
> 
> I suspect this is a bug and eventually it will grow and grow unless
> we manually compact the db.

Would you mind trying out an older ovsdb-server, for example the one
from OVS 2.8?  Some of the logic in ovsdb-server around compaction
changed in OVS 2.9, so it would be nice to know whether this was a
regression or an existing bug.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

Right, thanks Mark! Good point about the 4x, I missed that one.
I'm repeating the test for 1K ports. Not sure if I'll be able to reproduce
the 2.5GB part but it is still weird that while deleting ports (actually
deleting stuff from the DB) it doesn't get to compact the DB further
(last time it shrinked to 9MB) so maybe we have something odd here
going in the online compact. I'll post the results after this test.

On Wed, Mar 7, 2018 at 3:35 PM, Mark Michelson  wrote:

> On 03/07/2018 07:40 AM, Daniel Alvarez Sanchez wrote:
>
>> Hi folks,
>>
>> During the performance tests I've been doing lately I noticed
>> that the size of the Southbound database was around 2.5GB
>> in one of my setups. I couldn't dig further then but now I
>> decided to explore a bit more and these are the results in
>> my all-in-one OpenStack setup using OVN as a backend:
>>
>> * Created 800 ports on the same network (logical switch).
>> * Deleted those 800 ports.
>> * I logged the DB sizes for both NB and SB databases every second.
>>
>> See attached image for the results.
>>
>> At around x=2000, the creation task finished and deletion starts.
>> As you can see, there's automatic compact happening in the
>> NB database across the whole test. However, while I was deleting
>> ports, the SB database stop shrinking and keeps growing.
>>
>> After the test finished, the DB sizes remaining the same
>> thus SB database being around 34MB. It was not until I
>> manually compacted it when it finally shrinked:
>>
>>
>> [stack@ovn ovs]$ ls -alh ovnsb_db.db
>> -rw-r--r--. 1 stack stack 34M Mar  7 12:04 ovnsb_db.db
>>
>> [stack@ovn ovs]$ sudo ovs-appctl -t 
>> /usr/local/var/run/openvswitch/ovnsb_db.ctl
>> ovsdb-server/compact
>>
>> [stack@ovn ovs]$ ls -alh ovnsb_db.db
>> -rw-r--r--. 1 stack stack 207K Mar  7 13:32 ovnsb_db.db
>>
>> I'll try to investigate further in the code.
>> Thanks,
>>
>> Daniel
>>
>
> Daniel and I discussed this offline and I think the attached result does
> not necessarily indicate a bug yet.
>
> One of the requirements for the DB to compact automatically is that it
> needs to grow to at least 4x the size of the previous snapshot. In the
> graph, we can see that the first time the SB DB compacts (around x==1000),
> it shrinks down to ~5MB. We would expect the DB to compact again when it
> gets to ~20MB. This happens at around x==1900. The DB shrinks to ~9MB, so
> we would expect it to shrink again when it reaches ~36MB. When the test
> ends, the SB DB is ~34 MB, so it is not quite large enough to compact yet.
> If the test had gone a bit longer, then presumably we might have seen the
> DB shrink again.
>
> It will be interesting to see the test that leads to a 2.5GB southbound
> database.
>
> Mark!
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Mark Michelson]

On 03/07/2018 07:40 AM, Daniel Alvarez Sanchez wrote:

Hi folks,

During the performance tests I've been doing lately I noticed
that the size of the Southbound database was around 2.5GB
in one of my setups. I couldn't dig further then but now I
decided to explore a bit more and these are the results in
my all-in-one OpenStack setup using OVN as a backend:

* Created 800 ports on the same network (logical switch).
* Deleted those 800 ports.
* I logged the DB sizes for both NB and SB databases every second.

See attached image for the results.

At around x=2000, the creation task finished and deletion starts.
As you can see, there's automatic compact happening in the
NB database across the whole test. However, while I was deleting
ports, the SB database stop shrinking and keeps growing.

After the test finished, the DB sizes remaining the same
thus SB database being around 34MB. It was not until I
manually compacted it when it finally shrinked:


[stack@ovn ovs]$ ls -alh ovnsb_db.db
-rw-r--r--. 1 stack stack 34M Mar  7 12:04 ovnsb_db.db

[stack@ovn ovs]$ sudo ovs-appctl -t 
/usr/local/var/run/openvswitch/ovnsb_db.ctl ovsdb-server/compact


[stack@ovn ovs]$ ls -alh ovnsb_db.db
-rw-r--r--. 1 stack stack 207K Mar  7 13:32 ovnsb_db.db

I'll try to investigate further in the code.
Thanks,

Daniel


Daniel and I discussed this offline and I think the attached result does 
not necessarily indicate a bug yet.


One of the requirements for the DB to compact automatically is that it 
needs to grow to at least 4x the size of the previous snapshot. In the 
graph, we can see that the first time the SB DB compacts (around 
x==1000), it shrinks down to ~5MB. We would expect the DB to compact 
again when it gets to ~20MB. This happens at around x==1900. The DB 
shrinks to ~9MB, so we would expect it to shrink again when it reaches 
~36MB. When the test ends, the SB DB is ~34 MB, so it is not quite large 
enough to compact yet. If the test had gone a bit longer, then 
presumably we might have seen the DB shrink again.


It will be interesting to see the test that leads to a 2.5GB southbound 
database.


Mark!
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

BTW, I didn't spot any of these messages in the log:

https://github.com/openvswitch/ovs/blob/4cc9d1f03f83e9fac90a77ddaca0af662b2758b1/ovsdb/file.c#L615

I'll add a few traces to figure out why the auto compact is not triggering.

Also, I could see the trace when I ran it manually:
2018-03-07T13:32:21.672Z|00021|ovsdb_server|INFO|compacting OVN_Southbound
database by user request
2018-03-07T13:32:21.672Z|00022|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (1519124364.908 seconds old, 951 transactions)


On Wed, Mar 7, 2018 at 2:40 PM, Daniel Alvarez Sanchez 
wrote:

> Hi folks,
>
> During the performance tests I've been doing lately I noticed
> that the size of the Southbound database was around 2.5GB
> in one of my setups. I couldn't dig further then but now I
> decided to explore a bit more and these are the results in
> my all-in-one OpenStack setup using OVN as a backend:
>
> * Created 800 ports on the same network (logical switch).
> * Deleted those 800 ports.
> * I logged the DB sizes for both NB and SB databases every second.
>
> See attached image for the results.
>
> At around x=2000, the creation task finished and deletion starts.
> As you can see, there's automatic compact happening in the
> NB database across the whole test. However, while I was deleting
> ports, the SB database stop shrinking and keeps growing.
>
> After the test finished, the DB sizes remaining the same
> thus SB database being around 34MB. It was not until I
> manually compacted it when it finally shrinked:
>
>
> [stack@ovn ovs]$ ls -alh ovnsb_db.db
> -rw-r--r--. 1 stack stack 34M Mar  7 12:04 ovnsb_db.db
>
> [stack@ovn ovs]$ sudo ovs-appctl -t 
> /usr/local/var/run/openvswitch/ovnsb_db.ctl
> ovsdb-server/compact
>
> [stack@ovn ovs]$ ls -alh ovnsb_db.db
> -rw-r--r--. 1 stack stack 207K Mar  7 13:32 ovnsb_db.db
>
> I'll try to investigate further in the code.
> Thanks,
>
> Daniel
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] OVN Database sizes - Auto compact feature

[Daniel Alvarez Sanchez]

Hi folks,

During the performance tests I've been doing lately I noticed
that the size of the Southbound database was around 2.5GB
in one of my setups. I couldn't dig further then but now I
decided to explore a bit more and these are the results in
my all-in-one OpenStack setup using OVN as a backend:

* Created 800 ports on the same network (logical switch).
* Deleted those 800 ports.
* I logged the DB sizes for both NB and SB databases every second.

See attached image for the results.

At around x=2000, the creation task finished and deletion starts.
As you can see, there's automatic compact happening in the
NB database across the whole test. However, while I was deleting
ports, the SB database stop shrinking and keeps growing.

After the test finished, the DB sizes remaining the same
thus SB database being around 34MB. It was not until I
manually compacted it when it finally shrinked:


[stack@ovn ovs]$ ls -alh ovnsb_db.db
-rw-r--r--. 1 stack stack 34M Mar  7 12:04 ovnsb_db.db

[stack@ovn ovs]$ sudo ovs-appctl -t
/usr/local/var/run/openvswitch/ovnsb_db.ctl ovsdb-server/compact

[stack@ovn ovs]$ ls -alh ovnsb_db.db
-rw-r--r--. 1 stack stack 207K Mar  7 13:32 ovnsb_db.db

I'll try to investigate further in the code.
Thanks,

Daniel
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss