On 4/11/24 10:59, Chris Riches wrote:
> On 10/04/2024 23:31, Ilya Maximets wrote:
>> On 4/10/24 17:48, Chris Riches wrote:
>>> If the database is particularly large (multi-GB), ovsdb-server can take
>> Hi, Chris. May I ask how did you end up with multi-GB database?
>> I would understand if it was an OVN Southbound DB, for example,
>> but why the local database that only stores ports/bridges and
>> some other not that large things ends up with so much data?
>>
>> Sounds a little strange.
>>
>> Best regards, Ilya Maximets.
>
> I'd like to understand that too, and it's a separate RCA we're working
> on but haven't reached a conclusion yet.
>
> From what we know so far, the DB was full of stale connection-tracking
> information such as the following:
>
> {
> "_date": 1710858766431,
> "Bridge": {
> "49cb85cd-b085-4af8-98a2-56030dd614b9": {
> "external_ids": [
> "map",
> [
> [
> "ct-zone-lrp-ext_gw_port_48a89ae3-6528-4851-a277-e21db02518ad",
> "4"
> ],
> [
> "external",
> "true"
> ]
> ]
> ]
> }
> },
> "_comment": "ovn-controller: modifying OVS tunnels
> '5995b338-3080-44b1-9251-58080cc878f7'"
> }
>
> Once the host was recovered by putting in the timeout increase,
> ovsdb-server successfully started and GCed the database down from 2.4
> *GB* to 29 *KB*. Had this happened before the host restart, we would
> have never seen this problem. But since it seems possible to end up
> booting with such a large DB, we figured a timeout increase was a
> sensible measure to take.
Uff. Sounds like ovn-controller went off the rails.
Normally, ovsdb-server compacts the database once in 10-20 minutes,
if the database doubles the size since the previous check. If all
the transactions are that small, it would mean ovn-controller made
about 10K transactions per second in the 10-20 minutes before the
restart. That's huge.
I wonder if this can be addressed with a better compaction strategy.
Something like forcing compaction if "the database is more than 10 MB
and increased 10x" regardless of the time.
Bets regards, Ilya Maximets.
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
