On 5/6/21 2:47 PM, Ilya Maximets wrote: > With a big database writing snapshot could take a lot of time, for > example, on one of the systems compaction of 300MB database takes > about 10 seconds to complete. For the clustered database, 40% of this > time takes conversion of the database to the file transaction json > format, the rest of time is formatting a string and writing to disk. > Of course, this highly depends on the disc and CPU speeds. 300MB is > the very possible database size for the OVN Southbound DB, and it might > be even bigger than that. > > During compaction the database is not available and the ovsdb-server > doesn't do any other tasks. If leader spends 10-15 seconds writing a > snapshot, the cluster is not functional for that time period. Leader > also, likely, has some monitors to serve, so the one poll interval may > be 15-20 seconds long in the end. Systems with so big databases > typically has very high election timers configured (16 seconds), so > followers will start election only after this significant amount of > time. Once leader is back to the operational state, it will > re-connect and try to join the cluster back. In some cases, this might > also trigger the 'connected' state flapping on the old leader > triggering a re-connection of clients. This issue has been observed > with large-scale OVN deployments. > > One of the methods to improve the situation is to transfer leadership > before compacting. This allows to keep the cluster functional, > while one of the servers writes a snapshot. > > Additionally logging the time spent for compaction if it was longer > than 1 second. This adds a bit of visibility to 'unreasonably long > poll interval's. > > Signed-off-by: Ilya Maximets <[email protected]> > --- > > Any other suggestions on how to improve the situation are welcome. >
I don't have any alternatives, but maybe others have ideas. Otherwise, the code in the patch looks ok to me: Acked-by: Dumitru Ceara <[email protected]> Regards, Dumitru _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
