On Thu, Jun 30, 2022 at 12:55 PM Mark Michelson <[email protected]> wrote:
Hi OVS and OVN devs,
The OVN team has considered the idea of moving inactivity probes (i.e.
OVSDB echo requests/replies) into a background thread.
OVN logical networks can be very large, meaning that ovn components such
as ovn-northd and ovn-controller may take a while to process everything
in an OVSDB database. On large clusters, we end up seeing the following
loop occur:
1. The OVN component connects to the database.
2. The OVN component must compute the entire contents of database.
3. While the OVN component is executing its main loop, the inactivity
probe interval expires. The OVN component is disconnected from database.
4. The OVN component finishes its computation.
5. Since the OVN component is disconnected from the database, it must
reconnect. Go to step 1.
This makes for an unstable and slow experience. Typically if OVN can get
past the initial loop after connecting to the database, then incremental
processing will allow for subsequent loops to execute much more quickly.
However, the constant disconnect-reconnect makes OVN operate at its
slowest at all times.
The way we've dealt with this before is to try to optimize the
performance of OVN components, while also advising that the inactivity
probe gets set to a high value. The problem is that as the demand for
larger and larger logical networks grows, the execution time of OVN is
hard to bring down much more, but the inactivity probes have to keep
getting higher and higher to avoid the described scenario. Once OVN
reaches its "stable" state where incremental processing makes loops
execute quickly, this high inactivity probe becomes detrimental. It
means that if there is a legitimate disconnection, then we don't detect
it very quickly.
Hi Mark,
Sorry for the late reply.
For the slowness of ovn-northd/ovn-controller, our practice is to disable
the inactivity probe from DB server side. It seems harmless because if the
client network is recovered it would just reconnect, right?
The problem is more with the other direction: probe from client to server.
This probe is required because if a client (e.g. ovn-controller) doesn't
need to send any transactions to the server for a long time then it would
not detect a server crash sooner, thus would not trigger reconnecting to
another server in the cluster in time.
With the client -> server probe enabled, there is a scale problem with the
SB server, if it is connected with a huge number of clients and if the
probe interval is not high enough, because when it is busy serving the
clients it may fail responding the probes in time, causing some clients
reconnecting and re-transmitting data, a cascaded failure.
So for the background thread approach may appear to be helpful for the
server side.
However, there is already an alternative to the server side scale problem
(I haven't tried yet), the OVSDB relay, which can reduce the number of
clients per server to a very low number. In this case, a short probe should
not matter.
So based on the above thoughts, I am not sure if it is really necessary to
have the background probe handling.
As mentioned at the top of the email, a possible solution is to put the
inactivity probes into a background thread. Is this in the spirit of the
inactivity probe? From my point of view, the inactivity probe should
fail only in a serious error condition, such as a network outage, or a
program crash. If a program is "busy" it is still "active" and should
therefore not be subject to inactivity probe failures. However, I want
to get the opinions of the list on this.
This is indeed a controversial point. If a program is "busy" but "active",
I agree it shouldn't be subject to probe failures. However, if the program
is not responsive at all, due to bugs, e.g. the main thread is in a dead
loop (although it is not very likely to happen in OVN components), should
the probe fail? I think the answer is yes in this case, considering the
case when a SB server node is not responsive at all because of bugs while
the background thread is still responding to probes, the client wouldn't
notice the problem and would not reconnect to a healthy node, which defies
the purpose of the probe.
