Commit 6de8868d (reconnect: Fix broken inactivity probe if there is no
other reason to wake up.) enables poll_loop to run immediately on the
next millisecond when the probe interval has expired with no receive
attempted. Its reasoning notes that "in a correctly written application
we should not fall into this case more than once in a row".
The problem. An ovsdb-server session whose peer stops reading falls into
it forever, causing the ovsdb-server process to burn CPU proportionally
to the number of sessions. This behaviour has been observed in
production, where a few ovn-octavia-provider clients raised an exception
but did not die, keeping their backlog frozen and degrading the
ovsdb-server service.
The loop spins[1] from the moment the first interval expires -- here
t+8 s with the 5 s default, t+201 s with a 180 s probe -- and the cost
grows with the session count, because every wake-up walks every
connection: 10.5% of a core with the stalled session alone, 21.9%
alongside 50 healthy sessions, 42.3% alongside 300. Killing the stalled
sessions returns the pass rate and the CPU to baseline at once, which is
what separates the pin from load.
A merely slow consumer arms the same pin. ovsdb-server sees only its
own queue shrinking, and that happens only when the kernel accepts more
data -- which depends on the peer acknowledging what it already has, not
on the peer reading each byte. The 4 kB/s row below is such a peer.
ovsdb-server has no way to know this happened: a session with a standing
backlog does not have jsonrpc_session_recv() invoked, which blocks the
calls to reconnect_receive_attempted(), so the receive that would move
the expiration can never happen unless the peer resumes reading, which a
faulty implementation might never do.
The fix. The FSM is now aware of how much is queued for the peer, and
while anything is queued stops asking for the millisecond retry that
would attempt that receive. The queue length is reported where the
backlog is updated, so the FSM always has the current value. The probe
deadline is still reported, so no session is probed or disconnected at
any time it was not before; an ovsdb-server session with a standing
backlog reaches neither, as today -- it simply costs nothing to keep.
What changes is how often the poll loop wakes. Once the backlog drains
the session is read again, so it can reach the probe and answer it, and
the ordinary probe interval applies.
The 'now + 1' path is where the guard sits, and it remains valid where
it is still reached: nothing is queued there, so the receive it exists
to permit does happen, and it fires once rather than forever.
Testing. A python client reads its backlog at a fixed rate, from not at
all to fast enough to keep the queue moving, unpatched against patched on
the same tree, against an 8 MB dump at the 5 s inactivity_probe default.
It only ever reads, so it never answers a probe: a standing queue blocks
the probe, and a reader that drains lets it out unanswered and is dropped.
Testing TCP. Unpatched and patched are passes/s, meaned over the
seconds of a 72 s run with the session alive, two probe intervals past,
and the rig not polling the server -- its replies would otherwise count
as the peer's cost. Backlog is the maximum queue ovsdb-server reported
via memory/show; dropped runs on to the disconnect, after the whole
backlog is consumed.
rate backlog dropped unpatched patched
0 kB/s 5.3 MB never 866.5 0.5
4 kB/s 5.2 MB t+1347 s 681.0 0.5
20 kB/s 4.5 MB t+279 s 35.3 0.6
50 kB/s 4.2 MB t+117 s 0.5 0.5
The bottom row does not pin unpatched despite its 4.2 MB, so what arms
this is not a backlog but one that stopped moving for a probe interval.
The middle rows arm and disarm as the queue stutters, where the patched
build holds every row flat rather than merely lower. A row that reads
at all drains in the end, and is dropped when it does: t+1347 s, t+279 s
and t+117 s as it reads faster. The 0 kB/s peer never moves its queue,
so it is the only one never dropped.
Testing SSL. ssl_send() clones what it cannot send and reports it sent,
so ovsdb's queue holds only the probe queued afterwards, and refuses new
data while that clone stands.
rate backlog dropped unpatched patched
0 kB/s 41 B never 836.6 0.5
50 kB/s 41 B t+86 s 737.6 0.5
225 kB/s 41 B t+34 s 367.4 0.5
500 kB/s 0 t+19 s 0.7 0.3
50 kB/s pins here and does not over tcp: the same peer, dump and probe
interval, with only the queue length differing. Every row that clears
the 41 B is dropped once it does, the later the slower it read -- t+86 s,
t+34 s, t+19 s -- so a pin and a disconnect are alternatives, decided by
whether the queue can move, which is why a wedged peer is never dropped
today. Sampled every 0.5 s, the probe appears at t+6 s and the pin arms
at t+10 s.
1 - passes/s is voluntary_ctxt_switches in Linux /proc/PID/status, which
poll_block() advances once per pass; poll_create_node from "ovs-appctl
coverage/show" proxies it where /proc is not available.
Fixes: 6de8868d19ea ("reconnect: Fix broken inactivity probe if there is no
other reason to wake up.")
Signed-off-by: Aeliton G. Silva <[email protected]>
Assisted-by: Claude Opus 5, Claude Code
---
Notes:
v2:
- Report the queued bytes at the queueing point, not from activity
reporting, which left v1 holding a stale zero over TLS.
- Guard the 'now + 1' retry alone, not the whole probe deadline, so
raft, replication, the IDL and ovsdb-client -- which attempt a
receive every pass -- keep the probe and timeout they have today.
- Guard the same wake-up in S_IDLE, which the ssl check reaches: its
41 B backlog is the probe queued there and refused.
- Add the stall client and two ovsdb-server checks using it, in the
shape of "ovsdb-server combines updates on backlogged connections"
but over tcp and ssl.
lib/jsonrpc.c | 9 +-
lib/reconnect.c | 17 +-
lib/reconnect.h | 2 +
python/ovs/jsonrpc.py | 6 +-
python/ovs/reconnect.py | 13 +-
tests/automake.mk | 1 +
tests/ovsdb-server.at | 141 +++++++++++++++
tests/ovsdb-stall-client.py | 138 +++++++++++++++
tests/reconnect.at | 343 ++++++++++++++++++++++++++++++++++++
tests/test-reconnect.c | 21 ++-
tests/test-reconnect.py | 14 +-
11 files changed, 693 insertions(+), 12 deletions(-)
create mode 100644 tests/ovsdb-stall-client.py
diff --git a/lib/jsonrpc.c b/lib/jsonrpc.c
index f01a2e56f..844c8110f 100644
--- a/lib/jsonrpc.c
+++ b/lib/jsonrpc.c
@@ -1048,12 +1048,15 @@ jsonrpc_session_run(struct jsonrpc_session *s)
}
if (s->rpc) {
- size_t backlog;
+ size_t backlog_before;
+ size_t backlog_after;
int error;
- backlog = jsonrpc_get_backlog(s->rpc);
+ backlog_before = jsonrpc_get_backlog(s->rpc);
jsonrpc_run(s->rpc);
- if (jsonrpc_get_backlog(s->rpc) < backlog) {
+ backlog_after = jsonrpc_get_backlog(s->rpc);
+ reconnect_set_queued_bytes(s->reconnect, backlog_after);
+ if (backlog_after < backlog_before) {
/* Data previously caught in a queue was successfully sent (or
* there's an error, which we'll catch below.)
*
diff --git a/lib/reconnect.c b/lib/reconnect.c
index 918ecd203..87dc7715d 100644
--- a/lib/reconnect.c
+++ b/lib/reconnect.c
@@ -62,6 +62,7 @@ struct reconnect {
long long int last_connected;
long long int last_disconnected;
long long int last_receive_attempt;
+ size_t queued_bytes;
unsigned int max_tries;
unsigned int backoff_free_tries;
@@ -112,6 +113,7 @@ reconnect_create(long long int now)
fsm->last_connected = LLONG_MAX;
fsm->last_disconnected = LLONG_MAX;
fsm->last_receive_attempt = now;
+ fsm->queued_bytes = 0;
fsm->max_tries = UINT_MAX;
fsm->creation_time = now;
@@ -340,6 +342,7 @@ reconnect_force_reconnect(struct reconnect *fsm, long long
int now)
void
reconnect_disconnected(struct reconnect *fsm, long long int now, int error)
{
+ fsm->queued_bytes = 0;
if (!(fsm->state & (S_BACKOFF | S_VOID))) {
/* Report what happened. */
if (fsm->state & (S_ACTIVE | S_IDLE)) {
@@ -504,6 +507,16 @@ reconnect_activity(struct reconnect *fsm, long long int
now)
fsm->last_activity = now;
}
+/* Tell 'fsm' how much data is currently queued for the peer and has not been
+ * sent. While it is nonzero the FSM does not ask to be woken up to attempt a
+ * receive: the caller evidently cannot get data to this peer, so no receive it
+ * makes can settle anything. */
+void
+reconnect_set_queued_bytes(struct reconnect *fsm, size_t queued_bytes)
+{
+ fsm->queued_bytes = queued_bytes;
+}
+
/* Tell 'fsm' that some attempt to receive data on the connection was made at
* 'now'. The FSM only allows probe interval timer to expire when some attempt
* to receive data on the connection was received after the time when it should
@@ -564,7 +577,7 @@ reconnect_deadline__(const struct reconnect *fsm, long long
int now)
* we need to wait for the expiration, in the second - we're
* already past the deadline. */
return expiration;
- } else {
+ } else if (!fsm->queued_bytes) {
/* Time has already passed, but we didn't attempt to receive
* anything. We need to wake up and try to receive even if
* nothing is pending, so we can update the expiration time or
@@ -579,7 +592,7 @@ reconnect_deadline__(const struct reconnect *fsm, long long
int now)
long long int expiration = fsm->state_entered +
fsm->probe_interval;
if (now < expiration || fsm->last_receive_attempt >= expiration) {
return expiration;
- } else {
+ } else if (!fsm->queued_bytes) {
return now + 1;
}
}
diff --git a/lib/reconnect.h b/lib/reconnect.h
index 40cc569c4..f3c461d15 100644
--- a/lib/reconnect.h
+++ b/lib/reconnect.h
@@ -31,6 +31,7 @@
* revisited later.) */
#include <stdbool.h>
+#include <stddef.h>
struct reconnect *reconnect_create(long long int now);
void reconnect_destroy(struct reconnect *);
@@ -83,6 +84,7 @@ void reconnect_connected(struct reconnect *, long long int
now);
void reconnect_connect_failed(struct reconnect *, long long int now,
int error);
void reconnect_activity(struct reconnect *, long long int now);
+void reconnect_set_queued_bytes(struct reconnect *, size_t queued_bytes);
void reconnect_receive_attempted(struct reconnect *, long long int now);
enum reconnect_action {
diff --git a/python/ovs/jsonrpc.py b/python/ovs/jsonrpc.py
index 07b454a21..c7ec123e3 100644
--- a/python/ovs/jsonrpc.py
+++ b/python/ovs/jsonrpc.py
@@ -492,9 +492,11 @@ class Session(object):
self.pstream = None
if self.rpc:
- backlog = self.rpc.get_backlog()
+ backlog_before = self.rpc.get_backlog()
self.rpc.run()
- if self.rpc.get_backlog() < backlog:
+ backlog_after = self.rpc.get_backlog()
+ self.reconnect.set_queued_bytes(backlog_after)
+ if backlog_after < backlog_before:
# Data previously caught in a queue was successfully sent (or
# there's an error, which we'll catch below).
#
diff --git a/python/ovs/reconnect.py b/python/ovs/reconnect.py
index 6b8e49afd..38b2f7e32 100644
--- a/python/ovs/reconnect.py
+++ b/python/ovs/reconnect.py
@@ -104,7 +104,7 @@ class Reconnect(object):
# case we need to wait for the expiration, in the second -
# we're already past the deadline. */
return expiration
- else:
+ elif not fsm.queued_bytes:
# Time has already passed, but we didn't attempt to receive
# anything. We need to wake up and try to receive even if
# nothing is pending, so we can update the expiration time
@@ -132,7 +132,7 @@ class Reconnect(object):
fsm.last_receive_attempt is None or
fsm.last_receive_attempt >= expiration):
return expiration
- else:
+ elif not fsm.queued_bytes:
return now + 1
return None
@@ -174,6 +174,7 @@ class Reconnect(object):
self.last_connected = None
self.last_disconnected = None
self.last_receive_attempt = now
+ self.queued_bytes = 0
self.max_tries = None
self.backoff_free_tries = 0
@@ -347,6 +348,7 @@ class Reconnect(object):
error.
The FSM will back off, then reconnect."""
+ self.queued_bytes = 0
if self.state not in (Reconnect.Backoff, Reconnect.Void):
# Report what happened
if self.state in (Reconnect.Active, Reconnect.Idle):
@@ -493,6 +495,13 @@ class Reconnect(object):
self._transition(now, Reconnect.Active)
self.last_activity = now
+ def set_queued_bytes(self, queued_bytes):
+ """Tell this FSM how much data is currently queued for the peer and
+ has not been sent. While it is nonzero the FSM does not ask to be
+ woken up to attempt a receive: the caller evidently cannot get data to
+ this peer, so no receive it makes can settle anything."""
+ self.queued_bytes = queued_bytes
+
def receive_attempted(self, now):
"""Tell 'fsm' that some attempt to receive data on the connection was
made at 'now'. The FSM only allows probe interval timer to expire when
diff --git a/tests/automake.mk b/tests/automake.mk
index 2f3e4f7ef..fcbb1c439 100644
--- a/tests/automake.mk
+++ b/tests/automake.mk
@@ -534,6 +534,7 @@ CHECK_PYFILES = \
tests/appctl.py \
tests/flowgen.py \
tests/ovsdb-monitor-sort.py \
+ tests/ovsdb-stall-client.py \
tests/system-dpdk-find-device.py \
tests/test-daemon.py \
tests/test-dpparse.py \
diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at
index 7042b1fd3..c4c99964e 100644
--- a/tests/ovsdb-server.at
+++ b/tests/ovsdb-server.at
@@ -3161,3 +3161,144 @@ TEST_CONFIG_FILE([complex config], [
}
}
], [0])
+
+AT_BANNER([OVSDB -- ovsdb-server poll loop])
+
+AT_SETUP([ovsdb-server does not spin on a peer that stops reading])
+AT_KEYWORDS([ovsdb server positive tcp poll])
+dnl A session with a standing backlog is never received from, so the probe
+dnl expiration never moves and the poll loop can end up waking every
+dnl millisecond for as long as the peer stays wedged. poll_create_node counts
+dnl one registration per fd per pass, so its rate stands in for the wake-up
+dnl rate and is reachable through ovs-appctl.
+ordinal_schema > schema
+on_exit 'kill `cat *.pid` 2>/dev/null'
+AT_CHECK([ovsdb-tool create db schema], [0], [stdout], [ignore])
+AT_CHECK([ovsdb-server --log-file --detach --no-chdir --pidfile dnl
+ --remote=ptcp:0:127.0.0.1 db], [0], [ignore], [ignore])
+PARSE_LISTENING_PORT([ovsdb-server.log], [TCP_PORT])
+
+dnl The cumulative poll_create_node counter: one count per fd the poll loop
+dnl registers, per pass. It stands in for the pass rate, scaled by however
+dnl many fds the server holds, so the check below is a ratio against an idle
+dnl baseline sampled in this same run rather than a fixed number.
+m4_define([POLL_CREATE_NODE_TOTAL],
+ [ovs-appctl -t ovsdb-server coverage/show dnl
+ | awk '/^poll_create_node/ { for (i = 1; i <= NF; i++) dnl
+ if ($i == "total:") { print $(i + 1); exit } }'])
+dnl coverage/show omits a counter it has never hit, so an idle server
+dnl that has not folded a poll_create_node total yet prints no line and
+dnl the sample comes back empty. Default to 0 rather than letting expr
+dnl fail: an empty operand made a malformed comparison here before.
+idle_before=`POLL_CREATE_NODE_TOTAL`
+sleep 2
+idle_after=`POLL_CREATE_NODE_TOTAL`
+: ${idle_before:=0} ${idle_after:=0}
+dnl No parentheses: expr takes them as operands and fails with "syntax error:
+dnl unexpected ')'".
+idle_delta=`expr $idle_after - $idle_before`
+idle_rate=`expr $idle_delta / 2`
+
+dnl Seed the database and then monitor it from a connection that never reads.
+dnl The seed is what guarantees a backlog: the reply has to exceed whatever
+dnl the socket will buffer, and autotuning does not grow a buffer for a peer
+dnl that is not reading. If a host ever buffers all of it, the wait below
+dnl times out rather than measuring: raise the row count, do not shrink the
+dnl socket, which would stop a reading peer from arming at all.
+$PYTHON3 $srcdir/ovsdb-stall-client.py 127.0.0.1:$TCP_PORT 200 60000 dnl
+ > stall.log 2>&1 &
+echo $! > stall.pid
+OVS_WAIT_UNTIL([grep -q READY stall.log])
+
+dnl The arm is a hard precondition, not an assumption: without a standing
+dnl backlog this test would pass on a server that still spins.
+OVS_WAIT_UNTIL([test `ovs-appctl -t ovsdb-server memory/show dnl
+ | tr ' ' '\n' | sed -n 's/^backlog:\(.*\)/\1/p'` -gt 0])
+
+dnl Past the 5 s default probe of a command-line remote, which is when the
+dnl expiration lapses and the fast wake-up would begin.
+sleep 6
+
+stalled_before=`POLL_CREATE_NODE_TOTAL`
+sleep 2
+stalled_after=`POLL_CREATE_NODE_TOTAL`
+: ${stalled_before:=0} ${stalled_after:=0}
+stalled_delta=`expr $stalled_after - $stalled_before`
+stalled_rate=`expr $stalled_delta / 2`
+
+dnl Unpatched, this test measured 4266/s (tcp) and 3315/s (ssl) here against
+dnl an idle 10/s. A tenfold allowance over the idle baseline leaves the
+dnl limit a factor of twenty below the spinning case, with margin both ways.
+limit=`expr $idle_rate + 10`
+limit=`expr $limit \* 10`
+echo "poll_create_node/s: idle $idle_rate stalled $stalled_rate dnl
+ limit $limit"
+AT_CHECK([test -n "$stalled_rate" && test -n "$limit" dnl
+ && test $stalled_rate -lt $limit], [0], [], [])
+
+dnl Dropping the wedged peer at the ordinary probe is the CORRECT outcome and
+dnl logs an ERR; only the wake-up rate is under test here.
+OVSDB_SERVER_SHUTDOWN(["/no response to inactivity probe/d"])
+AT_CLEANUP
+
+AT_SETUP([ovsdb-server does not spin on an SSL peer that stops reading])
+AT_KEYWORDS([ovsdb server positive ssl tls poll])
+dnl The tcp case above is not enough on its own. Over tcp a stalled
+dnl peer leaves megabytes in ovsdb's own queue, so a guard fed a stale
+dnl copy of that length still holds a large value and the pin does not
+dnl appear. Over ssl the payload is accounted as sent and only the
+dnl queued probe stands -- tens of bytes -- which is the shape seen in
+dnl production, and the one that catches a guard reading anything but
+dnl the live queue length.
+AT_SKIP_IF([test "$HAVE_OPENSSL" = no])
+PKIDIR="$abs_top_builddir/tests"
+AT_SKIP_IF([expr "$PKIDIR" : ".*[[ '\"
+\\]]"])
+ordinal_schema > schema
+on_exit 'kill `cat *.pid` 2>/dev/null'
+AT_CHECK([ovsdb-tool create db schema], [0], [stdout], [ignore])
+AT_CHECK([ovsdb-server --log-file --detach --no-chdir --pidfile dnl
+ --private-key=$PKIDIR/testpki-privkey.pem dnl
+ --certificate=$PKIDIR/testpki-cert.pem dnl
+ --ca-cert=$PKIDIR/testpki-cacert.pem dnl
+ --remote=pssl:0:127.0.0.1 db], [0], [ignore], [ignore])
+PARSE_LISTENING_PORT([ovsdb-server.log], [SSL_PORT])
+
+idle_before=`POLL_CREATE_NODE_TOTAL`
+sleep 2
+idle_after=`POLL_CREATE_NODE_TOTAL`
+: ${idle_before:=0} ${idle_after:=0}
+idle_delta=`expr $idle_after - $idle_before`
+idle_rate=`expr $idle_delta / 2`
+
+$PYTHON3 $srcdir/ovsdb-stall-client.py 127.0.0.1:$SSL_PORT 200 60000 dnl
+ --tls $PKIDIR/testpki-privkey.pem $PKIDIR/testpki-cert.pem dnl
+ $PKIDIR/testpki-cacert.pem > stall.log 2>&1 &
+echo $! > stall.pid
+OVS_WAIT_UNTIL([grep -q READY stall.log])
+
+dnl Over ssl the standing queue is small -- the probe alone -- so this
+dnl waits for any backlog at all, not for a large one.
+OVS_WAIT_UNTIL([test `ovs-appctl -t ovsdb-server memory/show dnl
+ | tr ' ' '\n' | sed -n 's/^backlog:\(.*\)/\1/p'` -gt 0])
+
+sleep 6
+
+stalled_before=`POLL_CREATE_NODE_TOTAL`
+sleep 2
+stalled_after=`POLL_CREATE_NODE_TOTAL`
+: ${stalled_before:=0} ${stalled_after:=0}
+stalled_delta=`expr $stalled_after - $stalled_before`
+stalled_rate=`expr $stalled_delta / 2`
+
+limit=`expr $idle_rate + 10`
+limit=`expr $limit \* 10`
+echo "poll_create_node/s: idle $idle_rate stalled $stalled_rate dnl
+ limit $limit"
+AT_CHECK([test -n "$stalled_rate" && test -n "$limit" dnl
+ && test $stalled_rate -lt $limit], [0], [], [])
+
+dnl Dropping the wedged peer at the ordinary probe is the CORRECT outcome and
+dnl logs an ERR; only the wake-up rate is under test here.
+OVSDB_SERVER_SHUTDOWN(["/no response to inactivity probe/d"])
+AT_CLEANUP
diff --git a/tests/ovsdb-stall-client.py b/tests/ovsdb-stall-client.py
new file mode 100644
index 000000000..cfa0057f2
--- /dev/null
+++ b/tests/ovsdb-stall-client.py
@@ -0,0 +1,138 @@
+# Copyright (c) 2026 Open vSwitch project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Seeds a database, then monitors it from a peer that does not keep up.
+
+Used by the ovsdb-server checks that a peer which stops taking data does not
+drive the poll loop. Two connections are made: the first inserts enough rows
+that one monitor reply cannot fit in the socket buffers, the second issues the
+monitor and then reads at --throttle bytes a second, so the reply stands in
+ovsdb-server's own queue.
+
+--throttle 0, the default and what those checks use, never reads at all. A
+positive rate arms the same case whenever it is slow enough that the queue
+does not move within a probe interval, which is a property of the receive
+buffer as much as the rate: a receiver defers window updates until a
+worthwhile part of its buffer is free, so the queue stalls for roughly
+buffer/rate seconds. The socket is left with whatever the kernel gives it,
+as a real peer would be.
+
+Prints READY once the monitor request has been sent.
+"""
+
+import argparse
+import json
+import socket
+import ssl
+import sys
+import time
+
+TLS = None # (private key, certificate, CA certificate), or None for plain tcp
+
+
+def connect(target):
+ host, port = target.rsplit(':', 1)
+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ sock.connect((host, int(port)))
+ if TLS:
+ key, cert, ca = TLS
+ ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+ ctx.load_cert_chain(cert, key)
+ ctx.load_verify_locations(ca)
+ # No cert for "127.0.0.1"; ovsdb authenticates by CA, not name.
+ ctx.check_hostname = False
+ sock = ctx.wrap_socket(sock)
+ return sock
+
+
+def transact(sock, ops):
+ sock.sendall(json.dumps({"method": "transact",
+ "params": ops, "id": 0}).encode())
+ buf = b""
+ while True:
+ chunk = sock.recv(65536)
+ if not chunk:
+ sys.exit("ovsdb-stall-client: server closed while seeding")
+ buf += chunk
+ try:
+ return json.loads(buf.decode())
+ except ValueError:
+ continue
+
+
+def drain(sock, rate):
+ """Reads 'rate' bytes a second from 'sock', forever."""
+ slice_ = max(1, rate // 10)
+ while True:
+ deadline = time.time() + 0.1
+ got = 0
+ # recv() returns what is buffered, which is less than asked for
+ # whenever the receive buffer is small. Keep going until the slice
+ # is filled, or the window is up, so the rate is the one requested.
+ while got < slice_:
+ try:
+ data = sock.recv(slice_ - got)
+ except OSError:
+ return
+ if not data:
+ return
+ got += len(data)
+ if time.time() >= deadline:
+ break
+ delay = deadline - time.time()
+ if delay > 0:
+ time.sleep(delay)
+
+
+def main(argv):
+ global TLS
+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+ parser.add_argument("target", help="[HOST:]PORT of the ovsdb-server")
+ parser.add_argument("rows", type=int, help="rows to seed")
+ parser.add_argument("blobsize", type=int, help="bytes per seeded row")
+ parser.add_argument("--throttle", type=int, default=0, metavar="BYTES",
+ help="read this many bytes a second (0: never read)")
+ parser.add_argument("--tls", nargs=3, default=None,
+ metavar=("PRIVKEY", "CERT", "CACERT"),
+ help="speak TLS, with these PEM files")
+ args = parser.parse_args(argv[1:])
+ target, rows, blob = args.target, args.rows, args.blobsize
+ if args.tls:
+ TLS = tuple(args.tls)
+
+ seed = connect(target)
+ ops = ["ordinals"]
+ ops += [{"op": "insert", "table": "ordinals",
+ "row": {"number": i, "name": "x" * blob}} for i in range(rows)]
+ reply = transact(seed, ops)
+ if reply.get("error"):
+ sys.exit("ovsdb-stall-client: seed failed: %s" % reply["error"])
+
+ stall = connect(target)
+ stall.sendall(json.dumps(
+ {"method": "monitor",
+ "params": ["ordinals", None,
+ {"ordinals": [{"columns": ["number", "name"]}]}],
+ "id": 1}).encode())
+
+ print("READY")
+ sys.stdout.flush()
+ if args.throttle:
+ drain(stall, args.throttle)
+ while True:
+ time.sleep(60)
+
+
+if __name__ == '__main__':
+ main(sys.argv)
diff --git a/tests/reconnect.at b/tests/reconnect.at
index 5bca84351..7433ad5c5 100644
--- a/tests/reconnect.at
+++ b/tests/reconnect.at
@@ -1363,3 +1363,346 @@ run
listening
in LISTENING for 0 ms (0 ms backoff)
])
+
+######################################################################
+dnl The guard belongs on the 1 ms retry alone. That retry is reached only once
+dnl the interval has lapsed *and* no receive was attempted. ovsdb-server gates
+dnl its receive on any backlog at all -- "if (!jsonrpc_session_get_backlog())"
+dnl -- so a wedged session of its own never attempts one, never leaves that
+dnl retry, and is neither probed nor dropped: the echo is not even queued for
+dnl it. Every other jsonrpc_session user calls jsonrpc_session_recv()
+dnl unconditionally, so for those the deadline is still reached with a queue
+dnl standing. Guarding the whole probe-deadline branch instead would withhold
+dnl the probe and the disconnect from them -- a raft leader would hold a
+dnl follower that stopped taking data for ever -- and no ovsdb-server check
+dnl would notice.
+RECONNECT_CHECK([receive attempts still reach the probe with data queued],
+ [enable
+run
+connected
+
+# A caller that does not gate its receive on the backlog: raft, replication,
+# ovsdb-client and the IDL all call jsonrpc_session_recv() unconditionally,
+# so the receive attempt keeps landing while the queue stands.
+queued-bytes 182346528
+advance 5000
+receive-attempted now
+run
+
+# The probe is queued behind the backlog, so the peer never sees it and
+# cannot answer. The disconnect runs off the timer rather than off the
+# answer, so the session is still dropped at the ordinary deadline, which
+# is what a peer that has stopped taking data gets today.
+advance 5000
+receive-attempted now
+run
+],
+ [### t=1000 ###
+enable
+ in BACKOFF for 0 ms (0 ms backoff)
+run
+ should connect
+connected
+ in ACTIVE for 0 ms (0 ms backoff)
+ created 1000, last activity 1000, last connected 1000
+ 1 successful connections out of 1 attempts, seqno 1
+ connected
+ last connected 0 ms ago, connected 0 ms total
+
+# A caller that does not gate its receive on the backlog: raft, replication,
+# ovsdb-client and the IDL all call jsonrpc_session_recv() unconditionally,
+# so the receive attempt keeps landing while the queue stands.
+queued-bytes 182346528
+advance 5000
+
+### t=6000 ###
+ in ACTIVE for 5000 ms (0 ms backoff)
+receive-attempted now
+run
+ should send probe
+ in IDLE for 0 ms (0 ms backoff)
+
+# The probe is queued behind the backlog, so the peer never sees it and
+# cannot answer. The disconnect runs off the timer rather than off the
+# answer, so the session is still dropped at the ordinary deadline, which
+# is what a peer that has stopped taking data gets today.
+advance 5000
+
+### t=11000 ###
+ in IDLE for 5000 ms (0 ms backoff)
+receive-attempted now
+run
+ should disconnect
+])
+######################################################################
+RECONNECT_CHECK([no wake-up while data is queued],
+ [enable
+
+# Connection succeeds.
+run
+connected
+
+# Data is queued for the peer that we could not send.
+activity 1000
+
+# Long past the probe interval the FSM asks for no wake-up at all, and
+# nothing happens: the connection is neither probed nor disconnected.
+advance 60000
+timeout
+run
+],
+ [### t=1000 ###
+enable
+ in BACKOFF for 0 ms (0 ms backoff)
+
+# Connection succeeds.
+run
+ should connect
+connected
+ in ACTIVE for 0 ms (0 ms backoff)
+ created 1000, last activity 1000, last connected 1000
+ 1 successful connections out of 1 attempts, seqno 1
+ connected
+ last connected 0 ms ago, connected 0 ms total
+
+# Data is queued for the peer that we could not send.
+activity 1000
+
+# Long past the probe interval the FSM asks for no wake-up at all, and
+# nothing happens: the connection is neither probed nor disconnected.
+advance 60000
+
+### t=61000 ###
+ in ACTIVE for 60000 ms (0 ms backoff)
+timeout
+ no timeout
+run
+])
+
+######################################################################
+RECONNECT_CHECK([draining the queue restores probing],
+ [enable
+run
+connected
+
+# Queued: no wake-up, no probe.
+activity 1000
+advance 10000
+run
+
+# The queue drains, so the ordinary probe interval applies again.
+activity 0
+receive-attempted LLONG_MAX
+timeout
+run
+],
+ [### t=1000 ###
+enable
+ in BACKOFF for 0 ms (0 ms backoff)
+run
+ should connect
+connected
+ in ACTIVE for 0 ms (0 ms backoff)
+ created 1000, last activity 1000, last connected 1000
+ 1 successful connections out of 1 attempts, seqno 1
+ connected
+ last connected 0 ms ago, connected 0 ms total
+
+# Queued: no wake-up, no probe.
+activity 1000
+advance 10000
+
+### t=11000 ###
+ in ACTIVE for 10000 ms (0 ms backoff)
+run
+
+# The queue drains, so the ordinary probe interval applies again.
+activity 0
+ created 1000, last activity 11000, last connected 1000
+receive-attempted LLONG_MAX
+timeout
+ advance 5000 ms
+
+### t=16000 ###
+ in ACTIVE for 15000 ms (0 ms backoff)
+run
+ should send probe
+ in IDLE for 0 ms (0 ms backoff)
+])
+
+######################################################################
+RECONNECT_CHECK([disconnect forgets queued data],
+ [enable
+run
+connected
+activity 1000
+
+# Disconnecting forgets the queued data.
+disconnected
+run
+connected
+
+# So the ordinary probe interval applies again, and the millisecond retry
+# past it is reached -- which a queue left over from the old connection
+# would suppress.
+timeout
+timeout
+],
+ [### t=1000 ###
+enable
+ in BACKOFF for 0 ms (0 ms backoff)
+run
+ should connect
+connected
+ in ACTIVE for 0 ms (0 ms backoff)
+ created 1000, last activity 1000, last connected 1000
+ 1 successful connections out of 1 attempts, seqno 1
+ connected
+ last connected 0 ms ago, connected 0 ms total
+activity 1000
+
+# Disconnecting forgets the queued data.
+disconnected
+ in BACKOFF for 0 ms (1000 ms backoff)
+ 1 successful connections out of 1 attempts, seqno 2
+ disconnected
+ disconnected at 1000 ms (0 ms ago)
+run
+connected
+ in ACTIVE for 0 ms (1000 ms backoff)
+ 2 successful connections out of 2 attempts, seqno 3
+ connected
+
+# So the ordinary probe interval applies again, and the millisecond retry
+# past it is reached -- which a queue left over from the old connection
+# would suppress.
+timeout
+ advance 5000 ms
+
+### t=6000 ###
+ in ACTIVE for 5000 ms (1000 ms backoff)
+timeout
+ advance 1 ms
+
+### t=6001 ###
+ in ACTIVE for 5001 ms (1000 ms backoff)
+])
+
+dnl A peer that stops reading leaves the queue standing, so there is no
+dnl activity left to report and the length has to be reported on its own.
+dnl Remove the 'queued-bytes' line from the input below and the second
+dnl timeout returns "advance 1 ms": the wake-up this suppresses.
+RECONNECT_CHECK([no wake-up when data is queued after the last activity],
+ [enable
+
+# Connection succeeds.
+run
+connected
+
+# The queue drained just before the peer stalled, so the last activity the
+# caller had to report carried an empty queue.
+activity 0
+
+# The peer stops reading. There is no further activity to report; the length
+# of the standing queue is all the caller can still publish, and it must keep
+# publishing it.
+queued-bytes 182346528
+
+# The FSM still sleeps to the probe deadline: that is an ordinary wake-up,
+# not the 1 ms retry. Only once the deadline has lapsed with no receive
+# attempted -- the retry's own case -- does it ask for no wake-up at all,
+# because nothing a receive could do would settle anything while the caller
+# cannot get data to this peer.
+timeout
+timeout
+],
+ [### t=1000 ###
+enable
+ in BACKOFF for 0 ms (0 ms backoff)
+
+# Connection succeeds.
+run
+ should connect
+connected
+ in ACTIVE for 0 ms (0 ms backoff)
+ created 1000, last activity 1000, last connected 1000
+ 1 successful connections out of 1 attempts, seqno 1
+ connected
+ last connected 0 ms ago, connected 0 ms total
+
+# The queue drained just before the peer stalled, so the last activity the
+# caller had to report carried an empty queue.
+activity 0
+
+# The peer stops reading. There is no further activity to report; the length
+# of the standing queue is all the caller can still publish, and it must keep
+# publishing it.
+queued-bytes 182346528
+
+# The FSM still sleeps to the probe deadline: that is an ordinary wake-up,
+# not the 1 ms retry. Only once the deadline has lapsed with no receive
+# attempted -- the retry's own case -- does it ask for no wake-up at all,
+# because nothing a receive could do would settle anything while the caller
+# cannot get data to this peer.
+timeout
+ advance 5000 ms
+
+### t=6000 ###
+ in ACTIVE for 5000 ms (0 ms backoff)
+timeout
+ no timeout
+])
+
+dnl The same 1 ms wake-up is reachable from IDLE: a quiet connection is probed,
+dnl and only then does the peer stop taking data. The guard belongs in both
+dnl states, not only in ACTIVE. The probe deadline itself is still
+dnl reported, so a peer that is being received from is still dropped on
+dnl time.
+RECONNECT_CHECK([no wake-up while data is queued in IDLE],
+ [enable
+run
+connected
+
+# A quiet connection is probed, so the FSM is in IDLE.
+timeout
+receive-attempted now
+run
+
+# Only now does the peer stop taking data.
+queued-bytes 182346528
+timeout
+timeout
+],
+ [### t=1000 ###
+enable
+ in BACKOFF for 0 ms (0 ms backoff)
+run
+ should connect
+connected
+ in ACTIVE for 0 ms (0 ms backoff)
+ created 1000, last activity 1000, last connected 1000
+ 1 successful connections out of 1 attempts, seqno 1
+ connected
+ last connected 0 ms ago, connected 0 ms total
+
+# A quiet connection is probed, so the FSM is in IDLE.
+timeout
+ advance 5000 ms
+
+### t=6000 ###
+ in ACTIVE for 5000 ms (0 ms backoff)
+receive-attempted now
+run
+ should send probe
+ in IDLE for 0 ms (0 ms backoff)
+
+# Only now does the peer stop taking data.
+queued-bytes 182346528
+timeout
+ advance 5000 ms
+
+### t=11000 ###
+ in IDLE for 5000 ms (0 ms backoff)
+timeout
+ no timeout
+])
diff --git a/tests/test-reconnect.c b/tests/test-reconnect.c
index c84bb1cdb..26d522624 100644
--- a/tests/test-reconnect.c
+++ b/tests/test-reconnect.c
@@ -147,11 +147,26 @@ do_connected(struct ovs_cmdl_context *ctx OVS_UNUSED)
}
static void
-do_activity(struct ovs_cmdl_context *ctx OVS_UNUSED)
+do_set_probe_interval(struct ovs_cmdl_context *ctx)
{
+ reconnect_set_probe_interval(reconnect, atoi(ctx->argv[1]));
+}
+
+static void
+do_activity(struct ovs_cmdl_context *ctx)
+{
+ if (ctx->argc > 1) {
+ reconnect_set_queued_bytes(reconnect, atoi(ctx->argv[1]));
+ }
reconnect_activity(reconnect, now);
}
+static void
+do_queued_bytes(struct ovs_cmdl_context *ctx)
+{
+ reconnect_set_queued_bytes(reconnect, atoi(ctx->argv[1]));
+}
+
static void
do_run(struct ovs_cmdl_context *ctx)
{
@@ -297,7 +312,9 @@ static const struct ovs_cmdl_command all_commands[] = {
{ "connecting", NULL, 0, 0, do_connecting, OVS_RO },
{ "connect-failed", NULL, 0, 1, do_connect_failed, OVS_RO },
{ "connected", NULL, 0, 0, do_connected, OVS_RO },
- { "activity", NULL, 0, 0, do_activity, OVS_RO },
+ { "activity", NULL, 0, 1, do_activity, OVS_RO },
+ { "queued-bytes", NULL, 1, 1, do_queued_bytes, OVS_RO },
+ { "set-probe-interval", NULL, 1, 1, do_set_probe_interval, OVS_RO },
{ "run", NULL, 0, 1, do_run, OVS_RO },
{ "advance", NULL, 1, 1, do_advance, OVS_RO },
{ "timeout", NULL, 0, 0, do_timeout, OVS_RO },
diff --git a/tests/test-reconnect.py b/tests/test-reconnect.py
index cea48eb52..797a50ac2 100644
--- a/tests/test-reconnect.py
+++ b/tests/test-reconnect.py
@@ -61,10 +61,20 @@ def do_connected(_):
r.connected(now)
-def do_activity(_):
+def do_set_probe_interval(arg):
+ r.set_probe_interval(int(arg))
+
+
+def do_activity(arg):
+ if arg is not None:
+ r.set_queued_bytes(int(arg))
r.activity(now)
+def do_queued_bytes(arg):
+ r.set_queued_bytes(int(arg))
+
+
def do_run(arg):
global now
if arg is not None:
@@ -181,6 +191,8 @@ def main():
"connect-failed": do_connect_failed,
"connected": do_connected,
"activity": do_activity,
+ "queued-bytes": do_queued_bytes,
+ "set-probe-interval": do_set_probe_interval,
"run": do_run,
"advance": do_advance,
"timeout": do_timeout,
base-commit: b30f621502752463807af7bb8bcf3a0c763544cf
--
2.43.0
--
_'Esta mensagem é direcionada apenas para os endereços constantes no
cabeçalho inicial. Se você não está listado nos endereços constantes no
cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa
mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão
imediatamente anuladas e proibidas'._
* **'Apesar do Magazine Luiza tomar
todas as precauções razoáveis para assegurar que nenhum vírus esteja
presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por
quaisquer perdas ou danos causados por esse e-mail ou por seus anexos'.*
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
