lazedo opened a new issue, #6215:
URL: https://github.com/apache/couchdb/issues/6215
### Version
main (0a666d0fe). Not in any release: serialized worker startup is only on
main.
### Describe the problem you're encountering
With `[fabric] serialize_worker_startup = true`, the default on main, a
client that reads a document and then updates it with the `_rev` it just read
can get `409 Conflict` over and over. Nothing else writes the document and it
has no conflicts. The 409s stop only when the range owner's copy catches up.
What happens, from reading `fabric`:
1. An attachment PUT with a streamed body is written to all copies in
parallel (`any_streamed_atts/1` turns serialization off), and the coordinator
answers 201 once `w` copies are done. The range owner may be the copy that is
still behind.
2. The `GET` (r=2 of n=3) is answered by the first two copies that agree.
When they agree there is no read repair, so the client gets the newest revision.
3. The interactive `PUT` that follows goes first, and only, to the owner.
The owner does not have that revision yet and returns `conflict`. Since no copy
has answered `ok`, `update_quorum_met/3` treats the conflict as final, and the
coordinator answers 409 without asking the other copies, which would accept the
update.
4. Retrying gets the same revision and the same 409 until the owner catches
up.
### Expected Behaviour
A client that reads the latest revision through the read quorum can update
the document with it, without waiting for the owner's copy to catch up.
### Steps to Reproduce
1. On main (0a666d0fe): `./configure --dev --js-engine=quickjs && make &&
dev/run -n 3 -a adm:pass`.
2. With 3 dev nodes, a database named `sws_probe_4` with q=1 has owner order
node1, node2, node3. Create it through node2:
`curl -u adm:pass -X PUT 'localhost:25984/sws_probe_4?q=1&n=3'`
3. Make node1 slower than the others: `sh slow.sh node1 3600 0.15 0.05`
freezes its beam for 150 ms out of every 200 ms, without using any CPU.
4. Run `COUCHDB_USER=adm COUCHDB_PASSWORD=pass sh probe.sh sws_probe_4 30
1048576 3 localhost:25984`, then `touch /tmp/slow.stop` to stop the slowdown.
Both scripts expect Linux: they use `/proc/uptime` and `pgrep`.
- Each cycle writes three 1 MiB attachments, GETs the document and PUTs
it back with the `_rev` it got. On a 409 it retries GET + PUT, up to 100 times.
- With two more arguments,
`shards%2F00000000-ffffffff%2Fsws_probe_4<suffix>` and `'[email protected]
[email protected] [email protected]'`, the first 409 of a cycle also reads the
document on each node's own copy (`/_node/<node>/<shard>/probe`). The suffix is
the `shard_suffix` of `/_node/_local/_dbs/sws_probe_4`.
Results, 30 cycles per run, all through node2:
| owner | node1 frozen | `serialize_worker_startup` on node2 | cycles with
409 | 409 responses |
|---|---|---|---|---|
| node1 | 150 ms of every 200 ms | true (default) | 30/30 | 2910 |
| node3 | 150 ms of every 200 ms | true (default) | 0/30 | 0 |
| node1 | 150 ms of every 200 ms | false | 0/30 | 0 |
| node1 | 150 ms of every 200 ms | true (default) | 30/30 | 2901 |
Other runs with node1 as the owner (20 cycles each):
- with node1 frozen 50 ms of every 100 ms: 1 cycle with 409s in each of two
runs (3 and 13 in a row, over 0.75 s and 1.6 s);
- without slowing node1 down: no 409s.
In every run, at each first 409, the GET had returned the revision created
by the last attachment PUT. node2 and node3 had that revision, and node1, the
owner, was 1 to 9 revisions behind. For example:
```
cycle 1 att=201 0.332683s first_put=409 get_rev=att_rev final=201 tries=40
window=9.42s at_409: get=324-d81cc8 node1=321-ce7089 node2=324-d81cc8
node3=324-d81cc8
```
<details><summary>probe.sh</summary>
```sh
#!/bin/sh
# probe.sh DB CYCLES BYTES [ATTS_PER_CYCLE] [URL] [SHARD] [NODES]
# Run inside a couch container (credentials from its env, never printed).
# Each cycle: ATTS_PER_CYCLE attachment PUTs of BYTES random bytes with the
# current rev, then GET the doc and PUT it back (n set to the cycle) with the
# rev that GET returned; on 409 GET+PUT again right away (up to 100 tries).
# With SHARD (URL-encoded shard name) and NODES (space separated), the first
# 409 of a cycle also reads the doc's rev on each node's own copy.
# One line per cycle, then the totals.
set -u
DB=$1
CYCLES=$2
BYTES=$3
K=${4:-1}
U=${5:-localhost:5984}
SHARD=${6:-}
NODES=${7:-}
AUTH="$COUCHDB_USER:$COUCHDB_PASSWORD"
c() { curl -s -u "$AUTH" "$@"; }
rev_of() { sed -n 's/.*"_rev":"\([^"]*\)".*/\1/p;
s/.*"rev":"\([^"]*\)".*/\1/p' | head -1; }
now() { cut -d' ' -f1 /proc/uptime; }
short() { printf '%s' "$1" | cut -c1-10; }
copies() {
for n in $NODES; do
case "$n" in
couchdb@*) label=${n#couchdb@}; label=${label%%.*} ;;
*) label=${n%%@*} ;;
esac
printf ' %s=%s' "$label" "$(short "$(c "$U/_node/$n/$SHARD/probe" |
rev_of)")"
done
}
out=$(c -X PUT "$U/$DB/probe" -H 'Content-Type: application/json' -d
'{"n":0}')
REV=$(printf '%s' "$out" | rev_of)
[ -n "$REV" ] || REV=$(c "$U/$DB/probe" | rev_of)
[ -n "$REV" ] || { echo "no probe doc: $out"; exit 1; }
total409=0
cyc409=0
maxtries=0
i=1
while [ "$i" -le "$CYCLES" ]; do
j=1
while [ "$j" -le "$K" ]; do
out=$(head -c "$BYTES" /dev/urandom | c -X PUT
"$U/$DB/probe/f$j.bin?rev=$REV" \
-H 'Content-Type: application/octet-stream' --data-binary @- -w
' %{http_code} %{time_total}')
atime=${out##* }
rest=${out% *}
acode=${rest##* }
REV=$(printf '%s' "$rest" | rev_of)
j=$((j + 1))
done
attrev=$REV
tries=0
code=409
first=""
seen=""
t0=$(now)
while [ "$code" = 409 ] && [ "$tries" -lt 100 ]; do
doc=$(c "$U/$DB/probe")
grev=$(printf '%s' "$doc" | rev_of)
body=$(printf '%s' "$doc" | sed "s/\"n\":[0-9]*/\"n\":$i/")
out=$(c -X PUT "$U/$DB/probe" -H 'Content-Type: application/json'
--data-binary "$body" -w ' %{http_code}')
code=${out##* }
tries=$((tries + 1))
[ -n "$first" ] || first="$code get_rev=$([ "$grev" = "$attrev" ] &&
echo att_rev || short "$grev")"
if [ "$code" = 409 ]; then
total409=$((total409 + 1))
[ -n "$seen" ] || [ -z "$SHARD" ] || seen=" at_409: get=$(short
"$grev")$(copies)"
fi
done
t1=$(now)
[ "$tries" -gt 1 ] && cyc409=$((cyc409 + 1))
[ "$tries" -gt "$maxtries" ] && maxtries=$tries
REV=$(printf '%s' "${out% *}" | rev_of)
[ -n "$REV" ] || REV=$(c "$U/$DB/probe" | rev_of)
echo "cycle $i att=$acode ${atime}s first_put=$first final=$code
tries=$tries window=$(awk "BEGIN{printf \"%.2f\", $t1 - $t0}")s$seen"
i=$((i + 1))
done
echo "TOTAL db=$DB cycles=$CYCLES bytes=$BYTES atts_per_cycle=$K
put_409s=$total409 cycles_with_409=$cyc409 max_tries=$maxtries"
```
</details>
<details><summary>slow.sh</summary>
```sh
#!/bin/sh
# slow.sh NODE SECONDS [STOP_S] [RUN_S]
# Make a dev/run node slow without burning CPU: freeze its beam.smp for
STOP_S
# out of every STOP_S + RUN_S seconds (default 0.05 / 0.05), for SECONDS or
# until /tmp/slow.stop exists. Always leaves the process running (SIGCONT).
set -u
NODE=$1
SECS=$2
STOP=${3:-0.05}
RUN=${4:-0.05}
pid=$(pgrep -f "beam.smp.* -name $NODE@" | head -1)
[ -n "$pid" ] || { echo "no beam for $NODE"; exit 1; }
trap 'kill -CONT "$pid" 2>/dev/null' EXIT INT TERM
rm -f /tmp/slow.stop
end=$(( $(cut -d. -f1 /proc/uptime) + SECS ))
n=0
while [ ! -e /tmp/slow.stop ] && [ "$(cut -d. -f1 /proc/uptime)" -lt "$end"
]; do
kill -STOP "$pid"
sleep "$STOP"
kill -CONT "$pid"
sleep "$RUN"
n=$((n + 1))
done
echo "slow $NODE pid $pid: $n freezes of ${STOP}s"
```
</details>
### Your Environment
- dev/run with 3 nodes, main at 0a666d0fe, Erlang/OTP 28, Alpine in Docker.
- We first hit this without any artificial slowdown, on a 3-node cluster
where one copy was slower than the other two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
