Public bug reported:

[ Impact ]

Linux 6.17+ changed the way that the d_entry operations are set for a super 
block in 05fb0e666495c.
In particular, this means openafs silently fails to register its own d_entry 
operations, leading
to a stale d_entry cache in many situations. 

[ Test Plan ]

The cache revalidation issues occur when changes are made on a remote
client, so the test setup needs two machines: an openafs server which
will make all the modifications, and an openafs client which will only
observe the changes.

In order to accurately reproduce the issues it is important to only run
commands on the client when the testplan says as any read to the network
filesystem can populate the cache.

Launch two resolute VMs:

```bash
lxc launch ubuntu:resolute afs-server --vm -c limits.memory=4GiB -c 
limits.cpu=4 \
    -c boot.mode=uefi-nosecureboot
lxc launch ubuntu:resolute afs-client --vm -c limits.memory=2GiB -c 
limits.cpu=4 \
    -c boot.mode=uefi-nosecureboot
```

Secure Boot must be off: the openafs module is built by DKMS and signed
with a key that is not enrolled.

NOTE: Run every command below as root.

On the server, configure the cell and install the server packages. This
sets up the server without needing a KDC and creates a cell at
/afs/localcell:

```bash
debconf-set-selections <<EOF
openafs-client openafs-client/thiscell string localcell
openafs-client openafs-client/cachesize string 100000
openafs-client openafs-client/dynroot select Yes
openafs-client openafs-client/fakestat boolean true
openafs-fileserver openafs-fileserver/thiscell string localcell
EOF
apt update
apt install linux-headers-$(uname -r) openafs-modules-dkms \
    openafs-client openafs-fileserver openafs-dbserver openafs-krb5 krb5-user
```

Create the random cell key and the realm to match:

```bash
printf '[libdefaults]\n default_realm = LOCALCELL\n dns_lookup_realm = false\n 
dns_lookup_kdc = false\n' > /etc/krb5.conf
printf 'addent -password -p afs/localcell -k 1 -e aes256-cts\n%s\nwkt 
/etc/openafs/server/rxkad.keytab\nquit\n' \
    "$(head -c 24 /dev/urandom | base64)" | ktutil
akeyconvert
```

Point the server at its own cell:

```bash
for db in /etc/openafs/CellServDB /etc/openafs/server/CellServDB; do
    printf '>localcell\n%s  # afs-server\n' "$(hostname -I | awk '{print $1}')" 
> $db
done
```

Give the file server a partition and start the servers:

```bash
mkdir -p /vicepa && touch /vicepa/AlwaysAttach
systemctl start openafs-fileserver
until bos status localhost -localauth; do sleep 2; done
bos create localhost ptserver simple /usr/lib/openafs/ptserver -localauth
bos create localhost vlserver simple /usr/lib/openafs/vlserver -localauth
bos create localhost dafs dafs -cmd /usr/lib/openafs/dafileserver -cmd 
/usr/lib/openafs/davolserver \
    -cmd /usr/lib/openafs/salvageserver -cmd /usr/lib/openafs/dasalvager 
-localauth
until vos partinfo localhost a -localauth; do sleep 5; done   # about 30 s
```

Create the volume and the admin user:

```bash
vos create localhost a root.cell -localauth
pts createuser admin -localauth
pts adduser admin system:administrators -localauth
```

Restart the client and get an admin token:

```bash
systemctl restart openafs-client
aklog -c localcell -keytab /etc/openafs/server/rxkad.keytab -principal admin
```

Let any user read the volume and create the test directory:

```bash
fs setacl -dir /afs/localcell -acl system:anyuser rl
mkdir /afs/localcell/sru
```

NOTE: Run all later afs-server commands in this root shell, so that they
use the admin token. If the token expires, run the aklog command again.

On the client, configure the cell and install the client packages:

```bash
echo "openafs-client openafs-client/thiscell string localcell" | 
debconf-set-selections
apt update
apt install linux-headers-$(uname -r) openafs-modules-dkms openafs-client
dkms status openafs
```

Point the client at afs-server and restart:

```bash
echo ">localcell" > /etc/openafs/CellServDB
echo "$(getent ahostsv4 afs-server.lxd | awk 'NR==1{print $1}')  # afs-server" 
>> /etc/openafs/CellServDB
echo localcell > /etc/openafs/ThisCell
systemctl restart openafs-client
```

With both VMs set up we can demonstrate the stale dentry cache in a few
ways.

On the server, create a file:

```bash
echo hello > /afs/localcell/sru/repro
```

On the client, read it once to cache it:

```bash
cat /afs/localcell/sru/repro
```

Expected: `hello`. The patch should not change this.

On the server, delete and recreate the file:

```bash
rm -f /afs/localcell/sru/repro && echo newdata-1 > /afs/localcell/sru/repro
```

On the client, read it once:

```bash
cat /afs/localcell/sru/repro
```

Expected Output:

```text
newdata-1
```

Unpatched output:

```text
hello
```

On the client, flush the AFS cache only for that name, then read it:

```bash
fs flush -path /afs/localcell/sru/repro
cat /afs/localcell/sru/repro
```

Expected Output:

```text
newdata-1
```

Unpatched output:

```text
cat: /afs/localcell/sru/repro: Input/output error
```

On the client, flush both caches:

```bash
fs flush -path /afs/localcell/sru/repro >/dev/null 2>&1
fs flush -path /afs/localcell/sru
sync
echo 2 > /proc/sys/vm/drop_caches
cat /afs/localcell/sru/repro
```

Expected, with and without the patch:

```text
newdata-1
```

On the server, delete the file:

```bash
rm -f /afs/localcell/sru/repro
```

On the client, read it once:

```bash
cat /afs/localcell/sru/repro
```

Expected Output:

```text
cat: /afs/localcell/sru/repro: No such file or directory
```

Unpatched output:

```text
newdata-1
```

On the client, flush both caches:

```bash
fs flush -path /afs/localcell/sru/repro >/dev/null 2>&1
fs flush -path /afs/localcell/sru
sync
echo 2 > /proc/sys/vm/drop_caches
cat /afs/localcell/sru/repro
```

Expected, with and without the patch:

```text
cat: /afs/localcell/sru/repro: No such file or directory
```

On the server, recreate the file:

```bash
echo hello > /afs/localcell/sru/repro
```

On the client, read it once:

```bash
cat /afs/localcell/sru/repro
```

Expected Output:

```text
hello
```

Unpatched output:

```text
cat: /afs/localcell/sru/repro: No such file or directory
```

To run the test again, use a new directory instead of
/afs/localcell/sru, so that no cached entries from an earlier run can
change the result.

[ Where problems could occur ]

* This will trigger a rebuild of the openafs DKMS module. Which could put an 
unexepected load on systems using openafs when they upgrade.
* We're changing the dentry operations so, we could introduce other staleness 
issues, false cache misses. In principle, the patch should simply restore the 
pre 6.17 behavior, but it's possible that there could be some hidden kernel 
regression behind the move from s_d_op to set_default_d_op. Nonetheless, this 
would be a transition from a currently broken state to a different broken state.

[ Other Info ]

 * A patch for the issue landed upstream in 1.18.16
<https://gerrit.openafs.org/16700>

** Affects: openafs (Ubuntu)
     Importance: High
     Assignee: Zachary Raines (raineszm)
         Status: New

** Affects: openafs (Ubuntu Resolute)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2173580

Title:
  [SRU] dentry cache becomes stale on kernel >= 6.17

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openafs/+bug/2173580/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to