On Tue, Nov 13, 2018 at 08:46:28PM -0500, Theo Ouzhinski wrote: > Hi all, > > Sorry for my previous incorrectly formatted email. > Recently, I've seen an uptick in "no space left on device" errors for > some of the home directories I administer. > > For example, > > matsumoto <USERNAME> # touch a > touch: cannot touch 'a': No space left on device > > We are not even close to filling up the cache (located at > /var/cache/openafs) on this client machine. > > matsumoto ~ # fs getcacheparms > AFS using 10314 of the cache's available 10000000 1K byte blocks. > matsumoto ~ # df -h > Filesystem Size Used Avail Use% Mounted on > .... > /dev/mapper/vgwrkstn-root 456G 17G 417G 4% / > .... > AFS 2.0T 0 2.0T 0% /afs > > > Nor is this home directory or any other problematic home directory close > to their quota. > > matsumoto <USERNAME> # fs lq > Volume Name Quota Used %Used Partition > <VOLUME NAME> 4194304 194403 5% 37% > > According to previous posts on this list, many issues can be attributed > to high inode usage. However, this is not the case on our machines. > > Here is sample output from one of our OpenAFS servers, which is similar > to all of the four other ones. > > openafs1 ~ # df -i > Filesystem Inodes IUsed IFree IUse% Mounted on > udev 1903816 413 1903403 1% /dev > tmpfs 1911210 551 1910659 1% /run > /dev/vda1 1905008 154821 1750187 9% / > tmpfs 1911210 1 1911209 1% /dev/shm > tmpfs 1911210 5 1911205 1% /run/lock > tmpfs 1911210 17 1911193 1% /sys/fs/cgroup > /dev/vdb 19660800 3461203 16199597 18% /vicepa > /dev/vdc 19660800 1505958 18154842 8% /vicepb > tmpfs 1911210 4 1911206 1% /run/user/0 > AFS 2147483647 0 2147483647 0% /afs > > > We are running the latest HWE kernel (4.15.0-38-generic) for Ubuntu > 16.04 (which is the OS for both server and client machines). We are > running on the clients, the following versions: > > openafs-client/xenial,now 1.8.2-0ppa2~ubuntu16.04.1 amd64 [installed] > openafs-krb5/xenial,now 1.8.2-0ppa2~ubuntu16.04.1 amd64 [installed] > openafs-modules-dkms/xenial,xenial,now 1.8.2-0ppa2~ubuntu16.04.1 all > [installed] > > and on the servers, the following versions: > > openafs-client/xenial,now 1.6.15-1ubuntu1 amd64 [installed] > openafs-dbserver/xenial,now 1.6.15-1ubuntu1 amd64 [installed] > openafs-fileserver/xenial,now 1.6.15-1ubuntu1 amd64 [installed] > openafs-krb5/xenial,now 1.6.15-1ubuntu1 amd64 [installed] > openafs-modules-dkms/xenial,xenial,now 1.6.15-1ubuntu1 all [installed]
(Off-topic, but that looks to be missing some security fixes.) > What could be the problem? Is there something I missed? It's not really ringing a bell off the top of my head, no. That said, there's a number of potential ways to get ENOSPC, so it would be good to get more data, like an strace of the failing touch, and maybe a packet capture (port 7000) during the touch, both from a clean cache and potentially a second attempt. -Ben _______________________________________________ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
