Dear Admins, Our servers are running openafs 1.6.9, 1.6.10, and 1.6.20. With the hope to get rid of occasional getcwd-issues, we recently upgraded our clients (running CentOS 7) to openafs 1.8.7
About 3 weeks later, we started having occasional issues with afs-clients which suddenly stopped working. (5 times on 5 different centos machines at random times) First we get these kind of messages on the client: Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) Feb 12 08:27:35 kernel: afs: failed to store file (network problems) ... Later we see these messages: (all our afs-servers seem to be unreachable) [Fri Feb 12 08:30:31 2021] afs: Lost contact with file server x.y.4.208 in cell cellname (code -1) (all multi-homed ip addresses down for the server) [Fri Feb 12 08:30:31 2021] afs: Lost contact with file server x.y.4.205 in cell cellname (code -1) (all multi-homed ip addresses down for the server) [Fri Feb 12 08:30:33 2021] afs: Lost contact with file server x.y.4.206 in cell cellname (code -1) (all multi-homed ip addresses down for the server) [Fri Feb 12 08:30:34 2021] afs: Lost contact with file server x.y.4.220 in cell cellname (code -1) (all multi-homed ip addresses down for the server) [Fri Feb 12 08:31:08 2021] afs: Lost contact with file server x.y.4.203 in cell cellname (code -1) (all multi-homed ip addresses down for the server) [Fri Feb 12 08:31:09 2021] afs: Lost contact with file server x.y.4.204 in cell cellname (code -1) (all multi-homed ip addresses down for the server) [Fri Feb 12 08:32:27 2021] afs: Lost contact with file server x.y.4.207 in cell cellname (code -1) (all multi-homed ip addresses down for the server) [Fri Feb 12 08:36:02 2021] afs: Lost contact with file server x.y.4.209 in cell cellname (code -1) (all multi-homed ip addresses down for the server) Although these messages are logged, there seems to be no issue with the IP-connectivity (not noticed by our monitoring, and the moment we check the centos machine, we can ping all afs-servers) There is also no other sign in the logs that network connectivity might have been dropped or was gone at a particular time. Are there any other reasons for these log messages to be logged other then network problems? (eg if the server is still there and available to the clients)? What we have tried to debug/solve the issue so far: Restart the openafs-client but that does not work. (stop doesn't work; kill -9 of the client neither, unload of the afs module wont work) The only thing that has worked to get it working again, is reboot the client machine. No fstrace was done. (so far it seems to happen randomly so we have no clue when it will happen and on which client, not possible to start logging everywhere all the time) We did ran a tcpdump on the client machine when the issue was there. This is the dump: (some "duplicate packet acked" is shown. Does this point to something wrong?) 09:27:58.614227 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 6 reason duplicate packet acked 1 (66) 09:27:58.614245 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.614253 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.614256 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 6 reason duplicate packet acked 1 (66) 09:27:58.614286 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.614310 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 6 reason duplicate packet acked 1 (66) 09:27:58.614311 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.614314 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 6 reason duplicate packet acked 1 (66) 09:27:58.614322 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.614327 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.614341 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.614345 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 7 reason ping response acked 1 (66) 09:27:58.867110 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:27:59.616100 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616109 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616115 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616121 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616126 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616132 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616137 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616143 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616148 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616154 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616159 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616165 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616170 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616176 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616181 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616187 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616193 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616198 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616204 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616212 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616217 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616223 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616228 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616234 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616240 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616246 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616252 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616258 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616264 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616270 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616275 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616282 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616288 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616294 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:27:59.616301 IP a.b.34.156.afs3-callback > x.y.4.209.afs3-fileserver: rx version (29) 09:27:59.616307 IP a.b.34.156.afs3-callback > x.y.4.209.afs3-fileserver: rx version (29) 09:27:59.616313 IP a.b.34.156.afs3-callback > x.y.4.209.afs3-fileserver: rx version (29) 09:27:59.616319 IP a.b.34.156.afs3-callback > x.y.4.209.afs3-fileserver: rx version (29) 09:27:59.616325 IP a.b.34.156.afs3-callback > x.y.4.209.afs3-fileserver: rx version (29) 09:27:59.616331 IP a.b.34.156.afs3-callback > x.y.4.209.afs3-fileserver: rx version (29) 09:27:59.836618 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:27:59.934697 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:27:59.981095 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:00.094359 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:00.118057 IP a.b.34.156.afs3-callback > x.y.4.207.afs3-fileserver: rx version (29) 09:28:00.127549 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:00.128733 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:00.526543 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:00.867186 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:01.836673 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:01.934682 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:01.981197 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:02.094436 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:02.135810 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:02.137192 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:02.526616 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:02.541248 IP a.b.34.156 > x.y.4.204: ICMP echo request, id 6917, seq 1, length 64 09:28:02.541360 IP x.y.4.204 > a.b.34.156: ICMP echo reply, id 6917, seq 1, length 64 09:28:02.867266 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:03.541057 IP a.b.34.156 > x.y.4.204: ICMP echo request, id 6917, seq 2, length 64 09:28:03.541158 IP x.y.4.204 > a.b.34.156: ICMP echo reply, id 6917, seq 2, length 64 09:28:03.836743 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:03.934704 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:03.980029 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:04.094501 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:04.126055 IP a.b.34.156.afs3-callback > x.y.4.203.afs3-fileserver: rx version (29) 09:28:04.127780 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:04.128917 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:04.526702 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:04.627061 IP a.b.34.156.afs3-callback > x.y.4.209.afs3-fileserver: rx version (29) 09:28:04.867323 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:05.836802 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:05.934741 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:05.979001 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:06.094564 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:06.127917 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:06.129002 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:06.526780 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:06.867401 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:07.132067 IP a.b.34.156.afs3-callback > x.y.4.220.afs3-fileserver: rx version (29) 09:28:07.836875 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:07.934751 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:07.978448 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:08.094637 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:08.127991 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:08.129088 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:08.526873 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:08.867514 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:09.845475 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:09.934823 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:09.978445 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:10.094705 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:10.128139 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:10.129202 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:10.138066 IP a.b.34.156.afs3-callback > x.y.4.206.afs3-fileserver: rx version (29) 09:28:10.526950 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:10.867455 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:11.837020 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:11.934798 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:11.978432 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:12.094769 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:12.128193 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:12.129295 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:12.527037 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:12.867503 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:13.837062 IP x.y.4.206.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:13.934853 IP x.y.4.203.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:13.978486 IP x.y.4.207.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:14.094835 IP x.y.4.205.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:14.136372 IP x.y.4.208.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:14.137484 IP x.y.4.204.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:14.526753 IP x.y.4.209.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) 09:28:14.867718 IP x.y.4.220.afs3-fileserver > a.b.34.156.afs3-callback: rx ack first 1 serial 0 reason ping acked 1 (66) Are there any other known issues in openafs client 1.8.7 which could cause this? Did anybody see this same problem since using openafs 1.8.7? Does someone have tips on how to further debug/troubleshoot this issue? Greatly appreciated !
