Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
> On Aug 28, 2025, at 4:30 PM, Cheyenne Wills wrote: > > The patches for OpenAFS have been submitted for the master branch and > are currently under review. They will be included in an upcomming > 1.8.14pre1, that is still being finalized. > > Again thank's to Jeffery and Marc for their investigative work. S/Jeffery/Jeffrey/ > I do want to mention that the commit that introduced the bug wasn't yet > in a tagged stable release (though it was in the pending stack for the > upcoming 1.8.14 work). OpenAFS 0306f3fdac736e15620f5802bdce510d25bb2450 was included in packaged and tagged “openafs" releases from both OpenSUSE, Debian, Ubuntu and Fedora. Although it is true that the OpenAFS release team has not tagged a release within the openafs git repository that it manages, Debian, Ubuntu, OpenSUSE and Fedora took 0306f3fdac73 as part of the stack of changes necessary to support 6.14 kernels based upon advice received from openafs developers: https://lists.openafs.org/pipermail/openafs-devel/2025-April/021060.html In my opinion a CVE should be published by OpenAFS referencing the commit which can be referenced by all of the downstream distributions which included it. Jeffrey Altman ___ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
> On Aug 26, 2025, at 5:11 PM, Cheyenne Wills wrote: > > Thank you Jeff and Marc for identifying the area that was causing a > problem. > > The cause is a little more subtle than described. I disagree. > The Linux 6.14 commit (5be1fa8abd7b049f51e6e98e75a37eef5ae2c296) that > was responsible for the OpenAFS commit > (0306f3fdac736e15620f5802bdce510d25bb2450) added a new parameters for > the dop.d_revalidate() function. The new parameter, 'name', is, as > mentioned a qstr, but the string that it points to is NUL > terminated as well (according to Linux's porting documentation). The qstr.name is NUL terminated to prevent broken code from running off the end of the allocated memory. > However the length of the qstr does match the "strlen" of the entire > string -- which is the root cause of the problem. The qstr.len might be the same length as strlen(qstr.name) but it is not in all cases and in particular it is not in the case which is leaking full paths via DNS queries. > Prior to the 0306f3fdac73 commit, OpenAFS's d_validate routine was using > the d_name.name from the dentry parameter. Using the d_name.name from > the dentry parameter fixes the behavior (basically it reverts part of > the 0306f3fdac73 change). I'm working on a patch for OpenAFS to > correct the problem. Using the d_name.name might prevent the leakage of full paths but doing so is wrong because the d_name.name field is not stable. You cannot even be sure that it has the right value when the openafs module is entered. The qstr is the value that the openafs module is being asked to revalidate. Jeffrey Altman ___ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
The patches for OpenAFS have been submitted for the master branch and are currently under review. They will be included in an upcomming 1.8.14pre1, that is still being finalized. Again thank's to Jeffery and Marc for their investigative work. I do want to mention that the commit that introduced the bug wasn't yet in a tagged stable release (though it was in the pending stack for the upcoming 1.8.14 work). -- Cheyenne Wills [email protected] On Tue, 26 Aug 2025 12:01:16 -0500 "Ryan C. Underwood" wrote: > On Tue, Aug 26, 2025 at 10:59:32AM -0400, Jeffrey Altman wrote: > [...] > [...] > > Thanks Jeffrey, I am cc'ing the following people to raise awareness of > this critical bug > > Christof Hanke who appears to have > pulled this commit into the OpenSUSE build for kernel 6.14 > compatibility > > Jacob Martin who did the same for Ubuntu: > https://lists.ubuntu.com/archives/plucky-changes/2025-March/014784.html > > I filed the following Ubuntu bug report: > https://bugs.launchpad.net/ubuntu/+source/openafs/+bug/2121466 > > -- > Ryan C. Underwood, ___ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
Thank you Jeff and Marc for identifying the area that was causing a problem. The cause is a little more subtle than described. The Linux 6.14 commit (5be1fa8abd7b049f51e6e98e75a37eef5ae2c296) that was responsible for the OpenAFS commit (0306f3fdac736e15620f5802bdce510d25bb2450) added a new parameters for the dop.d_revalidate() function. The new parameter, 'name', is, as mentioned a qstr, but the string that it points to is NUL terminated as well (according to Linux's porting documentation). However the length of the qstr does match the "strlen" of the entire string -- which is the root cause of the problem. Prior to the 0306f3fdac73 commit, OpenAFS's d_validate routine was using the d_name.name from the dentry parameter. Using the d_name.name from the dentry parameter fixes the behavior (basically it reverts part of the 0306f3fdac73 change). I'm working on a patch for OpenAFS to correct the problem. Again thank you Jeff and Marc for your analysis. I was looking through the 1.8.10 -> 1.8.13.2 differences to try to determine the cause of the problem. -- Cheyenne Wills [email protected] On Tue, 26 Aug 2025 10:59:32 -0400 Jeffrey Altman wrote: > [...] > [...] > [...] > [...] > […] > > [...] > > Ryan observes the behavior on 25.04 and you observe it on 24.04. > Both running 6.14.0 based kernels. > > The bug was introduced to OpenAFS in commit > 0306f3fdac736e15620f5802bdce510d25bb2450 which must have been > cherry-picked into the Ubuntu builds. > > struct qstr “quick string” is a counted string not a NUL terminated > string. The mistake introduced in the aforementioned commit results > in the full path being evaluated as the dentry name instead of just > the path component the dentry actually refers to. > > As a result, the d_revalidate() call will always fail, the current > dentry will always be invalidated, and a full lookup() will be > performed each time a cached dentry is used. > > Marc Dionne should be credited with identifying the mistake. > > This mistake is not present in the AuriStorFS client. > > Jeffrey Altman > ___ > OpenAFS-info mailing list > [email protected] > https://lists.openafs.org/mailman/listinfo/openafs-info ___ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
On Tue, Aug 26, 2025 at 10:59:32AM -0400, Jeffrey Altman wrote: > > The process was running on Ubuntu 24.04, kernel 6.14.0-28-generic, OpenAFS > > Client 1.8.13.2-1ubuntu1. > > Ryan observes the behavior on 25.04 and you observe it on 24.04. Both > running 6.14.0 based kernels. > > The bug was introduced to OpenAFS in commit > 0306f3fdac736e15620f5802bdce510d25bb2450 which must have been cherry-picked > into the Ubuntu builds. > > struct qstr “quick string” is a counted string not a NUL terminated string. > The mistake introduced in the aforementioned commit results in the full path > being evaluated as the dentry name instead of just the path component the > dentry actually refers to. > > As a result, the d_revalidate() call will always fail, the current dentry > will always be invalidated, and a full lookup() will be performed each time a > cached dentry is used. > > Marc Dionne should be credited with identifying the mistake. > > This mistake is not present in the AuriStorFS client. Thanks Jeffrey, I am cc'ing the following people to raise awareness of this critical bug Christof Hanke who appears to have pulled this commit into the OpenSUSE build for kernel 6.14 compatibility Jacob Martin who did the same for Ubuntu: https://lists.ubuntu.com/archives/plucky-changes/2025-March/014784.html I filed the following Ubuntu bug report: https://bugs.launchpad.net/ubuntu/+source/openafs/+bug/2121466 -- Ryan C. Underwood, signature.asc Description: PGP signature
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
> On Aug 26, 2025, at 9:27 AM, Gunnar Krull > wrote: > > Hi Jeffrey, > > On 25/08/2025 17:19, Jeffrey Altman wrote: >>> 25-Aug-2025 07:15:47.756 client @0x7fa6b79fe168 172.27.2.4#34129 >>> (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de): >>> query: >>> _afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de >>> IN SRV +E(0) (134.76.81.212) >> Likewise a lookup is being performed for >> “informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator” >> as a single path component. >> An strace of the originating process would be useful to examine. > > here is one example of a strace that can be matched to the corresponding DNS > query. > > * strace of the process: > > 11442 13:56:30.961461 faccessat2(AT_FDCWD, > "/afs/informatik.uni-goettingen.de/user/m/x.x/.ifi-login/authorized_keys", > R_OK, AT_EACCESS) = -1 ENOENT (No such file or directory) <0.017024> > > * corresonding generated DNS queries: > > 26-Aug-2025 13:56:30.989 client @0x7fa6b799e168 172.27.2.4#39324 > (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/./.ifi-login/authorized_keys): > query: > _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/./.ifi-login/authorized_keys > IN SRV +E(0) (134.76.81.212) > […] > The process was running on Ubuntu 24.04, kernel 6.14.0-28-generic, OpenAFS > Client 1.8.13.2-1ubuntu1. Ryan observes the behavior on 25.04 and you observe it on 24.04. Both running 6.14.0 based kernels. The bug was introduced to OpenAFS in commit 0306f3fdac736e15620f5802bdce510d25bb2450 which must have been cherry-picked into the Ubuntu builds. struct qstr “quick string” is a counted string not a NUL terminated string. The mistake introduced in the aforementioned commit results in the full path being evaluated as the dentry name instead of just the path component the dentry actually refers to. As a result, the d_revalidate() call will always fail, the current dentry will always be invalidated, and a full lookup() will be performed each time a cached dentry is used. Marc Dionne should be credited with identifying the mistake. This mistake is not present in the AuriStorFS client. Jeffrey Altman ___ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
Reply inline > On Aug 26, 2025, at 9:04 AM, Ryan C. Underwood > wrote: > > > Hi Jeffrey: > > On Fri, Aug 22, 2025 at 01:08:25PM -0400, Jeffrey Altman wrote: >> >> What is the version of Ubuntu and most importantly, what is the kernel >> version? > > Ubuntu 25.04 with kernel 6.14 > >> Unless the afsd dynamic root functionality is disabled, DNS SRV or DNS AFSDB >> queries would be expected if the cache manager’s CellServDB does not contain >> a list of database servers for the cell “ice quake.net”. In particular we >> would expect to see a DNS SRV query for :_afs3-vlserver._udp.icequake.net >> and if that query failed, then a fallback query for DNS AFSDB >> “icequake.net”. Those queries are not present in the above trace. As far >> as I can tell there are neither DNS SRV nor DNS AFSDB records published for >> ice quake.net. > > Those records are there and haven't been changed in a very long time: > > […] > How did you test that led to the conclusion that there are no such records? Indeed they are and I can see them now. I tested two ways. First, by use of “vos eachvl -cell icequake.net”. Second, by entering the queries into nslookup. It’s possible I mistyped the domain name or that the hotel network I was on blocked the SRV and AFSDB queries. > >> What else changed between when 1.8.10 was installed and 1.8.13.2? > > Just the recent ubuntu distribution upgrade and regular package > upgrades. It's hard to contain all the variables on this particular > system since it's a daily driver. Understood. > >> I am curious if you experience the same behavior problems with the >> AuriStorFS client. If you do, that is a strong indication that the problem >> is external to OpenAFS. > > Never used it but happy to take a look. I will send details later today. Jeffrey Altman ___ OpenAFS-info mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
Hi Jeffrey, On 25/08/2025 17:19, Jeffrey Altman wrote: 25-Aug-2025 07:15:47.756 client @0x7fa6b79fe168 172.27.2.4#34129 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) Likewise a lookup is being performed for “informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator” as a single path component. An strace of the originating process would be useful to examine. here is one example of a strace that can be matched to the corresponding DNS query. * strace of the process: 11442 13:56:30.961461 faccessat2(AT_FDCWD, "/afs/informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys", R_OK, AT_EACCESS) = -1 ENOENT (No such file or directory) <0.017024> * corresonding generated DNS queries: 26-Aug-2025 13:56:30.989 client @0x7fa6b799e168 172.27.2.4#39324 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:30.993 client @0x7fa6b7d28168 172.27.2.4#32940 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:30.993 client @0x7fa6b799e168 172.27.2.4#45932 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:30.997 client @0x7fa6b799e168 172.27.2.4#43894 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:30.997 client @0x7fa6b799e168 172.27.2.4#37887 (informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys): query: informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys IN AFSDB +E(0) (134.76.81.212) 26-Aug-2025 13:56:30.997 client @0x7fa6b7d28168 172.27.2.4#59276 (informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de): query: informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de IN AFSDB +E(0) (134.76.81.212) 26-Aug-2025 13:56:31.001 client @0x7fa6b7d28168 172.27.2.4#35919 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:31.001 client @0x7fa6b799e168 172.27.2.4#36703 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:31.001 client @0x7fa6b7d28168 172.27.2.4#38808 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:31.005 client @0x7fa6b799e168 172.27.2.4#50967 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 26-Aug-2025 13:56:31.005 client @0x7fa6b7d28168 172.27.2.4#57572 (informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys): query: informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys IN AFSDB +E(0) (134.76.81.212) 26-Aug-2025 13:56:31.005 client @0x7fa6b7d28168 172.27.2.4#39195 (informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de): query: informatik.uni-goettingen.de/user/m/m.musterfrau/.ifi-login/authorized_keys.informatik.uni-goettingen.de IN AFSDB +E(0) (134.76.81.212) The same process generates many more DNS queries. These queries were triggered by some or all of these calls (copied from the strace log): 11442 13:56:30.762434 faccessat2(AT_FDCWD, "/afs/informatik.uni-goettinge
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
Hi Jeffrey: On Fri, Aug 22, 2025 at 01:08:25PM -0400, Jeffrey Altman wrote: > > What is the version of Ubuntu and most importantly, what is the kernel > version? Ubuntu 25.04 with kernel 6.14 > Unless the afsd dynamic root functionality is disabled, DNS SRV or DNS AFSDB > queries would be expected if the cache manager’s CellServDB does not contain > a list of database servers for the cell “ice quake.net”. In particular we > would expect to see a DNS SRV query for :_afs3-vlserver._udp.icequake.net > and if that query failed, then a fallback query for DNS AFSDB “icequake.net”. > Those queries are not present in the above trace. As far as I can tell > there are neither DNS SRV nor DNS AFSDB records published for ice quake.net. Those records are there and haven't been changed in a very long time: icequake.netIN SOA ns1.icequake.net. hostmaster.icequake.net. ( [..] IN AFSDB1 afs.icequake.net. IN AFSDB1 afs2.icequake.net. IN AFSDB1 afs3.icequake.net. IN AFSDB1 afs4.icequake.net. [..] $ORIGIN _udp.icequake.net. _afs3-vlserver IN SRV 10 10 7003 afs.icequake.net. _afs3-vlserver IN SRV 10 10 7003 afs2.icequake.net. _afs3-vlserver IN SRV 10 10 7003 afs3.icequake.net. _afs3-vlserver IN SRV 10 10 7003 afs4.icequake.net. _afs3-prserver IN SRV 10 10 7002 afs.icequake.net. _afs3-prserver IN SRV 10 10 7002 afs2.icequake.net. _afs3-prserver IN SRV 10 10 7002 afs3.icequake.net. _afs3-prserver IN SRV 10 10 7002 afs4.icequake.net. How did you test that led to the conclusion that there are no such records? > What else changed between when 1.8.10 was installed and 1.8.13.2? Just the recent ubuntu distribution upgrade and regular package upgrades. It's hard to contain all the variables on this particular system since it's a daily driver. > I am curious if you experience the same behavior problems with the AuriStorFS > client. If you do, that is a strong indication that the problem is external > to OpenAFS. Never used it but happy to take a look. -- Ryan C. Underwood, signature.asc Description: PGP signature
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
Hi Gunnar, Thanks for checking. Additional response inline ... > On Aug 25, 2025, at 3:41 AM, Gunnar Krull > wrote: > > Hi Ryan, hi Jeffrey, > > after seeing this issue, I looked into our nameserver log files and found > similar entries. > > There are many queries for _afs3-vlserver including a filename. Sometimes the > full path is included but in that case the "/afs/" prefix is omitted. > The @sys is not involved here. > > The queries are made for each domain suffix that is configured in > /etc/resolv.conf with the "search" directive. > > Here are some selected queries: > > * queries from Debian Bookworm with OpenAFS Client 1.8.13-1, Kernel > 6.1.0-38-amd64 > 24-Aug-2025 19:30:56.916 client @0x7fa6b6f11168 > 2001:638:60c:310::XXX:XXX#34611 (_afs3-vlserver._udp.htaccess): query: > _afs3-vlserver._udp.htaccess IN SRV + (2001:638:60c:310::81:212) > 25-Aug-2025 08:13:19.303 client @0x7fa6b793c168 > 2001:638:60c:310::XXX:XXX#52502 > (_afs3-vlserver._udp.htaccess.informatik.uni-goettingen.de): query: > _afs3-vlserver._udp.htaccess.informatik.uni-goettingen.de IN SRV + > (2001:638:60c:310::81:212) The above queries are ok. The application is attempting to open /afs/.htaccess which doesn’t exist and the OpenAFS cache manager is trying to resolve it as a rw-path to the cell htaccess.informatik.uni-goettingen.de which doesn’t exist. The AuriStorFS client does not attempt to resolve cell aliases via DNS SRV or AFSDB queries but these queries are not unreasonable. > > * queries from Ubuntu 24.04 with OpenAFS Client 1.8.13.2-1ubuntu1, Kernel > 6.14.0-28-generic > 24-Aug-2025 01:26:34.718 client @0x7fa6b794e168 172.27.1.150#36732 > (_afs3-vlserver._udp.timewarrior.json.informatik.uni-goettingen.de): query: > _afs3-vlserver._udp.timewarrior.json.informatik.uni-goettingen.de IN SRV > +E(0) (134.76.81.212) The above two queries could be for the path /afs/timewarrior.json which is being resolved as an alias to the non-existent time warrior.json.informatik.uni-goettingen.de cell. These queries are expected. > 24-Aug-2025 17:46:05.241 client @0x7fa6b783f168 172.27.1.155#41379 > (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/b/./ulsi-prefix/usr/share/desktop-directories.informatik.uni-goettingen.de): > query: > _afs3-vlserver._udp.informatik.uni-goettingen.de/user/b/./ulsi-prefix/usr/share/desktop-directories.informatik.uni-goettingen.de > IN SRV +E(0) (134.76.81.212) The above queries are unexpected. The path being access is /afs/informatik.uni-goettingen.de/user/b/./ulsi-prefix/usr/share/desktop-directories which should not be searched for as a cell or cell alias. A lookup is being performed for "informatik.uni-goettingen.de/user/b/./ulsi-prefix/usr/share/desktop-directories” as a single path component. If your end user can reproduce this query, it would be interesting to strace the process and see what syscalls are being executed. > 25-Aug-2025 07:15:47.756 client @0x7fa6b79fe168 172.27.2.4#34129 > (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de): > query: > _afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de > IN SRV +E(0) (134.76.81.212) Likewise a lookup is being performed for “informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator” as a single path component. An strace of the originating process would be useful to examine. > 25-Aug-2025 07:50:09.586 client @0x7fa6b6e9e168 172.21.0.1#35881 > (_afs3-vlserver._udp.kateconfig.informatik.uni-goettingen.de): query: > _afs3-vlserver._udp.kateconfig.informatik.uni-goettingen.de IN SRV +E(0) > (134.76.81.212) This query is likely for /afs/.kateconfig which is ok. > 25-Aug-2025 07:50:09.626 client @0x7fa6b6e9e168 172.21.0.1#38919 > (_afs3-vlserver._udp.editorconfig.informatik.uni-goettingen.de): query: > _afs3-vlserver._udp.editorconfig.informatik.uni-goettingen.de IN SRV +E(0) > (134.76.81.212) This query is likely for /afs/.editorconfig which is likely ok. > 25-Aug-2025 07:50:10.098 client @0x7fa6b78c5168 172.21.0.1#51962 > (_afs3-vlserver._udp.kateproject.informatik.uni-goettingen.de): query: > _afs3-vlserver._udp.kateproject.informatik.uni-goettingen.de IN SRV +E(0) > (134.76.81.212) This query is likely for /afs/.kateproject which is ok. > The OpenAFS servers are running on Debian Bookworm with version 1.8.13-1. The OpenAFS database and fileservers are uninvolved in the client side queries but thank you for letting us know. > > The DNS SRV records for afs are configured on our nameservers. The Ubuntu 24.04 6.14.0-28-generic kernel supports BPF scripting so it might be possible for someone to write a script to capture the path components passed to afs_linux_lookup() and afs_linux_create() when the dentry->d_parent is /afs. I do not believe OpenAFS fstrace logging has any trace point
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
Hi Ryan, hi Jeffrey, after seeing this issue, I looked into our nameserver log files and found similar entries. There are many queries for _afs3-vlserver including a filename. Sometimes the full path is included but in that case the "/afs/" prefix is omitted. The @sys is not involved here. The queries are made for each domain suffix that is configured in /etc/resolv.conf with the "search" directive. Here are some selected queries: * queries from Debian Bookworm with OpenAFS Client 1.8.13-1, Kernel 6.1.0-38-amd64 24-Aug-2025 19:30:56.916 client @0x7fa6b6f11168 2001:638:60c:310::XXX:XXX#34611 (_afs3-vlserver._udp.htaccess): query: _afs3-vlserver._udp.htaccess IN SRV + (2001:638:60c:310::81:212) 25-Aug-2025 08:13:19.303 client @0x7fa6b793c168 2001:638:60c:310::XXX:XXX#52502 (_afs3-vlserver._udp.htaccess.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.htaccess.informatik.uni-goettingen.de IN SRV + (2001:638:60c:310::81:212) * queries from Ubuntu 24.04 with OpenAFS Client 1.8.13.2-1ubuntu1, Kernel 6.14.0-28-generic 24-Aug-2025 01:26:34.718 client @0x7fa6b794e168 172.27.1.150#36732 (_afs3-vlserver._udp.timewarrior.json.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.timewarrior.json.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 24-Aug-2025 17:46:05.241 client @0x7fa6b783f168 172.27.1.155#41379 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/b/./ulsi-prefix/usr/share/desktop-directories.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/b/./ulsi-prefix/usr/share/desktop-directories.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 25-Aug-2025 07:15:47.756 client @0x7fa6b79fe168 172.27.2.4#34129 (_afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.informatik.uni-goettingen.de/user/a/xx/.xxxlogin/.google_authenticator.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 25-Aug-2025 07:50:09.586 client @0x7fa6b6e9e168 172.21.0.1#35881 (_afs3-vlserver._udp.kateconfig.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.kateconfig.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 25-Aug-2025 07:50:09.626 client @0x7fa6b6e9e168 172.21.0.1#38919 (_afs3-vlserver._udp.editorconfig.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.editorconfig.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) 25-Aug-2025 07:50:10.098 client @0x7fa6b78c5168 172.21.0.1#51962 (_afs3-vlserver._udp.kateproject.informatik.uni-goettingen.de): query: _afs3-vlserver._udp.kateproject.informatik.uni-goettingen.de IN SRV +E(0) (134.76.81.212) The OpenAFS servers are running on Debian Bookworm with version 1.8.13-1. The DNS SRV records for afs are configured on our nameservers. Regards, Gunnar On 22/08/2025 19:08, Jeffrey Altman wrote: Ryan, Since no one else has replied, I will try to explain what I think is happening although I have no idea why or how it is happening. I will also ask a few questions. On Aug 17, 2025, at 7:15 AM, Ryan C. Underwood wrote: I recently tried out the 1.8.13.2 client on my Ubuntu laptop, upgrading from a somewhat older client version. What is the version of Ubuntu and most importantly, what is the kernel version? Historically, I have used @sys-based components in PATH as a way of centrally managing CLI tools without having to distribute them to every system I use. However, this has become problematic in the following way. I’m not sure that @sys is involved. Immediately after system startup, all is well, my shell profile loads immediately and CLI commands also return immediately. However, after some time of system utilization, not necessarily including any heavy AFS usage, doing anything within my shell profile slows to a crawl. The reason is because of the AFS-based PATH component. Instead of stat() calls returning immediately, at this point each one creates many DNS roundtrips fetching some very strange SRV records that I do not recall seeing before, and which are perhaps malformed: 12:52:12.101148 IP 192.168.8.248.46167 > 192.168.8.1.53: 25095+ AFSDB? icequake.net/users/nemesis/bin/bash. (53) 12:52:12.245481 IP 192.168.8.1.53 > 192.168.8.248.46167: 25095 NXDomain 0/1/0 (128) 12:52:12.246822 IP 192.168.8.248.57487 > 192.168.8.1.53: 46166+ AFSDB? icequake.net/users/nemesis/bin/bash.icequake.net. (66) 12:52:12.407925 IP 192.168.8.1.53 > 192.168.8.248.57487: 46166 NXDomain* 0/1/0 (134) 12:52:12.408765 IP 192.168.8.248.60842 > 192.168.8.1.53: 45806+ SRV? _afs3-vlserver._udp.icequake.net/users/nemesis/bin/bash. (73) 12:52:12.612530 IP 192.168.8.1.53 > 192.168.8.248.60842: 45806 NXDomain 0/1/0 (148) 12:52:12.613264 IP 192.168.8.248.33038 > 192.168.8.1.53: 54461+ SRV? _afs3-vlserver._udp.icequake.net/users/nemesis/bin/bash.icequake.net. (86) 12:52:12.817311 IP 192.168.8.1.53 > 192.168.8.248.33038: 54461 NX
Re: [OpenAFS] Strange DNS SRV traffic resulting from stat() in 1.8.13.2
Ryan, Since no one else has replied, I will try to explain what I think is happening although I have no idea why or how it is happening. I will also ask a few questions. > On Aug 17, 2025, at 7:15 AM, Ryan C. Underwood > wrote: > I recently tried out the 1.8.13.2 client on my Ubuntu laptop, > upgrading from a somewhat older client version. What is the version of Ubuntu and most importantly, what is the kernel version? > Historically, I have > used @sys-based components in PATH as a way of centrally managing > CLI tools without having to distribute them to every system I use. > However, this has become problematic in the following way. I’m not sure that @sys is involved. > > Immediately after system startup, all is well, my shell profile loads > immediately and CLI commands also return immediately. However, after > some time of system utilization, not necessarily including any heavy > AFS usage, doing anything within my shell profile slows to a crawl. > The reason is because of the AFS-based PATH component. Instead of > stat() calls returning immediately, at this point each one creates > many DNS roundtrips fetching some very strange SRV records that I do > not recall seeing before, and which are perhaps malformed: > > 12:52:12.101148 IP 192.168.8.248.46167 > 192.168.8.1.53: 25095+ AFSDB? > icequake.net/users/nemesis/bin/bash. (53) > 12:52:12.245481 IP 192.168.8.1.53 > 192.168.8.248.46167: 25095 NXDomain 0/1/0 > (128) > 12:52:12.246822 IP 192.168.8.248.57487 > 192.168.8.1.53: 46166+ AFSDB? > icequake.net/users/nemesis/bin/bash.icequake.net. (66) > 12:52:12.407925 IP 192.168.8.1.53 > 192.168.8.248.57487: 46166 NXDomain* > 0/1/0 (134) > 12:52:12.408765 IP 192.168.8.248.60842 > 192.168.8.1.53: 45806+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/bin/bash. (73) > 12:52:12.612530 IP 192.168.8.1.53 > 192.168.8.248.60842: 45806 NXDomain 0/1/0 > (148) > 12:52:12.613264 IP 192.168.8.248.33038 > 192.168.8.1.53: 54461+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/bin/bash.icequake.net. (86) > 12:52:12.817311 IP 192.168.8.1.53 > 192.168.8.248.33038: 54461 NXDomain* > 0/1/0 (154) > 12:52:12.817829 IP 192.168.8.248.33177 > 192.168.8.1.53: 53983+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/bin/bash. (73) > 12:52:12.958650 IP 192.168.8.1.53 > 192.168.8.248.33177: 53983 NXDomain 0/1/0 > (148) > 12:52:12.959152 IP 192.168.8.248.50256 > 192.168.8.1.53: 12967+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/bin/bash.icequake.net. (86) > 12:52:13.124390 IP 192.168.8.1.53 > 192.168.8.248.50256: 12967 NXDomain* > 0/1/0 (154) > 12:52:13.125189 IP 192.168.8.248.47935 > 192.168.8.1.53: 56862+ AFSDB? > icequake.net/users/nemesis/bin/bash. (53) > 12:52:13.329797 IP 192.168.8.1.53 > 192.168.8.248.47935: 56862 NXDomain 0/1/0 > (128) > 12:52:13.330370 IP 192.168.8.248.58386 > 192.168.8.1.53: 55085+ AFSDB? > icequake.net/users/nemesis/bin/bash.icequake.net. (66) > 12:52:13.537502 IP 192.168.8.1.53 > 192.168.8.248.58386: 55085 NXDomain* > 0/1/0 (134) > 12:52:13.540776 IP 192.168.8.248.41895 > 192.168.8.1.53: 6729+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/@sys/bin/bash. (78) > 12:52:13.740662 IP 192.168.8.1.53 > 192.168.8.248.41895: 6729 NXDomain 0/1/0 > (153) > 12:52:13.741143 IP 192.168.8.248.37102 > 192.168.8.1.53: 33110+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/@sys/bin/bash.icequake.net. > (91) > 12:52:14.159513 IP 192.168.8.1.53 > 192.168.8.248.37102: 33110 NXDomain* > 0/1/0 (159) > 12:52:14.160055 IP 192.168.8.248.58652 > 192.168.8.1.53: 64126+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/@sys/bin/bash. (78) > 12:52:14.731726 IP 192.168.8.1.53 > 192.168.8.248.58652: 64126 NXDomain 0/1/0 > (153) > 12:52:14.732197 IP 192.168.8.248.41774 > 192.168.8.1.53: 19149+ SRV? > _afs3-vlserver._udp.icequake.net/users/nemesis/@sys/bin/bash.icequake.net. > (91) > [...] There are two types of DNS queries being issued. The first type are DNS SRV queries for service “afs3-vlserver” using protocol “udp”. The second type of query are the deprecated DNS AFSDB queries the DNS SRV queries are intended to replace. However, the queries are malformed and are leaking to the world every path that is traversed. > Can someone explain what is going on here? Whatever is happening, it > is bypassing the cache manager because the same strange queries are > issued repeatedly as the shell profile executes. It's additionally > painful in the case of my laptop, since the ultimate DNS server is on > the other end of a VPN link. Unless the afsd dynamic root functionality is disabled, DNS SRV or DNS AFSDB queries would be expected if the cache manager’s CellServDB does not contain a list of database servers for the cell “ice quake.net”. In particular we would expect to see a DNS SRV query for :_afs3-vlserver._udp.icequake.net and if that query failed, then a fallback query for DNS AFSDB “icequake.net”. Those queries are not pr
