Re: [OpenAFS] problems with ubuntu 18.04 client
Am 02.10.2018 um 15:44 schrieb Andreas Ladanyi: We were probably just lucky, or the packages from the 1.8 ppa http://ppa.launchpad.net/openafs/stable/ubuntu never had the problem. Did you use 1.8.0 from ppa for the clients in the past or did you start at 1.8.2 when switching from 1.6 release ? Apologies for the late answer. We had to switch to the PPA ages ago (since some time in 2013), since the regular Ubuntu DKMS-module made occasionally troubles with Ubuntu kernel updates. So the switch for us was at the same time when 1.8 appeared in the PPA. Looking back, it was on the 25th of April 2018 - we had to change our "apt-get update" line to something like this, before it worked, though apt-get --allow-releaseinfo-change-label update Without that, Ubuntu 18.04 wouldn't accept the name-change of 'OpenAFS 1.6.x stable releases' to 'OpenAFS stable releases (1.8.x)' (Ubuntu 14.04 and 16.04 accepted it without that) Greetings, Gaja Peters ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
Hi ! Thanks all for input and assistance ! For now i want to say all-clear ... The situation is not so dramaticly as i thought, We've got 3 of about 20 ubuntu 18.04 machines, on which this problem appears ... but all 3 machines comes up with this problem in 2 days and i want rather be active early than late ;-) two of them got only once, and one, has got this problem multiple times In this case i've done the upgrade to 1.8.2 ... After it appears again, i've deleted the cache total with "rm -rf" and start the client again ... till now the user hasn't contacted us again .. :-) Only short comment to the hint of Harald in terms of PAG i think, it won't be a PAG problem, because the user couldn't see /afs/desy.de/user and at the same time i'm as well, for me it seems to be a client-problem not a PAG and nothing suspicious in the fileserver log ... in terms to Andreas Does a reboot solve the issue ? Yes Did you use ubuntu 18.04 and afs 1.6 before switching to afs 1.8 ? No, we use the ubuntu openafs-client '1.8.0 ~ pre5-1' out of the box Are there issues from volumes in the salvager log ? No messages in salvager logs and in terms to Benjamin Thanks for the hint with apparmor policy, maybe we need this sometime ;-) All the best and i wll report again, if Problems comes again thanks & cheers, martin On Fri, 5 Oct 2018, Benjamin Kaduk wrote: On Fri, Oct 05, 2018 at 09:21:31AM +0200, Andreas Ladanyi wrote: You need to update your apparmor policy to allow rw access to /var/cache/openafs/**; accesses are performed by the kernel cache manager on behalf of all processes and apparmor's view of the credentials do not line up. MIT's configuration does this as of https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7 -Ben Is the openafs bug which is workaround by the apparmor config since release 1.8 or also in 1.6 ? The date of the commit is from february this year. AFS 1.8 was released at april this year. The details of the openafs bug are not known (otherwise it would be fixed already). It may be new in 1.8 -- January/February was when we started introducing 1.8.0 prereleases into Debian/Ubuntu, so the apparmor issues may have been new at that time. -Ben ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
On Fri, Oct 05, 2018 at 09:21:31AM +0200, Andreas Ladanyi wrote: > > You need to update your apparmor policy to allow rw access to > > /var/cache/openafs/**; accesses are performed by the kernel cache manager > > on behalf of all processes and apparmor's view of the credentials do not > > line up. MIT's configuration does this as of > > https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7 > > > > -Ben > > Is the openafs bug which is workaround by the apparmor config since > release 1.8 or also in 1.6 ? > > The date of the commit is from february this year. AFS 1.8 was released > at april this year. The details of the openafs bug are not known (otherwise it would be fixed already). It may be new in 1.8 -- January/February was when we started introducing 1.8.0 prereleases into Debian/Ubuntu, so the apparmor issues may have been new at that time. -Ben ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
> You need to update your apparmor policy to allow rw access to > /var/cache/openafs/**; accesses are performed by the kernel cache manager > on behalf of all processes and apparmor's view of the credentials do not > line up. MIT's configuration does this as of > https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7 > > -Ben Is the openafs bug which is workaround by the apparmor config since release 1.8 or also in 1.6 ? The date of the commit is from february this year. AFS 1.8 was released at april this year. Andy ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
You need to update your apparmor policy to allow rw access to /var/cache/openafs/**; accesses are performed by the kernel cache manager on behalf of all processes and apparmor's view of the credentials do not line up. MIT's configuration does this as of https://github.com/mit-athena/apparmor-config/commit/e3b34ce4d455574a235bbb8a512ad99f75155bc7 -Ben On Thu, Oct 04, 2018 at 11:49:04AM +0200, Martin Flemming wrote: > > Hi, again ! > > Shortly after i send this mail to the list, one of the user report > back ... same problemes like before ... :-( > > In an old terminal (where afs was running well) everyhing seems to be ok, > create files,folder, pwd... etc) > but for every new one terminal or GUI-Application the AFS-Tree > /afs/desy.de/user/ is not available anymore > > Other directories in /afs/desy.de/ are available ... > > > > My first idea: I've to delete the afs-cache, after the upgrade from 1.8.0 ~ > pre5-1 to 1.8.2, haven't i ? > > > in dmesg we see follow messages > > [42798.792549] audit: type=1400 audit(1538639477.910:42): apparmor="DENIED" > operation="file_perm" profile="/usr/bin/evince" > name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" > requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 > [42798.792552] audit: type=1400 audit(1538639477.910:43): apparmor="DENIED" > operation="file_perm" profile="/usr/bin/evince" > name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" > requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 > [42798.795332] audit: type=1400 audit(1538639477.914:44): apparmor="DENIED" > operation="file_perm" profile="/usr/bin/evince" > name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" > requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 > [42798.795334] audit: type=1400 audit(1538639477.914:45): apparmor="DENIED" > operation="file_perm" profile="/usr/bin/evince" > name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" > requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 > [42811.588120] audit: type=1400 audit(1538639490.705:46): apparmor="DENIED" > operation="file_perm" profile="/usr/bin/evince" > name="/var/cache/openafs/D71/V146768" pid=29664 comm="EvJobScheduler" > requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 > > > Any further suggestions for debugging ? > > thanks & cheers, > > martin > > On Thu, 4 Oct 2018, Martin Flemming wrote: > > > > > Thanks all for input ! > > > > We will try at first upgrade to > > > > https://launchpad.net/~openafs/+archive/ubuntu/stable (1.8.2) > > > > and see ... if there also problems then we'll report it and switch maybe > > to > > an downgrade https://launchpad.net/~openafs/+archive/ubuntu/stable (1.6.23) > > > > thanks & cheers > > > > Martin > ___ > OpenAFS-info mailing list > OpenAFS-info@openafs.org > https://lists.openafs.org/mailman/listinfo/openafs-info ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
>> Are you in the same or different PAG? > > Hmm, i think that's not the reason, > if i login into the same Computer, the tree /afs/desy.de/user is also > missing for me ... Yes, but a new login probably gives you a new pag and a new security context. Question is if the old pag and security context still continues to work and if the answer is yes why in that case that differs from a new pag. Any complaints in the file server log about the client in question? Harald. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
Hi, > if i login into the same Computer, the tree /afs/desy.de/user is also > missing for me ... > Does a reboot solve the issue ? Did you use ubuntu 18.04 and afs 1.6 before switching to afs 1.8 ? Are there issues from volumes in the salvager log ? regards, Andy ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
On Thu, 4 Oct 2018, Harald Barth wrote: In an old terminal (where afs was running well) everyhing seems to be ok, create files,folder, pwd... etc) Are you in the same or different PAG? Hmm, i think that's not the reason, if i login into the same Computer, the tree /afs/desy.de/user is also missing for me ... martin ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
On Thu, 4 Oct 2018, Andreas Ladanyi wrote: Hi Martin, Hi, again ! Shortly after i send this mail to the list, one of the user report back ... same problemes like before ... :-( In an old terminal (where afs was running well) everyhing seems to be ok, create files,folder, pwd... etc) but for every new one terminal or GUI-Application the AFS-Tree /afs/desy.de/user/ is not available anymore are the volumes from the users online or offline ? If the volumes are offline could you salvage (bos solvage ...) the volumes which result in the volumes go online and later the volumes go offline again ? Other directories in /afs/desy.de/ are available ... The Volumes are still online, from any other Computer the hole tree /afs/desy.de/user/ is available and also the user-Volume itself ... cheers, martin
Re: [OpenAFS] problems with ubuntu 18.04 client
> In an old terminal (where afs was running well) everyhing seems to be > ok, create files,folder, pwd... etc) Are you in the same or different PAG? Harald. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
Hi Martin, > > Hi, again ! > > Shortly after i send this mail to the list, one of the user report > back ... same problemes like before ... :-( > > In an old terminal (where afs was running well) everyhing seems to be > ok, create files,folder, pwd... etc) but for every new one terminal > or GUI-Application the AFS-Tree /afs/desy.de/user/ is not available > anymore are the volumes from the users online or offline ? If the volumes are offline could you salvage (bos solvage ...) the volumes which result in the volumes go online and later the volumes go offline again ? > > Other directories in /afs/desy.de/ are available ... > cheers, Andy ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
Hi, again ! Shortly after i send this mail to the list, one of the user report back ... same problemes like before ... :-( In an old terminal (where afs was running well) everyhing seems to be ok, create files,folder, pwd... etc) but for every new one terminal or GUI-Application the AFS-Tree /afs/desy.de/user/ is not available anymore Other directories in /afs/desy.de/ are available ... My first idea: I've to delete the afs-cache, after the upgrade from 1.8.0 ~ pre5-1 to 1.8.2, haven't i ? in dmesg we see follow messages [42798.792549] audit: type=1400 audit(1538639477.910:42): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 [42798.792552] audit: type=1400 audit(1538639477.910:43): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 [42798.795332] audit: type=1400 audit(1538639477.914:44): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D81/V166635" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 [42798.795334] audit: type=1400 audit(1538639477.914:45): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D80/V165031" pid=29610 comm="evince" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 [42811.588120] audit: type=1400 audit(1538639490.705:46): apparmor="DENIED" operation="file_perm" profile="/usr/bin/evince" name="/var/cache/openafs/D71/V146768" pid=29664 comm="EvJobScheduler" requested_mask="r" denied_mask="r" fsuid=25574 ouid=0 Any further suggestions for debugging ? thanks & cheers, martin On Thu, 4 Oct 2018, Martin Flemming wrote: Thanks all for input ! We will try at first upgrade to https://launchpad.net/~openafs/+archive/ubuntu/stable (1.8.2) and see ... if there also problems then we'll report it and switch maybe to an downgrade https://launchpad.net/~openafs/+archive/ubuntu/stable (1.6.23) thanks & cheers Martin ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
Thanks all for input ! We will try at first upgrade to https://launchpad.net/~openafs/+archive/ubuntu/stable (1.8.2) and see ... if there also problems then we'll report it and switch maybe to an downgrade https://launchpad.net/~openafs/+archive/ubuntu/stable (1.6.23) thanks & cheers Martin ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
> We were probably just lucky, or the packages from the 1.8 ppa > http://ppa.launchpad.net/openafs/stable/ubuntu never had the problem. Did you use 1.8.0 from ppa for the clients in the past or did you start at 1.8.2 when switching from 1.6 release ? > > > Greetings, > Gaja Peters > cheers, Andy ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
Am 02.10.2018 um 08:58 schrieb Andreas Ladanyi: we had the same problems. We are using the 1.6 release from ppa ( https://launchpad.net/~openafs/%2Barchive/ubuntu/stable ) on server and client now and there seems to be no problems anymore. We were probably just lucky, or the packages from the 1.8 ppa http://ppa.launchpad.net/openafs/stable/ubuntu never had the problem. At the moment, we live with all clients and all servers at 1.8.2 and (crossing fingers and knocking wood) everything seems to work. The one problem we occasionally have (which has nothing to do with the above) is that when people force a shutdown or reboot at an inopportune moment during an OpenAFS update, the AFS is entirely lost (the old DKMS-package deleted, the new not yet built), but the system thinks, "oh the home-directory should be /afs/math.uni-hamburg.de/users/../... - let's just create that directory and the entire tree". In that case, a simple rebuild of the DKMS-module and "service openafs-client force-start" fixes the problem however. Greetings, Gaja Peters ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
Hi Martin, we had the same problems. We are using the 1.6 release from ppa ( https://launchpad.net/~openafs/%2Barchive/ubuntu/stable ) on server and client now and there seems to be no problems anymore. 1.8 pre on client from default ubuntu 18.04 repo and if i remember correctly 1.8.0 from paa on the client was a problem. And yes after rebooting a 1.8 client the problem has gone. Andy > > Hi ! > > In the last days we're getting more and more messages from users about > having problems accessing their home-directories or subfolder or the > hole user-tree /afs/desy.de/user is missing ... > > Using ubuntu 18.04 with the regular openafs-client 1.8.0 ~ pre5-1 and > 4.15.0-34-generic kernel ... > Server are running with mostly 1.6.22 / 1.6.23 with Centos7 ... > > Are any problems known here with this combination ? > > Solution seems to be only a reboot of the system ... Unfortunatley > i've got not time to analyze one of these Desktops, because the user > report this problem afterwards of the reboot ... > > Thanks & Cheers > > > Martin > > ___ > OpenAFS-info mailing list > OpenAFS-info@openafs.org > https://lists.openafs.org/mailman/listinfo/openafs-info -- Karlsruher Institut für Technologie (KIT) Fakultät für Informatik ATIS – Abteilung Technische Infrastruktur Dipl.-Ing. Andreas Ladanyi - Systemadministrator - Am Fasanengarten 5, Gebäude 50.34, Raum 013 76131 Karlsruhe Telefon: +49 721 608 - 4 3663 Fax: +49 721 608 - 4 6699 E-Mail: andreas.lada...@kit.edu www.atis.informatik.kit.edu www.kit.edu KIT - Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] problems with ubuntu 18.04 client
On Mon, Oct 01, 2018 at 11:55:48PM +0200, martin.flemm...@desy.de wrote: > > Hi ! > > In the last days we're getting more and more messages from users about > having problems accessing their home-directories or subfolder or the hole > user-tree /afs/desy.de/user is missing ... > > Using ubuntu 18.04 with the regular openafs-client 1.8.0 ~ pre5-1 and > 4.15.0-34-generic kernel ... > Server are running with mostly 1.6.22 / 1.6.23 with Centos7 ... > > Are any problems known here with this combination ? > > Solution seems to be only a reboot of the system ... Unfortunatley i've > got not time to analyze one of these Desktops, because the user report > this problem afterwards of the reboot ... The only thing that comes to mind would be the fix for broken volume callbacks from https://gerrit.openafs.org/13140 , since https://gerrit.openafs.org/12851 (already in 1.8.0pre5) was supposed to take care of the main source of spurious ENOENTs. Any chance you could try a 1.8.1 or newer (e.g., the 1.8.2 available at https://launchpad.net/~openafs/+archive/ubuntu/stable)? Thanks, Ben ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
[OpenAFS] problems with ubuntu 18.04 client
Hi ! In the last days we're getting more and more messages from users about having problems accessing their home-directories or subfolder or the hole user-tree /afs/desy.de/user is missing ... Using ubuntu 18.04 with the regular openafs-client 1.8.0 ~ pre5-1 and 4.15.0-34-generic kernel ... Server are running with mostly 1.6.22 / 1.6.23 with Centos7 ... Are any problems known here with this combination ? Solution seems to be only a reboot of the system ... Unfortunatley i've got not time to analyze one of these Desktops, because the user report this problem afterwards of the reboot ... Thanks & Cheers Martin ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
