Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance
I'm still seeing the same problem after disabling AppArmor, so I think it must be some other root problem. On Wed, Oct 24, 2018 at 2:41 PM Neil Jerram wrote: > > Thanks so much for these hints, Erlon. I will look closer at AppArmor. > > Neil > > On Wed, Oct 24, 2018 at 1:41 PM Erlon Cruz wrote: > > > > PS. Don't forget that if you change or disable AppArmor you will have to > > reboot the host so the kernel gets reloaded. > > > > Em qua, 24 de out de 2018 às 09:40, Erlon Cruz > > escreveu: > >> > >> I think that there's a change that AppArmor is blocking the access. Have > >> you checked the dmesg messages related with apparmor? > >> > >> Em sex, 19 de out de 2018 às 09:38, Neil Jerram escreveu: > >>> > >>> Wracking my brains over this one, would appreciate any pointers... > >>> > >>> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu > >>> Bionic. The first compute node is an NFS server for > >>> /var/lib/nova/instances, and the other compute nodes mount that as NFS > >>> clients. > >>> > >>> Problem: Sometimes, when launching an instance which is scheduled to one > >>> of the client nodes, nova-compute (in imagebackend.py) gets Permission > >>> Denied (errno 13) when calling utime to touch the timestamp on the > >>> instance file. > >>> > >>> Through various bits of debugging and hackery, I've established that: > >>> > >>> - it looks like the problem never occurs when this is the call that > >>> bootstraps the privsep setup; but it does occur quite frequently on later > >>> calls > >>> > >>> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in > >>> between) > >>> > >>> - the instance file does exist, and is owned by root with read/write > >>> permission for root > >>> > >>> - the privsep helper is running as root > >>> > >>> - the privsep helper receives and executes the request - so it's not a > >>> problem with communication between nova-compute and the helper > >>> > >>> - root is uid 0 on both NFS server and client > >>> > >>> - NFS setup does not have the root_squash option > >>> > >>> - there is some AppArmor setup, on both client and server, and I haven't > >>> yet worked out whether that might be relevant. > >>> > >>> Any ideas? > >>> > >>> Many thanks, > >>> Neil > >>> > >>> __ > >>> OpenStack Development Mailing List (not for usage questions) > >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance
Thanks so much for these hints, Erlon. I will look closer at AppArmor. Neil On Wed, Oct 24, 2018 at 1:41 PM Erlon Cruz wrote: > > PS. Don't forget that if you change or disable AppArmor you will have to > reboot the host so the kernel gets reloaded. > > Em qua, 24 de out de 2018 às 09:40, Erlon Cruz escreveu: >> >> I think that there's a change that AppArmor is blocking the access. Have you >> checked the dmesg messages related with apparmor? >> >> Em sex, 19 de out de 2018 às 09:38, Neil Jerram escreveu: >>> >>> Wracking my brains over this one, would appreciate any pointers... >>> >>> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu >>> Bionic. The first compute node is an NFS server for >>> /var/lib/nova/instances, and the other compute nodes mount that as NFS >>> clients. >>> >>> Problem: Sometimes, when launching an instance which is scheduled to one of >>> the client nodes, nova-compute (in imagebackend.py) gets Permission Denied >>> (errno 13) when calling utime to touch the timestamp on the instance file. >>> >>> Through various bits of debugging and hackery, I've established that: >>> >>> - it looks like the problem never occurs when this is the call that >>> bootstraps the privsep setup; but it does occur quite frequently on later >>> calls >>> >>> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in >>> between) >>> >>> - the instance file does exist, and is owned by root with read/write >>> permission for root >>> >>> - the privsep helper is running as root >>> >>> - the privsep helper receives and executes the request - so it's not a >>> problem with communication between nova-compute and the helper >>> >>> - root is uid 0 on both NFS server and client >>> >>> - NFS setup does not have the root_squash option >>> >>> - there is some AppArmor setup, on both client and server, and I haven't >>> yet worked out whether that might be relevant. >>> >>> Any ideas? >>> >>> Many thanks, >>> Neil >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance
PS. Don't forget that if you change or disable AppArmor you will have to reboot the host so the kernel gets reloaded. Em qua, 24 de out de 2018 às 09:40, Erlon Cruz escreveu: > I think that there's a change that AppArmor is blocking the access. Have > you checked the dmesg messages related with apparmor? > > Em sex, 19 de out de 2018 às 09:38, Neil Jerram escreveu: > >> Wracking my brains over this one, would appreciate any pointers... >> >> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu >> Bionic. The first compute node is an NFS server for >> /var/lib/nova/instances, and the other compute nodes mount that as NFS >> clients. >> >> Problem: Sometimes, when launching an instance which is scheduled to one >> of the client nodes, nova-compute (in imagebackend.py) gets Permission >> Denied (errno 13) when calling utime to touch the timestamp on the instance >> file. >> >> Through various bits of debugging and hackery, I've established that: >> >> - it looks like the problem never occurs when this is the call that >> bootstraps the privsep setup; but it does occur quite frequently on later >> calls >> >> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in >> between) >> >> - the instance file does exist, and is owned by root with read/write >> permission for root >> >> - the privsep helper is running as root >> >> - the privsep helper receives and executes the request - so it's not a >> problem with communication between nova-compute and the helper >> >> - root is uid 0 on both NFS server and client >> >> - NFS setup does not have the root_squash option >> >> - there is some AppArmor setup, on both client and server, and I haven't >> yet worked out whether that might be relevant. >> >> Any ideas? >> >> Many thanks, >> Neil >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance
I think that there's a change that AppArmor is blocking the access. Have you checked the dmesg messages related with apparmor? Em sex, 19 de out de 2018 às 09:38, Neil Jerram escreveu: > Wracking my brains over this one, would appreciate any pointers... > > Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu > Bionic. The first compute node is an NFS server for > /var/lib/nova/instances, and the other compute nodes mount that as NFS > clients. > > Problem: Sometimes, when launching an instance which is scheduled to one > of the client nodes, nova-compute (in imagebackend.py) gets Permission > Denied (errno 13) when calling utime to touch the timestamp on the instance > file. > > Through various bits of debugging and hackery, I've established that: > > - it looks like the problem never occurs when this is the call that > bootstraps the privsep setup; but it does occur quite frequently on later > calls > > - when the problem occurs, retrying doesn't help (5 times, with 0.5s in > between) > > - the instance file does exist, and is owned by root with read/write > permission for root > > - the privsep helper is running as root > > - the privsep helper receives and executes the request - so it's not a > problem with communication between nova-compute and the helper > > - root is uid 0 on both NFS server and client > > - NFS setup does not have the root_squash option > > - there is some AppArmor setup, on both client and server, and I haven't > yet worked out whether that might be relevant. > > Any ideas? > > Many thanks, > Neil > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance
Wracking my brains over this one, would appreciate any pointers... Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu Bionic. The first compute node is an NFS server for /var/lib/nova/instances, and the other compute nodes mount that as NFS clients. Problem: Sometimes, when launching an instance which is scheduled to one of the client nodes, nova-compute (in imagebackend.py) gets Permission Denied (errno 13) when calling utime to touch the timestamp on the instance file. Through various bits of debugging and hackery, I've established that: - it looks like the problem never occurs when this is the call that bootstraps the privsep setup; but it does occur quite frequently on later calls - when the problem occurs, retrying doesn't help (5 times, with 0.5s in between) - the instance file does exist, and is owned by root with read/write permission for root - the privsep helper is running as root - the privsep helper receives and executes the request - so it's not a problem with communication between nova-compute and the helper - root is uid 0 on both NFS server and client - NFS setup does not have the root_squash option - there is some AppArmor setup, on both client and server, and I haven't yet worked out whether that might be relevant. Any ideas? Many thanks, Neil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
