Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance

2018-10-25 Thread Neil Jerram
I'm still seeing the same problem after disabling AppArmor, so I think
it must be some other root problem.

On Wed, Oct 24, 2018 at 2:41 PM Neil Jerram  wrote:
>
> Thanks so much for these hints, Erlon.  I will look closer at AppArmor.
>
> Neil
>
> On Wed, Oct 24, 2018 at 1:41 PM Erlon Cruz  wrote:
> >
> > PS. Don't forget that if you change or disable AppArmor you will have to 
> > reboot the host so the kernel gets reloaded.
> >
> > Em qua, 24 de out de 2018 às 09:40, Erlon Cruz  
> > escreveu:
> >>
> >> I think that there's a change that AppArmor is blocking the access. Have 
> >> you checked the dmesg messages related with apparmor?
> >>
> >> Em sex, 19 de out de 2018 às 09:38, Neil Jerram  escreveu:
> >>>
> >>> Wracking my brains over this one, would appreciate any pointers...
> >>>
> >>> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu 
> >>> Bionic. The first compute node is an NFS server for 
> >>> /var/lib/nova/instances, and the other compute nodes mount that as NFS 
> >>> clients.
> >>>
> >>> Problem: Sometimes, when launching an instance which is scheduled to one 
> >>> of the client nodes, nova-compute (in imagebackend.py) gets Permission 
> >>> Denied (errno 13) when calling utime to touch the timestamp on the 
> >>> instance file.
> >>>
> >>> Through various bits of debugging and hackery, I've established that:
> >>>
> >>> - it looks like the problem never occurs when this is the call that 
> >>> bootstraps the privsep setup; but it does occur quite frequently on later 
> >>> calls
> >>>
> >>> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in 
> >>> between)
> >>>
> >>> - the instance file does exist, and is owned by root with read/write 
> >>> permission for root
> >>>
> >>> - the privsep helper is running as root
> >>>
> >>> - the privsep helper receives and executes the request - so it's not a 
> >>> problem with communication between nova-compute and the helper
> >>>
> >>> - root is uid 0 on both NFS server and client
> >>>
> >>> - NFS setup does not have the root_squash option
> >>>
> >>> - there is some AppArmor setup, on both client and server, and I haven't 
> >>> yet worked out whether that might be relevant.
> >>>
> >>> Any ideas?
> >>>
> >>> Many thanks,
> >>>   Neil
> >>>
> >>> __
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance

2018-10-24 Thread Neil Jerram
Thanks so much for these hints, Erlon.  I will look closer at AppArmor.

Neil

On Wed, Oct 24, 2018 at 1:41 PM Erlon Cruz  wrote:
>
> PS. Don't forget that if you change or disable AppArmor you will have to 
> reboot the host so the kernel gets reloaded.
>
> Em qua, 24 de out de 2018 às 09:40, Erlon Cruz  escreveu:
>>
>> I think that there's a change that AppArmor is blocking the access. Have you 
>> checked the dmesg messages related with apparmor?
>>
>> Em sex, 19 de out de 2018 às 09:38, Neil Jerram  escreveu:
>>>
>>> Wracking my brains over this one, would appreciate any pointers...
>>>
>>> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu 
>>> Bionic. The first compute node is an NFS server for 
>>> /var/lib/nova/instances, and the other compute nodes mount that as NFS 
>>> clients.
>>>
>>> Problem: Sometimes, when launching an instance which is scheduled to one of 
>>> the client nodes, nova-compute (in imagebackend.py) gets Permission Denied 
>>> (errno 13) when calling utime to touch the timestamp on the instance file.
>>>
>>> Through various bits of debugging and hackery, I've established that:
>>>
>>> - it looks like the problem never occurs when this is the call that 
>>> bootstraps the privsep setup; but it does occur quite frequently on later 
>>> calls
>>>
>>> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in 
>>> between)
>>>
>>> - the instance file does exist, and is owned by root with read/write 
>>> permission for root
>>>
>>> - the privsep helper is running as root
>>>
>>> - the privsep helper receives and executes the request - so it's not a 
>>> problem with communication between nova-compute and the helper
>>>
>>> - root is uid 0 on both NFS server and client
>>>
>>> - NFS setup does not have the root_squash option
>>>
>>> - there is some AppArmor setup, on both client and server, and I haven't 
>>> yet worked out whether that might be relevant.
>>>
>>> Any ideas?
>>>
>>> Many thanks,
>>>   Neil
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance

2018-10-24 Thread Erlon Cruz
PS. Don't forget that if you change or disable AppArmor you will have to
reboot the host so the kernel gets reloaded.

Em qua, 24 de out de 2018 às 09:40, Erlon Cruz 
escreveu:

> I think that there's a change that AppArmor is blocking the access. Have
> you checked the dmesg messages related with apparmor?
>
> Em sex, 19 de out de 2018 às 09:38, Neil Jerram  escreveu:
>
>> Wracking my brains over this one, would appreciate any pointers...
>>
>> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu
>> Bionic. The first compute node is an NFS server for
>> /var/lib/nova/instances, and the other compute nodes mount that as NFS
>> clients.
>>
>> Problem: Sometimes, when launching an instance which is scheduled to one
>> of the client nodes, nova-compute (in imagebackend.py) gets Permission
>> Denied (errno 13) when calling utime to touch the timestamp on the instance
>> file.
>>
>> Through various bits of debugging and hackery, I've established that:
>>
>> - it looks like the problem never occurs when this is the call that
>> bootstraps the privsep setup; but it does occur quite frequently on later
>> calls
>>
>> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in
>> between)
>>
>> - the instance file does exist, and is owned by root with read/write
>> permission for root
>>
>> - the privsep helper is running as root
>>
>> - the privsep helper receives and executes the request - so it's not a
>> problem with communication between nova-compute and the helper
>>
>> - root is uid 0 on both NFS server and client
>>
>> - NFS setup does not have the root_squash option
>>
>> - there is some AppArmor setup, on both client and server, and I haven't
>> yet worked out whether that might be relevant.
>>
>> Any ideas?
>>
>> Many thanks,
>>   Neil
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance

2018-10-24 Thread Erlon Cruz
I think that there's a change that AppArmor is blocking the access. Have
you checked the dmesg messages related with apparmor?

Em sex, 19 de out de 2018 às 09:38, Neil Jerram  escreveu:

> Wracking my brains over this one, would appreciate any pointers...
>
> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu
> Bionic. The first compute node is an NFS server for
> /var/lib/nova/instances, and the other compute nodes mount that as NFS
> clients.
>
> Problem: Sometimes, when launching an instance which is scheduled to one
> of the client nodes, nova-compute (in imagebackend.py) gets Permission
> Denied (errno 13) when calling utime to touch the timestamp on the instance
> file.
>
> Through various bits of debugging and hackery, I've established that:
>
> - it looks like the problem never occurs when this is the call that
> bootstraps the privsep setup; but it does occur quite frequently on later
> calls
>
> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in
> between)
>
> - the instance file does exist, and is owned by root with read/write
> permission for root
>
> - the privsep helper is running as root
>
> - the privsep helper receives and executes the request - so it's not a
> problem with communication between nova-compute and the helper
>
> - root is uid 0 on both NFS server and client
>
> - NFS setup does not have the root_squash option
>
> - there is some AppArmor setup, on both client and server, and I haven't
> yet worked out whether that might be relevant.
>
> Any ideas?
>
> Many thanks,
>   Neil
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev