Re: [ceph-users] hanging slow requests: failed to authpin, subtree is being exported

2019-10-21 Thread Marc Roos
 
I think I am having this issue also (at least I had with luminous) I had 
to remove the hidden temp files rsync had left, when the cephfs mount 
'stalled', otherwise I would never be able to complete the rsync.


-Original Message-
Cc: ceph-users
Subject: Re: [ceph-users] hanging slow requests: failed to authpin, 
subtree is being exported


I've made a ticket for this issue: https://tracker.ceph.com/issues/42338

Thanks again!

K

On 15/10/2019 18:00, Kenneth Waegeman wrote:
> Hi Robert, all,
>
>
> On 23/09/2019 17:37, Robert LeBlanc wrote:
>> On Mon, Sep 23, 2019 at 4:14 AM Kenneth Waegeman 
>>  wrote:
>>> Hi all,
>>>
>>> When syncing data with rsync, I'm often getting blocked slow 
>>> requests, which also block access to this path.
>>>
>>>> 2019-09-23 11:25:49.477 7f4f401e8700 0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 31.895478 seconds old, received at 2019-09-23
>>>> 11:25:17.598152: client_request(client.38352684:92684 lookup
>>>> #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:26:19.477 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 61.896079 seconds old, received at 2019-09-23
>>>> 11:25:17.598152: client_request(client.38352684:92684 lookup
>>>> #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:27:19.478 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 121.897268 seconds old, received at 2019-09-23
>>>> 11:25:17.598152: client_request(client.38352684:92684 lookup
>>>> #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:29:19.488 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 241.899467 seconds old, received at 2019-09-23
>>>> 11:25:17.598152: client_request(client.38352684:92684 lookup
>>>> #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:33:19.680 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 482.087927 seconds old, received at 2019-09-23
>>>> 11:25:17.598152: client_request(client.38352684:92684 lookup
>>>> #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:36:09.881 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 32.677511 seconds old, received at 2019-09-23
>>>> 11:35:37.217113: client_request(client.38347357:111963 lookup 
>>>> #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:36:39.881 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 62.678132 seconds old, received at 2019-09-23
>>>> 11:35:37.217113: client_request(client.38347357:111963 lookup 
>>>> #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:37:39.891 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 122.679273 seconds old, received at 2019-09-23
>>>> 11:35:37.217113: client_request(client.38347357:111963 lookup 
>>>> #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:39:39.892 7f4f401e8700  0 log_channel(cluster) log 
>>>> [WRN]
>>>> : slow request 242.684667 seconds old, received at 2019-09-23
>>>> 11:35:37.217113: client_request(client.38347357:111963 lookup 
>>>> #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
>>>> caller_gid=0{0,}) currently failed to authpin, subtree is being 
>>>> exported
>>>> 2019-09-23 11:41:19.893 7f4f401e8700  0 log_ch

Re: [ceph-users] hanging slow requests: failed to authpin, subtree is being exported

2019-10-21 Thread Kenneth Waegeman


I've made a ticket for this issue: https://tracker.ceph.com/issues/42338

Thanks again!

K

On 15/10/2019 18:00, Kenneth Waegeman wrote:

Hi Robert, all,


On 23/09/2019 17:37, Robert LeBlanc wrote:

On Mon, Sep 23, 2019 at 4:14 AM Kenneth Waegeman
 wrote:

Hi all,

When syncing data with rsync, I'm often getting blocked slow requests,
which also block access to this path.


2019-09-23 11:25:49.477 7f4f401e8700 0 log_channel(cluster) log [WRN]
: slow request 31.895478 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:26:19.477 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 61.896079 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:27:19.478 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 121.897268 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:29:19.488 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 241.899467 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:33:19.680 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 482.087927 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:36:09.881 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 32.677511 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:36:39.881 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 62.678132 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:37:39.891 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 122.679273 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:39:39.892 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 242.684667 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:41:19.893 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 962.305681 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:43:39.923 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 482.712888 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:51:40.236 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 963.037049 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 11:57:20.308 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 1922.719287 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being 
exported

2019-09-23 12:07:40.621 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 1923.409501 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) cur

Re: [ceph-users] hanging slow requests: failed to authpin, subtree is being exported

2019-10-15 Thread Kenneth Waegeman

Hi Robert, all,


On 23/09/2019 17:37, Robert LeBlanc wrote:

On Mon, Sep 23, 2019 at 4:14 AM Kenneth Waegeman
 wrote:

Hi all,

When syncing data with rsync, I'm often getting blocked slow requests,
which also block access to this path.


2019-09-23 11:25:49.477 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 31.895478 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:26:19.477 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 61.896079 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:27:19.478 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 121.897268 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:29:19.488 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 241.899467 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:33:19.680 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 482.087927 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:36:09.881 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 32.677511 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:36:39.881 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 62.678132 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:37:39.891 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 122.679273 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:39:39.892 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 242.684667 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:41:19.893 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 962.305681 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:43:39.923 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 482.712888 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:51:40.236 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 963.037049 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:57:20.308 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 1922.719287 seconds old, received at 2019-09-23
11:25:17.598152: client_request(client.38352684:92684 lookup
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 12:07:40.621 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 1923.409501 seconds old, received at 2019-09-23
11:35:37.217113: client_request(client.38347357:111963 lookup
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 12:29:20.639 7f4f401e8700  0 log_channel(cluster) log [WRN]
: slow request 3843.057602 seconds old, r

Re: [ceph-users] hanging slow requests: failed to authpin, subtree is being exported

2019-09-23 Thread Robert LeBlanc
On Mon, Sep 23, 2019 at 4:14 AM Kenneth Waegeman
 wrote:
>
> Hi all,
>
> When syncing data with rsync, I'm often getting blocked slow requests,
> which also block access to this path.
>
> > 2019-09-23 11:25:49.477 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 31.895478 seconds old, received at 2019-09-23
> > 11:25:17.598152: client_request(client.38352684:92684 lookup
> > #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:26:19.477 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 61.896079 seconds old, received at 2019-09-23
> > 11:25:17.598152: client_request(client.38352684:92684 lookup
> > #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:27:19.478 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 121.897268 seconds old, received at 2019-09-23
> > 11:25:17.598152: client_request(client.38352684:92684 lookup
> > #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:29:19.488 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 241.899467 seconds old, received at 2019-09-23
> > 11:25:17.598152: client_request(client.38352684:92684 lookup
> > #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:33:19.680 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 482.087927 seconds old, received at 2019-09-23
> > 11:25:17.598152: client_request(client.38352684:92684 lookup
> > #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:36:09.881 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 32.677511 seconds old, received at 2019-09-23
> > 11:35:37.217113: client_request(client.38347357:111963 lookup
> > #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:36:39.881 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 62.678132 seconds old, received at 2019-09-23
> > 11:35:37.217113: client_request(client.38347357:111963 lookup
> > #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:37:39.891 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 122.679273 seconds old, received at 2019-09-23
> > 11:35:37.217113: client_request(client.38347357:111963 lookup
> > #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:39:39.892 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 242.684667 seconds old, received at 2019-09-23
> > 11:35:37.217113: client_request(client.38347357:111963 lookup
> > #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:41:19.893 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 962.305681 seconds old, received at 2019-09-23
> > 11:25:17.598152: client_request(client.38352684:92684 lookup
> > #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:43:39.923 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 482.712888 seconds old, received at 2019-09-23
> > 11:35:37.217113: client_request(client.38347357:111963 lookup
> > #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:51:40.236 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 963.037049 seconds old, received at 2019-09-23
> > 11:35:37.217113: client_request(client.38347357:111963 lookup
> > #0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 11:57:20.308 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 1922.719287 seconds old, received at 2019-09-23
> > 11:25:17.598152: client_request(client.38352684:92684 lookup
> > #0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0,
> > caller_gid=0{0,}) currently failed to authpin, subtree is being exported
> > 2019-09-23 12:07:40.621 7f4f401e8700  0 log_channel(cluster) log [WRN]
> > : slow request 1923.409501 seconds old, received at 2019-09-23
> > 11:35:37.217113: client_request(client.38347357:111963 lookup
> > #0x20005b0130c/testing 2019-

[ceph-users] hanging slow requests: failed to authpin, subtree is being exported

2019-09-23 Thread Kenneth Waegeman

Hi all,

When syncing data with rsync, I'm often getting blocked slow requests, 
which also block access to this path.


2019-09-23 11:25:49.477 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 31.895478 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.38352684:92684 lookup 
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:26:19.477 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 61.896079 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.38352684:92684 lookup 
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:27:19.478 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 121.897268 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.38352684:92684 lookup 
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:29:19.488 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 241.899467 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.38352684:92684 lookup 
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:33:19.680 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 482.087927 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.38352684:92684 lookup 
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:36:09.881 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 32.677511 seconds old, received at 2019-09-23 
11:35:37.217113: client_request(client.38347357:111963 lookup 
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:36:39.881 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 62.678132 seconds old, received at 2019-09-23 
11:35:37.217113: client_request(client.38347357:111963 lookup 
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:37:39.891 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 122.679273 seconds old, received at 2019-09-23 
11:35:37.217113: client_request(client.38347357:111963 lookup 
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:39:39.892 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 242.684667 seconds old, received at 2019-09-23 
11:35:37.217113: client_request(client.38347357:111963 lookup 
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:41:19.893 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 962.305681 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.38352684:92684 lookup 
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:43:39.923 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 482.712888 seconds old, received at 2019-09-23 
11:35:37.217113: client_request(client.38347357:111963 lookup 
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:51:40.236 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 963.037049 seconds old, received at 2019-09-23 
11:35:37.217113: client_request(client.38347357:111963 lookup 
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 11:57:20.308 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 1922.719287 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.38352684:92684 lookup 
#0x100152383ce/vsc42531 2019-09-23 11:25:17.598077 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 12:07:40.621 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 1923.409501 seconds old, received at 2019-09-23 
11:35:37.217113: client_request(client.38347357:111963 lookup 
#0x20005b0130c/testing 2019-09-23 11:35:37.217015 caller_uid=0, 
caller_gid=0{0,}) currently failed to authpin, subtree is being exported
2019-09-23 12:29:20.639 7f4f401e8700  0 log_channel(cluster) log [WRN] 
: slow request 3843.057602 seconds old, received at 2019-09-23 
11:25:17.598152: client_request(client.