Re: [OpenAFS] AFS client hanged
Hi Qiulan, Can you please confirm that there are no other "INFO: task ... blocked for more than 120 seconds" messages? This is rather curious, since of the three tasks listed here, one is waiting trying to acquire the AFS_GLOCK, and the other two are sleeping (having dropped the AFS_GLOCK in order to sleep_. So if something is holding on to the AFS_GLOCK that keeps the afs_callback task from acquiring it, that task would be expected to also show up in the list of blocked tasks as output by the kernel. Thanks, Ben On Mon, Dec 16, 2019 at 02:53:43PM +0800, huangql wrote: > Dear all, > > Recently, I'm stuck with some AFS issues. > > AFS client hanged with the following log message. In this case, the AFS > instance blocked and jobs failed to access any files located in AFS. I have > to reboot the work node to recover service. > > Dec 6 15:03:18 bws0825 kernel: INFO: task afs_callback:19124 blocked for > more than 120 seconds. > Dec 6 15:03:18 bws0825 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Dec 6 15:03:18 bws0825 kernel: afs_callbackD 9860d826e180 0 > 19124 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: afs_callbackD 9860d826e180 0 > 19124 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: [] > schedule_preempt_disabled+0x29/0x70 > Dec 6 15:03:18 bws0825 kernel: [] > __mutex_lock_slowpath+0xc7/0x1d0 > Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f > Dec 6 15:03:18 bws0825 kernel: [] > SRXAFSCB_InitCallBackState+0x34/0x470 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_xdr_vector+0x57/0x90 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > SRXAFSCB_InitCallBackState3+0xe/0x10 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > RXAFSCB_ExecuteRequest+0x6f3/0x8a0 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > getnstimeofday64+0xe/0x30 > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_mutex_exit+0x29/0x40 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > rxi_ServerProc+0xcd/0x1e0 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] rx_ServerProc+0x87/0xe0 > [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > afs_RXCallBackServer+0x3d/0x50 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] afsd_thread+0x1e5/0x730 > [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] kthread+0xd1/0xe0 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: [] > ret_from_fork_nospec_begin+0x7/0x21 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: INFO: task afs_rxevent:19127 blocked for more > than 120 seconds. > Dec 6 15:03:18 bws0825 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Dec 6 15:03:18 bws0825 kernel: afs_rxevent D 9860cbbf6180 0 > 19127 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: [] ? > del_timer_sync+0x52/0x60 > Dec 6 15:03:18 bws0825 kernel: [] > schedule_preempt_disabled+0x29/0x70 > Dec 6 15:03:18 bws0825 kernel: [] > __mutex_lock_slowpath+0xc7/0x1d0 > Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f > Dec 6 15:03:18 bws0825 kernel: [] > afs_osi_TimedSleep+0x118/0x210 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? wake_up_state+0x20/0x20 > Dec 6 15:03:18 bws0825 kernel: [] afs_osi_Wait+0x98/0xd0 > [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > afs_rxevent_daemon+0x95/0x140 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] afsd_thread+0x636/0x730 > [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] kthread+0xd1/0xe0 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: [] > ret_from_fork_nospec_begin+0x7/0x21 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: [] > ret_from_fork_nospec_begin+0x7/0x21 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: INFO: task afs_checkserver:19870 blocked for > more than 120 seconds. > Dec 6 15:03:18 bws0825 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Dec 6 15:03:18 bws0825 kernel: afs_checkserver D 9860c7811040 0 > 19870 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: [] ? > del_timer_sync+0x52/0x60 > Dec 6 15:03:18 bws0825 kernel: [] > schedule_preempt_disabled+0x29/0x70 > Dec 6 15:03:18 bws0825 kernel: [] > __mutex_lock_slowpath+0xc7/0x1d0 > Dec 6 15:03:18 bws0825
Re: Re: [OpenAFS] AFS client hanged
> -Original Messages- > From: "Stephan Wiesand" > Sent Time: 2019-12-18 18:56:12 (Wednesday) > To: openafs-info@openafs.org > Cc: > Subject: Re: [OpenAFS] AFS client hanged > > Hi Andreas, > > > On 18. Dec 2019, at 10:48, Andreas Ladanyi wrote: > > > > Hi, > > > >>> kernel-2.6.32-696.20.1.el6.x86_64. After we upgrade to the new linux > >>> kernel and install the default openafs client version using yum(the > >>> version we used listed in the following), we have the hang issue. That's > >>> why I suspect the version compatibility. > >>> AFS clinet--sl7 : l.6.23 > >>> [root@bws0825 ~]# rpm -qa|grep openafs > >>> openafs-1.6-sl-client-1.6.23-289.sl7.x86_64 > >>> openafs-1.6-sl-authlibs-1.6.23-289.sl7.x86_64 > >>> openafs-1.6-sl-devel-1.6.23-289.sl7.x86_64 > >>> openafs-1.6-sl-module-tools-1.6.23-289.sl7.x86_64 > >>> openafs-1.6-sl-krb5-1.6.23-289.sl7.x86_64 > >>> openafs-1.6-sl-1.6.23-289.sl7.x86_64 > >>> openafs-1.6-sl-authlibs-devel-1.6.23-289.sl7.x86_64 > >>> kmod-openafs-1.6-sl-957-1.6.23-289.sl7.957.x86_64 > >>> > >>> AFS client-SL6: 1.6.23 > >>> openafs-krb5-1.6.23-289.sl6.x86_64 > >>> openafs-client-1.6.23-289.sl6.x86_64 > >>> openafs-1.6.23-289.sl6.x86_64 > >>> openafs-kpasswd-1.6.23-289.sl6.x86_64 > >>> openafs-module-tools-1.6.23-289.sl6.x86_64 > >>> openafs-kernel-source-1.6.23-289.sl6.x86_64 > >>> openafs-firstboot-1.6-1.sl6.noarch > >>> openafs-authlibs-1.6.23-289.sl6.x86_64 > >>> kmod-openafs-1.6.22.3-1.SL610.el6.noarch > >>> openafs-compat-1.6.23-289.sl6.x86_64 > >>> > > What i could see here is a version difference between kmod-openafs 1.6.22 > > and openafs-client 1.6.23 > > While 1.6.23 was a security update and yes, this looks kind of manual, it > shouldn't matter. > > > Does the issue appear on one client only or all clients which are upgraded ? The issue appear on many clients in the cluster but not at the same time. Every morning we check the status of compute nodes we got some of them hanged. Also, we found some clients failed to access some file located in AFS. The file looks well by "ls". But it failed to read it. I use strace command to trace the cp command to know the operation was stuck after reading several KB. It's weird. And from server side using tcpdump to know the package sent to server only version information without data. It seems like the client think the cache data is the latest one. Actually the cache is dirty . But the data access didn't trigger data fetch. 17:40:07.167220 IP lhws136.ihep.ac.cn.afs3-callback > 202.122.35.133.afs3-fileserver: rx version (29) 17:40:07.171580 IP bws0666.ihep.ac.cn.afs3-callback > 202.122.35.133.afs3-fileserver: rx version (29) 17:40:07.196307 IP acc-ap18.ihep.ac.cn.afs3-callback > 202.122.35.133.afs3-fileserver: rx data fs call setlock fid 536887265/30/43 (48) 17:40:07.222616 IP lxslc604.ihep.ac.cn.afs3-callback > 202.122.35.133.afs3-fileserver: rx data fs call setlock fid 536887265/30/43 (48) 17:40:07.240442 IP bws0420.ihep.ac.cn.afs3-callback > 202.122.35.133.afs3-fileserver: rx version (29) > > We run these packages on a lot of SL6 and SL7 systems, and the issue reported > here at least isn't common. We seem to have a project with a usage pattern > able to provoke hangs though. That has yet to be investigated. It was about > something like using zsh tab completion in a git repo... > > > ___ > OpenAFS-info mailing list > OpenAFS-info@openafs.org > https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] AFS client hanged
Hi Andreas, > On 18. Dec 2019, at 10:48, Andreas Ladanyi wrote: > > Hi, > >>> kernel-2.6.32-696.20.1.el6.x86_64. After we upgrade to the new linux kernel >>> and install the default openafs client version using yum(the version we >>> used listed in the following), we have the hang issue. That's why I suspect >>> the version compatibility. >>> AFS clinet--sl7 : l.6.23 >>> [root@bws0825 ~]# rpm -qa|grep openafs >>> openafs-1.6-sl-client-1.6.23-289.sl7.x86_64 >>> openafs-1.6-sl-authlibs-1.6.23-289.sl7.x86_64 >>> openafs-1.6-sl-devel-1.6.23-289.sl7.x86_64 >>> openafs-1.6-sl-module-tools-1.6.23-289.sl7.x86_64 >>> openafs-1.6-sl-krb5-1.6.23-289.sl7.x86_64 >>> openafs-1.6-sl-1.6.23-289.sl7.x86_64 >>> openafs-1.6-sl-authlibs-devel-1.6.23-289.sl7.x86_64 >>> kmod-openafs-1.6-sl-957-1.6.23-289.sl7.957.x86_64 >>> >>> AFS client-SL6: 1.6.23 >>> openafs-krb5-1.6.23-289.sl6.x86_64 >>> openafs-client-1.6.23-289.sl6.x86_64 >>> openafs-1.6.23-289.sl6.x86_64 >>> openafs-kpasswd-1.6.23-289.sl6.x86_64 >>> openafs-module-tools-1.6.23-289.sl6.x86_64 >>> openafs-kernel-source-1.6.23-289.sl6.x86_64 >>> openafs-firstboot-1.6-1.sl6.noarch >>> openafs-authlibs-1.6.23-289.sl6.x86_64 >>> kmod-openafs-1.6.22.3-1.SL610.el6.noarch >>> openafs-compat-1.6.23-289.sl6.x86_64 >>> > What i could see here is a version difference between kmod-openafs 1.6.22 and > openafs-client 1.6.23 While 1.6.23 was a security update and yes, this looks kind of manual, it shouldn't matter. > Does the issue appear on one client only or all clients which are upgraded ? We run these packages on a lot of SL6 and SL7 systems, and the issue reported here at least isn't common. We seem to have a project with a usage pattern able to provoke hangs though. That has yet to be investigated. It was about something like using zsh tab completion in a git repo... ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] AFS client hanged
Hi, >> kernel-2.6.32-696.20.1.el6.x86_64. After we upgrade to the new linux >> kernel and install the default openafs client version using yum(the version >> we used listed in the following), we have the hang issue. That's why I >> suspect the version compatibility. >> >> AFS clinet--sl7 : l.6.23 >> [root@bws0825 ~]# rpm -qa|grep openafs >> openafs-1.6-sl-client-1.6.23-289.sl7.x86_64 >> openafs-1.6-sl-authlibs-1.6.23-289.sl7.x86_64 >> openafs-1.6-sl-devel-1.6.23-289.sl7.x86_64 >> openafs-1.6-sl-module-tools-1.6.23-289.sl7.x86_64 >> openafs-1.6-sl-krb5-1.6.23-289.sl7.x86_64 >> openafs-1.6-sl-1.6.23-289.sl7.x86_64 >> openafs-1.6-sl-authlibs-devel-1.6.23-289.sl7.x86_64 >> kmod-openafs-1.6-sl-957-1.6.23-289.sl7.957.x86_64 >> >> AFS client-SL6: 1.6.23 >> openafs-krb5-1.6.23-289.sl6.x86_64 >> openafs-client-1.6.23-289.sl6.x86_64 >> openafs-1.6.23-289.sl6.x86_64 >> openafs-kpasswd-1.6.23-289.sl6.x86_64 >> openafs-module-tools-1.6.23-289.sl6.x86_64 >> openafs-kernel-source-1.6.23-289.sl6.x86_64 >> openafs-firstboot-1.6-1.sl6.noarch >> openafs-authlibs-1.6.23-289.sl6.x86_64 >> kmod-openafs-1.6.22.3-1.SL610.el6.noarch >> openafs-compat-1.6.23-289.sl6.x86_64 >> What i could see here is a version difference between kmod-openafs 1.6.22 and openafs-client 1.6.23 Does the issue appear on one client only or all clients which are upgraded ?
Re: Re: [OpenAFS] AFS client hanged
-原始邮件- 发件人:"Andreas Ladanyi" 发送时间:2019-12-16 18:34:10 (星期一) 收件人: huangql , openafs-info 抄送: 主题: Re: [OpenAFS] AFS client hanged Hi , Dear all, Recently, I'm stuck with some AFS issues. AFS client hanged with the following log message. In this case, the AFS instance blocked and jobs failed to access any files located in AFS. I have to reboot the work node to recover service. Dec 6 15:03:18 bws0825 kernel: INFO: task afs_callback:19124 blocked for more than 120 seconds. Dec 6 15:03:18 bws0825 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 6 15:03:18 bws0825 kernel: afs_callbackD 9860d826e180 0 19124 2 0x Dec 6 15:03:18 bws0825 kernel: Call Trace: Dec 6 15:03:18 bws0825 kernel: afs_callbackD 9860d826e180 0 19124 2 0x Dec 6 15:03:18 bws0825 kernel: Call Trace: Dec 6 15:03:18 bws0825 kernel: [] schedule_preempt_disabled+0x29/0x70 Dec 6 15:03:18 bws0825 kernel: [] __mutex_lock_slowpath+0xc7/0x1d0 Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f Dec 6 15:03:18 bws0825 kernel: [] SRXAFSCB_InitCallBackState+0x34/0x470 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? afs_xdr_vector+0x57/0x90 [openafs] Dec 6 15:03:18 bws0825 kernel: [] SRXAFSCB_InitCallBackState3+0xe/0x10 [openafs] Dec 6 15:03:18 bws0825 kernel: [] RXAFSCB_ExecuteRequest+0x6f3/0x8a0 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? getnstimeofday64+0xe/0x30 Dec 6 15:03:18 bws0825 kernel: [] ? afs_mutex_exit+0x29/0x40 [openafs] Dec 6 15:03:18 bws0825 kernel: [] rxi_ServerProc+0xcd/0x1e0 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? afs_shutdown_pagecopy+0x20/0x20 [openafs] Dec 6 15:03:18 bws0825 kernel: [] rx_ServerProc+0x87/0xe0 [openafs] Dec 6 15:03:18 bws0825 kernel: [] afs_RXCallBackServer+0x3d/0x50 [openafs] Dec 6 15:03:18 bws0825 kernel: [] afsd_thread+0x1e5/0x730 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? afs_shutdown_pagecopy+0x20/0x20 [openafs] Dec 6 15:03:18 bws0825 kernel: [] kthread+0xd1/0xe0 Dec 6 15:03:18 bws0825 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 6 15:03:18 bws0825 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Dec 6 15:03:18 bws0825 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 6 15:03:18 bws0825 kernel: INFO: task afs_rxevent:19127 blocked for more than 120 seconds. Dec 6 15:03:18 bws0825 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 6 15:03:18 bws0825 kernel: afs_rxevent D 9860cbbf6180 0 19127 2 0x Dec 6 15:03:18 bws0825 kernel: Call Trace: Dec 6 15:03:18 bws0825 kernel: [] ? del_timer_sync+0x52/0x60 Dec 6 15:03:18 bws0825 kernel: [] schedule_preempt_disabled+0x29/0x70 Dec 6 15:03:18 bws0825 kernel: [] __mutex_lock_slowpath+0xc7/0x1d0 Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f Dec 6 15:03:18 bws0825 kernel: [] afs_osi_TimedSleep+0x118/0x210 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? wake_up_state+0x20/0x20 Dec 6 15:03:18 bws0825 kernel: [] afs_osi_Wait+0x98/0xd0 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? afs_shutdown_pagecopy+0x20/0x20 [openafs] Dec 6 15:03:18 bws0825 kernel: [] afs_rxevent_daemon+0x95/0x140 [openafs] Dec 6 15:03:18 bws0825 kernel: [] afsd_thread+0x636/0x730 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? afs_shutdown_pagecopy+0x20/0x20 [openafs] Dec 6 15:03:18 bws0825 kernel: [] kthread+0xd1/0xe0 Dec 6 15:03:18 bws0825 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 6 15:03:18 bws0825 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Dec 6 15:03:18 bws0825 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 6 15:03:18 bws0825 kernel: [] ret_from_fork_nospec_begin+0x7/0x21 Dec 6 15:03:18 bws0825 kernel: [] ? insert_kthread_work+0x40/0x40 Dec 6 15:03:18 bws0825 kernel: INFO: task afs_checkserver:19870 blocked for more than 120 seconds. Dec 6 15:03:18 bws0825 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 6 15:03:18 bws0825 kernel: afs_checkserver D 9860c7811040 0 19870 2 0x Dec 6 15:03:18 bws0825 kernel: Call Trace: Dec 6 15:03:18 bws0825 kernel: [] ? del_timer_sync+0x52/0x60 Dec 6 15:03:18 bws0825 kernel: [] schedule_preempt_disabled+0x29/0x70 Dec 6 15:03:18 bws0825 kernel: [] __mutex_lock_slowpath+0xc7/0x1d0 Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f Dec 6 15:03:18 bws0825 kernel: [] afs_osi_TimedSleep+0x118/0x210 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? wake_up_state+0x20/0x20 Dec 6 15:03:18 bws0825 kernel: [] afs_osi_Wait+0x98/0xd0 [openafs] Dec 6 15:03:18 bws0825 kernel: [] afs_CheckServerDaemon+0x118/0x1a0 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? afs_shutdown_pagecopy+0x20/0x20 [openafs] Dec 6 15:03:18 bws0825 kernel: [] afsd_thread+0x470/0x730 [openafs] Dec 6 15:03:18 bws0825 kernel: [] ? afs_shutdown_pagecopy+0x20/0x20 [openafs] Dec 6 15:03:18 bws0825 kernel: []
Re: [OpenAFS] AFS client hanged
Hi , > Dear all, > > Recently, I'm stuck with some AFS issues. > > AFS client hanged with the following log message. In this case, > the AFS instance blocked and jobs failed to access any files > located in AFS. I have to reboot the work node to recover service. > > Dec 6 15:03:18 bws0825 kernel: INFO: task afs_callback:19124 blocked for > more than 120 seconds. > Dec 6 15:03:18 bws0825 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Dec 6 15:03:18 bws0825 kernel: afs_callbackD 9860d826e180 0 > 19124 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: afs_callbackD 9860d826e180 0 > 19124 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: [] > schedule_preempt_disabled+0x29/0x70 > Dec 6 15:03:18 bws0825 kernel: [] > __mutex_lock_slowpath+0xc7/0x1d0 > Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f > Dec 6 15:03:18 bws0825 kernel: [] > SRXAFSCB_InitCallBackState+0x34/0x470 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_xdr_vector+0x57/0x90 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > SRXAFSCB_InitCallBackState3+0xe/0x10 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > RXAFSCB_ExecuteRequest+0x6f3/0x8a0 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > getnstimeofday64+0xe/0x30 > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_mutex_exit+0x29/0x40 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > rxi_ServerProc+0xcd/0x1e0 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > rx_ServerProc+0x87/0xe0 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > afs_RXCallBackServer+0x3d/0x50 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > afsd_thread+0x1e5/0x730 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] kthread+0xd1/0xe0 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: [] > ret_from_fork_nospec_begin+0x7/0x21 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: INFO: task afs_rxevent:19127 blocked for > more than 120 seconds. > Dec 6 15:03:18 bws0825 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Dec 6 15:03:18 bws0825 kernel: afs_rxevent D 9860cbbf6180 0 > 19127 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: [] ? > del_timer_sync+0x52/0x60 > Dec 6 15:03:18 bws0825 kernel: [] > schedule_preempt_disabled+0x29/0x70 > Dec 6 15:03:18 bws0825 kernel: [] > __mutex_lock_slowpath+0xc7/0x1d0 > Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f > Dec 6 15:03:18 bws0825 kernel: [] > afs_osi_TimedSleep+0x118/0x210 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > wake_up_state+0x20/0x20 > Dec 6 15:03:18 bws0825 kernel: [] > afs_osi_Wait+0x98/0xd0 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > afs_rxevent_daemon+0x95/0x140 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > afsd_thread+0x636/0x730 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > afs_shutdown_pagecopy+0x20/0x20 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] kthread+0xd1/0xe0 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: [] > ret_from_fork_nospec_begin+0x7/0x21 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: [] > ret_from_fork_nospec_begin+0x7/0x21 > Dec 6 15:03:18 bws0825 kernel: [] ? > insert_kthread_work+0x40/0x40 > Dec 6 15:03:18 bws0825 kernel: INFO: task afs_checkserver:19870 blocked > for more than 120 seconds. > Dec 6 15:03:18 bws0825 kernel: "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Dec 6 15:03:18 bws0825 kernel: afs_checkserver D 9860c7811040 0 > 19870 2 0x > Dec 6 15:03:18 bws0825 kernel: Call Trace: > Dec 6 15:03:18 bws0825 kernel: [] ? > del_timer_sync+0x52/0x60 > Dec 6 15:03:18 bws0825 kernel: [] > schedule_preempt_disabled+0x29/0x70 > Dec 6 15:03:18 bws0825 kernel: [] > __mutex_lock_slowpath+0xc7/0x1d0 > Dec 6 15:03:18 bws0825 kernel: [] mutex_lock+0x1f/0x2f > Dec 6 15:03:18 bws0825 kernel: [] > afs_osi_TimedSleep+0x118/0x210 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] ? > wake_up_state+0x20/0x20 > Dec 6 15:03:18 bws0825 kernel: [] > afs_osi_Wait+0x98/0xd0 [openafs] > Dec 6 15:03:18 bws0825 kernel: [] > afs_CheckServerDaemon+0x118/0x1a0
