Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-28 Thread gjprabu
Hi Joseph, Again we are facing same issue. Please find the logs when the problem occurred. Dec 27 21:45:44 integ-hm5 kernel: (dlm_thread,46268,24):dlm_update_lvb:206 getting lvb from lockres for master node Dec 27 21:45:44 integ-hm5 kernel:

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-28 Thread Joseph Qi
So which process hangs? And which lockres it is waiting for? From the log I cannot get those information. On 2015/12/28 16:46, gjprabu wrote: > Hi Joseph, > > Again we are facing same issue. Please find the logs when the > problem occurred. > > Dec 27 21:45:44 integ-hm5 kernel:

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-28 Thread gjprabu
Joseph, Do you feel anything like kernel issue in below logs. After certain point of time no dlm logs found. Dec 27 22:21:22 integ-hm8 kernel: (ocfs2rec,68213,10):dlmconvert_remote:270 type=0, convert_type=-1, busy=0 Dec 27 22:21:22 integ-hm8 kernel:

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-28 Thread gjprabu
Yes, its got hanged all 5 nodes, after restart everything fine Regards Prabu On Mon, 28 Dec 2015 15:00:57 +0530 Joseph Qi joseph...@huawei.comwrote So which process hangs? And which lockres it is waiting for? >From the log I cannot get those information. On

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-28 Thread Joseph Qi
If system hangs, you should figure out which process as well as its stack before restarting the system. On 2015/12/28 20:16, gjprabu wrote: > Joseph, > > > Do you feel anything like kernel issue in below logs. After certain > point of time no dlm logs found. > > >

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread gjprabu
HI Joseph, Our current setup is having below details and DLM is now allowed (DLM allow). Do you suggest any other option to get more logs. debugfs.ocfs2 -l DLM off ( DLM allow) MSG off TCP off CONN off VOTE off DLM_DOMAIN off HB_BIO off BASTS off DLMFS off ERROR allow

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread Joseph Qi
Please also switch on BASTS and DLM_RECOVERY. On 2015/12/23 10:11, gjprabu wrote: > HI Joseph, > > Our current setup is having below details and DLM is now allowed > (DLM allow). Do you suggest any other option to get more logs. > > debugfs.ocfs2 -l > DLM off ( DLM allow) > MSG off

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread gjprabu
Hi Joseph, I have enabled requested and Is the DLM log will capture to analyze further. Also do we need to enable network side setting for allow max packets. debugfs.ocfs2 -l DLM allow MSG off TCP off CONN off VOTE off DLM_DOMAIN off HB_BIO off BASTS allow DLMFS off ERROR

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread Joseph Qi
So you mean the four nodes are manually rebooted? If so you must analyze messages before you rebooted. If there are not enough messages, you can switch on some messages. IMO, mostly hang problems are caused by DLM bug, so I suggest switch on DLM related log and reproduce. You can use debugfs.ocfs2

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread gjprabu
Ok, thanks On Wed, 23 Dec 2015 09:08:13 +0530 Joseph Qi joseph...@huawei.comwrote I don't think there is relation with packet size. Once reproduced, you can share the messages and I will try my best if free. On 2015/12/23 10:45, gjprabu wrote: Hi Joseph, I

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread Joseph Qi
I don't think there is relation with packet size. Once reproduced, you can share the messages and I will try my best if free. On 2015/12/23 10:45, gjprabu wrote: > Hi Joseph, > > I have enabled requested and Is the DLM log will capture to analyze > further. Also do we need to enable

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread gjprabu
Hi, Anybody please help me on this issue. Regards Prabu On Mon, 21 Dec 2015 15:16:49 +0530 gjprabu gjpr...@zohocorp.comwrote Dear Team, Ocfs2 clients are getting hang often and unusable. Please find the logs. Kindly provide the solution, it will be

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread Joseph Qi
Hi Prabu, >From the log you provided, I can only see that node 5 disconnected with node 2, 3, 1 and 4. It seemed that something wrong happened on the four nodes, and node 5 did recovery for them. After that, the four nodes joined again. On 2015/12/22 16:23, gjprabu wrote: > Hi, > >

Re: [Ocfs2-users] Ocfs2 clients hang

2015-12-22 Thread gjprabu
Hi Joseph, We are facing ocfs2 server hang problem frequently and suddenly 4 nodes going to hang stat expect 1 node. After reboot everything is come to normal, this behavior happend many times. Do we have any debug and fix for this issue. Regards Prabu On Tue, 22 Dec