Re: [Gluster-users] glusterfs 4.1.7 + nfs-ganesha 2.7.1 freeze during write

[Soumya Koduri]

On 2/8/19 11:53 AM, Soumya Koduri wrote:


On 2/8/19 3:20 AM, Maurits Lamers wrote:
Hi,


[2019-02-07 10:11:24.812606] E [MSGID: 104055] 
[glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
event_type(1) and gfid(yøêÙ
         Mz„–îSL4_@) failed
[2019-02-07 10:11:24.819376] E [MSGID: 104055] 
[glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
event_type(1) and gfid(eTnôEU«H.<Y4failed
[2019-02-07 10:11:24.833299] E [MSGID: 104055] 
[glfs-fops.c:4955:glfs_cbk_upcall_data] 0-gfapi: Synctak for Upcall 
event_type(1) and gfid(gÇLÁèFà»0bЯk) failed
[2019-02-07 10:25:01.642509] C 
[rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-2: 
server [node1]:49152 has not responded in the last 42 seconds, 
disconnecting.
[2019-02-07 10:25:01.642805] C 
[rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-1: 
server [node2]:49152 has not responded in the last 42 seconds, 
disconnecting.
[2019-02-07 10:25:01.642946] C 
[rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-4: 
server [node3]:49152 has not responded in the last 42 seconds, 
disconnecting.
[2019-02-07 10:25:02.643120] C 
[rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-3: 
server 127.0.1.1:49152 has not responded in the last 42 seconds, 
disconnecting.
[2019-02-07 10:25:02.643314] C 
[rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-gv0-client-0: 
server [node4]:49152 has not responded in the last 42 seconds, 
disconnecting.

Strange that synctask failed. Could you please turn off 
features.cache-invalidation volume option and check if the issue 
still persists.


Turning the cache invalidation option off seems to have solved the 
freeze. Still testing, but it looks promising.


If thats the case, please turn on cache invalidation option back and 
collect couple of stack traces (using gstack) when the system freezes 
again.

FYI - Have got a chance to reproduce and RCA the issue [1]. Posted fix 
for review in the upstream [2]

Thanks,
Soumya

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1693575
[2] https://review.gluster.org/22436


Thanks,
Soumya
cheers

Maurits

_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users