Re: [Gluster-users] gluster stuck when trying to list a successful mount
OK. The issue was bad disk/mount on one of the machines. When I tried to: df -h from Node1 it stucked. So removed volume, peer, remounted the device. Now its ok. On Mon, Aug 7, 2017 at 5:18 PM, Ilan Schwartswrote: > Hi all, > My infrastructure is GlusterFS with 2 Nodes: > L137B-GlusterFS-Node1.L137B-root.com > L137B-GlusterFS-Node2.L137B-root.com > > > I have followed this guide: > http://www.itzgeek.com/how-tos/linux/centos-how-tos/install-and-configure-glusterfs-on-centos-7-rhel-7.html/2 > > I have created a device, formatted as ext4, mounted it. > The next step was to install gluster FS. > I installed glusterfs 3.10.3, created a volume and started it > [root@L137B-GlusterFS-Node2 someuser]# gluster volume info > > Volume Name: gv0 > Type: Replicate > Volume ID: a606f77b-c0df-427c-99ec-cee98b3ecd71 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: L137B-GlusterFS-Node1.L137B-root.com:/mnt/gluster/gv0 > Brick2: L137B-GlusterFS-Node2.L137B-root.com:/mnt/gluster/gv0 > Options Reconfigured: > transport.address-family: inet > nfs.disable: on > > > Now when I try to mount to Node1 from Node2, with the command: > mount -t glusterfs L137B-GlusterFS-Node2.L137B-root.com:/gv0 /mnt > > L137B-GlusterFS-Node2.L137B-root.com:/gv0 on /mnt type fuse.glusterfs > (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) > > > > it returns ok, no errors, but when i do: ls -la /mnt the console get stucked. > on dmesg -T i get (from a duplicate session on another console): > [Mon Aug 7 17:05:58 2017] fuse init (API version 7.22) > [Mon Aug 7 17:09:32 2017] INFO: task glusterfsd:3273 blocked for more > than 120 seconds. > [Mon Aug 7 17:09:32 2017] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Mon Aug 7 17:09:32 2017] glusterfsd D 880092e380b0 0 > 3273 1 0x0080 > [Mon Aug 7 17:09:32 2017] 8800a1e3bb70 0086 > 880139b7af10 8800a1e3bfd8 > [Mon Aug 7 17:09:32 2017] 8800a1e3bfd8 8800a1e3bfd8 > 880139b7af10 880092e380a8 > [Mon Aug 7 17:09:32 2017] 880092e380ac 880139b7af10 > 880092e380b0 > [Mon Aug 7 17:09:32 2017] Call Trace: > [Mon Aug 7 17:09:32 2017] [] > schedule_preempt_disabled+0x29/0x70 > [Mon Aug 7 17:09:32 2017] [] > __mutex_lock_slowpath+0xc5/0x1c0 > [Mon Aug 7 17:09:32 2017] [] ? unlazy_walk+0x87/0x140 > [Mon Aug 7 17:09:32 2017] [] mutex_lock+0x1f/0x2f > [Mon Aug 7 17:09:32 2017] [] lookup_slow+0x33/0xa7 > [Mon Aug 7 17:09:32 2017] [] link_path_walk+0x80f/0x8b0 > [Mon Aug 7 17:09:32 2017] [] ? __remove_hrtimer+0x46/0xe0 > [Mon Aug 7 17:09:32 2017] [] path_lookupat+0x6b/0x7a0 > [Mon Aug 7 17:09:32 2017] [] ? futex_wait+0x1a3/0x280 > [Mon Aug 7 17:09:32 2017] [] ? kmem_cache_alloc+0x35/0x1e0 > [Mon Aug 7 17:09:32 2017] [] ? getname_flags+0x4f/0x1a0 > [Mon Aug 7 17:09:32 2017] [] filename_lookup+0x2b/0xc0 > [Mon Aug 7 17:09:32 2017] [] user_path_at_empty+0x67/0xc0 > [Mon Aug 7 17:09:32 2017] [] ? futex_wake+0x80/0x160 > [Mon Aug 7 17:09:32 2017] [] user_path_at+0x11/0x20 > [Mon Aug 7 17:09:32 2017] [] vfs_fstatat+0x63/0xc0 > [Mon Aug 7 17:09:32 2017] [] SYSC_newlstat+0x31/0x60 > [Mon Aug 7 17:09:32 2017] [] ? SyS_futex+0x80/0x180 > [Mon Aug 7 17:09:32 2017] [] ? > __audit_syscall_exit+0x1e6/0x280 > [Mon Aug 7 17:09:32 2017] [] SyS_newlstat+0xe/0x10 > [Mon Aug 7 17:09:32 2017] [] > system_call_fastpath+0x16/0x1b > [Mon Aug 7 17:11:32 2017] INFO: task glusterfsd:3273 blocked for more > than 120 seconds. > [Mon Aug 7 17:11:32 2017] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Mon Aug 7 17:11:32 2017] glusterfsd D 880092e380b0 0 > 3273 1 0x0080 > [Mon Aug 7 17:11:32 2017] 8800a1e3bb70 0086 > 880139b7af10 8800a1e3bfd8 > [Mon Aug 7 17:11:32 2017] 8800a1e3bfd8 8800a1e3bfd8 > 880139b7af10 880092e380a8 > [Mon Aug 7 17:11:32 2017] 880092e380ac 880139b7af10 > 880092e380b0 > [Mon Aug 7 17:11:32 2017] Call Trace: > [Mon Aug 7 17:11:32 2017] [] > schedule_preempt_disabled+0x29/0x70 > [Mon Aug 7 17:11:32 2017] [] > __mutex_lock_slowpath+0xc5/0x1c0 > [Mon Aug 7 17:11:32 2017] [] ? unlazy_walk+0x87/0x140 > [Mon Aug 7 17:11:32 2017] [] mutex_lock+0x1f/0x2f > [Mon Aug 7 17:11:32 2017] [] lookup_slow+0x33/0xa7 > [Mon Aug 7 17:11:32 2017] [] link_path_walk+0x80f/0x8b0 > [Mon Aug 7 17:11:32 2017] [] ? __remove_hrtimer+0x46/0xe0 > [Mon Aug 7 17:11:32 2017] [] path_lookupat+0x6b/0x7a0 > [Mon Aug 7 17:11:32 2017] [] ? futex_wait+0x1a3/0x280 > [Mon Aug 7 17:11:32 2017] [] ? kmem_cache_alloc+0x35/0x1e0 > [Mon Aug 7 17:11:32 2017] [] ? getname_flags+0x4f/0x1a0 > [Mon Aug 7 17:11:32 2017] [] filename_lookup+0x2b/0xc0 > [Mon Aug 7 17:11:32 2017] []
[Gluster-users] gluster stuck when trying to list a successful mount
Hi all, My infrastructure is GlusterFS with 2 Nodes: L137B-GlusterFS-Node1.L137B-root.com L137B-GlusterFS-Node2.L137B-root.com I have followed this guide: http://www.itzgeek.com/how-tos/linux/centos-how-tos/install-and-configure-glusterfs-on-centos-7-rhel-7.html/2 I have created a device, formatted as ext4, mounted it. The next step was to install gluster FS. I installed glusterfs 3.10.3, created a volume and started it [root@L137B-GlusterFS-Node2 someuser]# gluster volume info Volume Name: gv0 Type: Replicate Volume ID: a606f77b-c0df-427c-99ec-cee98b3ecd71 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: L137B-GlusterFS-Node1.L137B-root.com:/mnt/gluster/gv0 Brick2: L137B-GlusterFS-Node2.L137B-root.com:/mnt/gluster/gv0 Options Reconfigured: transport.address-family: inet nfs.disable: on Now when I try to mount to Node1 from Node2, with the command: mount -t glusterfs L137B-GlusterFS-Node2.L137B-root.com:/gv0 /mnt L137B-GlusterFS-Node2.L137B-root.com:/gv0 on /mnt type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime) it returns ok, no errors, but when i do: ls -la /mnt the console get stucked. on dmesg -T i get (from a duplicate session on another console): [Mon Aug 7 17:05:58 2017] fuse init (API version 7.22) [Mon Aug 7 17:09:32 2017] INFO: task glusterfsd:3273 blocked for more than 120 seconds. [Mon Aug 7 17:09:32 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Aug 7 17:09:32 2017] glusterfsd D 880092e380b0 0 3273 1 0x0080 [Mon Aug 7 17:09:32 2017] 8800a1e3bb70 0086 880139b7af10 8800a1e3bfd8 [Mon Aug 7 17:09:32 2017] 8800a1e3bfd8 8800a1e3bfd8 880139b7af10 880092e380a8 [Mon Aug 7 17:09:32 2017] 880092e380ac 880139b7af10 880092e380b0 [Mon Aug 7 17:09:32 2017] Call Trace: [Mon Aug 7 17:09:32 2017] [] schedule_preempt_disabled+0x29/0x70 [Mon Aug 7 17:09:32 2017] [] __mutex_lock_slowpath+0xc5/0x1c0 [Mon Aug 7 17:09:32 2017] [] ? unlazy_walk+0x87/0x140 [Mon Aug 7 17:09:32 2017] [] mutex_lock+0x1f/0x2f [Mon Aug 7 17:09:32 2017] [] lookup_slow+0x33/0xa7 [Mon Aug 7 17:09:32 2017] [] link_path_walk+0x80f/0x8b0 [Mon Aug 7 17:09:32 2017] [] ? __remove_hrtimer+0x46/0xe0 [Mon Aug 7 17:09:32 2017] [] path_lookupat+0x6b/0x7a0 [Mon Aug 7 17:09:32 2017] [] ? futex_wait+0x1a3/0x280 [Mon Aug 7 17:09:32 2017] [] ? kmem_cache_alloc+0x35/0x1e0 [Mon Aug 7 17:09:32 2017] [] ? getname_flags+0x4f/0x1a0 [Mon Aug 7 17:09:32 2017] [] filename_lookup+0x2b/0xc0 [Mon Aug 7 17:09:32 2017] [] user_path_at_empty+0x67/0xc0 [Mon Aug 7 17:09:32 2017] [] ? futex_wake+0x80/0x160 [Mon Aug 7 17:09:32 2017] [] user_path_at+0x11/0x20 [Mon Aug 7 17:09:32 2017] [] vfs_fstatat+0x63/0xc0 [Mon Aug 7 17:09:32 2017] [] SYSC_newlstat+0x31/0x60 [Mon Aug 7 17:09:32 2017] [] ? SyS_futex+0x80/0x180 [Mon Aug 7 17:09:32 2017] [] ? __audit_syscall_exit+0x1e6/0x280 [Mon Aug 7 17:09:32 2017] [] SyS_newlstat+0xe/0x10 [Mon Aug 7 17:09:32 2017] [] system_call_fastpath+0x16/0x1b [Mon Aug 7 17:11:32 2017] INFO: task glusterfsd:3273 blocked for more than 120 seconds. [Mon Aug 7 17:11:32 2017] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Mon Aug 7 17:11:32 2017] glusterfsd D 880092e380b0 0 3273 1 0x0080 [Mon Aug 7 17:11:32 2017] 8800a1e3bb70 0086 880139b7af10 8800a1e3bfd8 [Mon Aug 7 17:11:32 2017] 8800a1e3bfd8 8800a1e3bfd8 880139b7af10 880092e380a8 [Mon Aug 7 17:11:32 2017] 880092e380ac 880139b7af10 880092e380b0 [Mon Aug 7 17:11:32 2017] Call Trace: [Mon Aug 7 17:11:32 2017] [] schedule_preempt_disabled+0x29/0x70 [Mon Aug 7 17:11:32 2017] [] __mutex_lock_slowpath+0xc5/0x1c0 [Mon Aug 7 17:11:32 2017] [] ? unlazy_walk+0x87/0x140 [Mon Aug 7 17:11:32 2017] [] mutex_lock+0x1f/0x2f [Mon Aug 7 17:11:32 2017] [] lookup_slow+0x33/0xa7 [Mon Aug 7 17:11:32 2017] [] link_path_walk+0x80f/0x8b0 [Mon Aug 7 17:11:32 2017] [] ? __remove_hrtimer+0x46/0xe0 [Mon Aug 7 17:11:32 2017] [] path_lookupat+0x6b/0x7a0 [Mon Aug 7 17:11:32 2017] [] ? futex_wait+0x1a3/0x280 [Mon Aug 7 17:11:32 2017] [] ? kmem_cache_alloc+0x35/0x1e0 [Mon Aug 7 17:11:32 2017] [] ? getname_flags+0x4f/0x1a0 [Mon Aug 7 17:11:32 2017] [] filename_lookup+0x2b/0xc0 [Mon Aug 7 17:11:32 2017] [] user_path_at_empty+0x67/0xc0 [Mon Aug 7 17:11:32 2017] [] ? futex_wake+0x80/0x160 [Mon Aug 7 17:11:32 2017] [] user_path_at+0x11/0x20 [Mon Aug 7 17:11:32 2017] [] vfs_fstatat+0x63/0xc0 [Mon Aug 7 17:11:32 2017] [] SYSC_newlstat+0x31/0x60 [Mon Aug 7 17:11:32 2017] [] ? SyS_futex+0x80/0x180 [Mon Aug 7 17:11:32 2017] [] ? __audit_syscall_exit+0x1e6/0x280 [Mon Aug 7 17:11:32 2017] [] SyS_newlstat+0xe/0x10 [Mon Aug 7 17:11:32