Re: [Gluster-users] Failed to get quota limits
Hi Hari, Sorry for not providing you more details from the start. Here below you will find all the relevant log entries and info. Regarding the quota.conf file I have found one for my volume but it is a binary file. Is it supposed to be binary or text? Regards, M. *** gluster volume info myvolume *** Volume Name: myvolume Type: Replicate Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: gfs1a.domain.local:/data/myvolume/brick Brick2: gfs1b.domain.local:/data/myvolume/brick Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) Options Reconfigured: server.event-threads: 4 client.event-threads: 4 performance.readdir-ahead: on nfs.disable: on features.quota: on features.inode-quota: on features.quota-deem-statfs: on transport.address-family: inet *** /var/log/glusterfs/glusterd.log *** [2018-02-13 08:16:09.929568] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ [2018-02-13 08:16:28.596527] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvolume [2018-02-13 08:16:28.601097] I [MSGID: 106419] [glusterd-utils.c:6110:glusterd_add_inode_size_to_dict] 0-management: the brick on data/myvolume (zfs) uses dynamic inode sizes *** /var/log/glusterfs/cmd_history.log *** [2018-02-13 08:16:28.605478] : volume status myvolume detail : SUCCESS *** /var/log/glusterfs/quota-mount-myvolume.log *** [2018-02-13 08:16:09.934117] I [MSGID: 100030] [glusterfsd.c:2503:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.7 (args: /usr/sbin/glusterfs --volfile-server localhost --volfile-id myvolume -l /var/log/glusterfs/quota-mount-myvolume.log -p /var/run/gluster/myvolume_quota_list.pid --client-pid -5 /var/run/gluster/myvolume_quota_list/) [2018-02-13 08:16:09.940432] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-02-13 08:16:09.940491] E [socket.c:2327:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused); disconnecting socket [2018-02-13 08:16:09.940519] I [glusterfsd-mgmt.c:2134:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: localhost [2018-02-13 08:16:13.943827] I [afr.c:94:fix_quorum_options] 0-myvolume-replicate-0: reindeer: incoming qtype = none [2018-02-13 08:16:13.943857] I [afr.c:116:fix_quorum_options] 0-myvolume-replicate-0: reindeer: quorum_count = 2147483647 [2018-02-13 08:16:13.945194] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2018-02-13 08:16:13.945237] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4 [2018-02-13 08:16:13.945247] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3 [2018-02-13 08:16:13.945941] W [MSGID: 101174] [graph.c:361:_log_if_unknown_option] 0-myvolume-readdir-ahead: option 'parallel-readdir' is not recognized [2018-02-13 08:16:13.946342] I [MSGID: 114020] [client.c:2352:notify] 0-myvolume-client-0: parent translators are ready, attempting connect on transport [2018-02-13 08:16:13.946789] I [MSGID: 114020] [client.c:2352:notify] 0-myvolume-client-1: parent translators are ready, attempting connect on transport [2018-02-13 08:16:13.947151] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-myvolume-client-0: changing port to 49153 (from 0) [2018-02-13 08:16:13.947846] I [MSGID: 114057] [client-handshake.c:1451:select_server_supported_programs] 0-myvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-13 08:16:13.948073] I [MSGID: 114020] [client.c:2352:notify] 0-myvolume-client-2: parent translators are ready, attempting connect on transport [2018-02-13 08:16:13.948579] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-myvolume-client-1: changing port to 49154 (from 0) [2018-02-13 08:16:13.948699] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-0: Connected to myvolume-client-0, attached to remote volume '/data/myvolume/brick'. [2018-02-13 08:16:13.948747] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-0: Server and Client lk-version numbers are not same, reopening the fds [2018-02-13 08:16:13.948842] I [MSGID: 108005] [afr-common.c:4813:afr_notify] 0-myvolume-replicate-0: Subvolume 'myvolume-client-0' came back up; going online. Final graph: +--+ 1: volume myvolume-client-0 2: type protocol/client 3: option opversion 31004 4: option clnt-lk-version 1 5: option volfile-checksum 0 6: option volfile-key myvolume 7: option client-version 3.10.7 8: option process-uuid gfs1a-14348-2018/02/13-
Re: [Gluster-users] Failed to get quota limits
Hi, A part of the log won't be enough to debug the issue. Need the whole log messages till date. You can send it as attachments. Yes the quota.conf is a binary file. And I need the volume status output too. On Tue, Feb 13, 2018 at 1:56 PM, mabi wrote: > Hi Hari, > Sorry for not providing you more details from the start. Here below you will > find all the relevant log entries and info. Regarding the quota.conf file I > have found one for my volume but it is a binary file. Is it supposed to be > binary or text? > Regards, > M. > > *** gluster volume info myvolume *** > > Volume Name: myvolume > Type: Replicate > Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: gfs1a.domain.local:/data/myvolume/brick > Brick2: gfs1b.domain.local:/data/myvolume/brick > Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) > Options Reconfigured: > server.event-threads: 4 > client.event-threads: 4 > performance.readdir-ahead: on > nfs.disable: on > features.quota: on > features.inode-quota: on > features.quota-deem-statfs: on > transport.address-family: inet > > > *** /var/log/glusterfs/glusterd.log *** > > [2018-02-13 08:16:09.929568] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] > 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ > [2018-02-13 08:16:28.596527] I [MSGID: 106499] > [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume myvolume > [2018-02-13 08:16:28.601097] I [MSGID: 106419] > [glusterd-utils.c:6110:glusterd_add_inode_size_to_dict] 0-management: the > brick on data/myvolume (zfs) uses dynamic inode sizes > > > *** /var/log/glusterfs/cmd_history.log *** > > [2018-02-13 08:16:28.605478] : volume status myvolume detail : SUCCESS > > > *** /var/log/glusterfs/quota-mount-myvolume.log *** > > [2018-02-13 08:16:09.934117] I [MSGID: 100030] [glusterfsd.c:2503:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.7 > (args: /usr/sbin/glusterfs --volfile-server localhost --volfile-id myvolume > -l /var/log/glusterfs/quota-mount-myvolume.log -p > /var/run/gluster/myvolume_quota_list.pid --client-pid -5 > /var/run/gluster/myvolume_quota_list/) > [2018-02-13 08:16:09.940432] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 1 > [2018-02-13 08:16:09.940491] E [socket.c:2327:socket_connect_finish] > 0-glusterfs: connection to ::1:24007 failed (Connection refused); > disconnecting socket > [2018-02-13 08:16:09.940519] I [glusterfsd-mgmt.c:2134:mgmt_rpc_notify] > 0-glusterfsd-mgmt: disconnected from remote-host: localhost > [2018-02-13 08:16:13.943827] I [afr.c:94:fix_quorum_options] > 0-myvolume-replicate-0: reindeer: incoming qtype = none > [2018-02-13 08:16:13.943857] I [afr.c:116:fix_quorum_options] > 0-myvolume-replicate-0: reindeer: quorum_count = 2147483647 > [2018-02-13 08:16:13.945194] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 2 > [2018-02-13 08:16:13.945237] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 4 > [2018-02-13 08:16:13.945247] I [MSGID: 101190] > [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with > index 3 > [2018-02-13 08:16:13.945941] W [MSGID: 101174] > [graph.c:361:_log_if_unknown_option] 0-myvolume-readdir-ahead: option > 'parallel-readdir' is not recognized > [2018-02-13 08:16:13.946342] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-0: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.946789] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-1: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.947151] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-0: changing port to 49153 (from 0) > [2018-02-13 08:16:13.947846] I [MSGID: 114057] > [client-handshake.c:1451:select_server_supported_programs] > 0-myvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version > (330) > [2018-02-13 08:16:13.948073] I [MSGID: 114020] [client.c:2352:notify] > 0-myvolume-client-2: parent translators are ready, attempting connect on > transport > [2018-02-13 08:16:13.948579] I [rpc-clnt.c:2000:rpc_clnt_reconfig] > 0-myvolume-client-1: changing port to 49154 (from 0) > [2018-02-13 08:16:13.948699] I [MSGID: 114046] > [client-handshake.c:1216:client_setvolume_cbk] 0-myvolume-client-0: > Connected to myvolume-client-0, attached to remote volume > '/data/myvolume/brick'. > [2018-02-13 08:16:13.948747] I [MSGID: 114047] > [client-handshake.c:1227:client_setvolume_cbk] 0-myvolume-client-0: Server > and Client lk-version numbers are not same, reopening the fds > [2018-02-13 08:16:13.948842] I [MSGID: 108005] > [afr-common.c:4813:afr_notify] 0-myvolume-replicate-0: Subvolume > 'myvolume-
Re: [Gluster-users] Failed to get quota limits
Hi Hari, Sure no problem, I will send you in a minute another mail where you can download all the relevant log files including the quota.conf binary file. Let me know if you need anything else. In the mean time here below is the output of a volume status. Best regards, M. Status of volume: myvolume Gluster process TCP Port RDMA Port Online Pid -- Brick gfs1a.domain.local:/data/myvolume /brick 49153 0 Y 3214 Brick gfs1b.domain.local:/data/myvolume /brick 49154 0 Y 3256 Brick gfs1c.domain.local:/srv/glusterf s/myvolume/brick 49153 0 Y 515 Self-heal Daemon on localhost N/A N/AY 3186 Quota Daemon on localhost N/A N/AY 3195 Self-heal Daemon on gfs1b.domain.local N/A N/AY 3217 Quota Daemon on gfs1b.domain.local N/A N/AY 3229 Self-heal Daemon on gfs1c.domain.local N/A N/AY 486 Quota Daemon on gfs1c.domain.local N/A N/AY 495 Task Status of Volume myvolume -- There are no active volume tasks Original Message On February 13, 2018 10:09 AM, Hari Gowtham wrote: >Hi, > > A part of the log won't be enough to debug the issue. > Need the whole log messages till date. > You can send it as attachments. > > Yes the quota.conf is a binary file. > > And I need the volume status output too. > > On Tue, Feb 13, 2018 at 1:56 PM, mabi m...@protonmail.ch wrote: >>Hi Hari, >> Sorry for not providing you more details from the start. Here below you will >> find all the relevant log entries and info. Regarding the quota.conf file I >> have found one for my volume but it is a binary file. Is it supposed to be >> binary or text? >> Regards, >> M. >>*** gluster volume info myvolume *** >>Volume Name: myvolume >> Type: Replicate >> Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: gfs1a.domain.local:/data/myvolume/brick >> Brick2: gfs1b.domain.local:/data/myvolume/brick >> Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) >> Options Reconfigured: >> server.event-threads: 4 >> client.event-threads: 4 >> performance.readdir-ahead: on >> nfs.disable: on >> features.quota: on >> features.inode-quota: on >> features.quota-deem-statfs: on >> transport.address-family: inet >>*** /var/log/glusterfs/glusterd.log *** >>[2018-02-13 08:16:09.929568] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] >> 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ >> [2018-02-13 08:16:28.596527] I [MSGID: 106499] >> [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: >> Received status volume req for volume myvolume >> [2018-02-13 08:16:28.601097] I [MSGID: 106419] >> [glusterd-utils.c:6110:glusterd_add_inode_size_to_dict] 0-management: the >> brick on data/myvolume (zfs) uses dynamic inode sizes >>*** /var/log/glusterfs/cmd_history.log *** >>[2018-02-13 08:16:28.605478] : volume status myvolume detail : SUCCESS >>*** /var/log/glusterfs/quota-mount-myvolume.log *** >>[2018-02-13 08:16:09.934117] I [MSGID: 100030] [glusterfsd.c:2503:main] >> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.7 >> (args: /usr/sbin/glusterfs --volfile-server localhost --volfile-id myvolume >> -l /var/log/glusterfs/quota-mount-myvolume.log -p >> /var/run/gluster/myvolume_quota_list.pid --client-pid -5 >> /var/run/gluster/myvolume_quota_list/) >> [2018-02-13 08:16:09.940432] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 1 >> [2018-02-13 08:16:09.940491] E [socket.c:2327:socket_connect_finish] >> 0-glusterfs: connection to ::1:24007 failed (Connection refused); >> disconnecting socket >> [2018-02-13 08:16:09.940519] I [glusterfsd-mgmt.c:2134:mgmt_rpc_notify] >> 0-glusterfsd-mgmt: disconnected from remote-host: localhost >> [2018-02-13 08:16:13.943827] I [afr.c:94:fix_quorum_options] >> 0-myvolume-replicate-0: reindeer: incoming qtype = none >> [2018-02-13 08:16:13.943857] I [afr.c:116:fix_quorum_options] >> 0-myvolume-replicate-0: reindeer: quorum_count = 2147483647 >> [2018-02-13 08:16:13.945194] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 2 >> [2018-02-13 08:16:13.945237] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 4 >> [2018-02-13 08:16:13.945247] I [MSGID: 101190] >> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with >> index 3 >> [2018-02-13 08:16:13.945941] W [MSGID:
Re: [Gluster-users] Failed to get quota limits
Thank you for your answer. This problem seem to have started since last week, so should I also send you the same log files but for last week? I think logrotate rotates them on a weekly basis. The only two quota commands we use are the following: gluster volume quota myvolume limit-usage /directory 10GB gluster volume quota myvolume list basically to set a new quota or to list the current quotas. The quota list was working in the past yes but we already had a similar issue where the quotas disappeared last August 2017: http://lists.gluster.org/pipermail/gluster-users/2017-August/031946.html In the mean time the only thing we did is to upgrade from 3.8 to 3.10. There are actually no errors to be seen using any gluster commands. The "quota myvolume list" returns simply nothing. In order to lookup the directories should I run a "stat" on them? and if yes should I do that on a client through the fuse mount? Original Message On February 13, 2018 10:58 AM, Hari Gowtham wrote: >The log provided are from 11th, you have seen the issue a while before > that itself. > > The logs help us to know if something has actually went wrong. > once something goes wrong the output might get affected and i need to know > what > went wrong. Which means i need the log from the beginning. > > and i need to know a few more things, > Was the quota list command was working as expected at the beginning? > If yes, what were the commands issued, before you noticed this problem. > Is there any other error that you see other than this? > > And can you try looking up the directories the limits are set on and > check if that fixes the error? > >> Original Message >> On February 13, 2018 10:44 AM, mabi m...@protonmail.ch wrote: >>>Hi Hari, >>>Sure no problem, I will send you in a minute another mail where you can >>>download all the relevant log files including the quota.conf binary file. >>>Let me know if you need anything else. In the mean time here below is the >>>output of a volume status. >>>Best regards, >>> M. >>>Status of volume: myvolume >>> Gluster process TCP Port RDMA Port Online Pid >>>Brick gfs1a.domain.local:/data/myvolume >>> /brick 49153 0 Y 3214 >>> Brick gfs1b.domain.local:/data/myvolume >>> /brick 49154 0 Y 3256 >>> Brick gfs1c.domain.local:/srv/glusterf >>> s/myvolume/brick 49153 0 Y 515 >>> Self-heal Daemon on localhost N/A N/AY >>> 3186 >>> Quota Daemon on localhost N/A N/AY >>> 3195 >>> Self-heal Daemon on gfs1b.domain.local N/A N/AY 3217 >>> Quota Daemon on gfs1b.domain.local N/A N/AY 3229 >>> Self-heal Daemon on gfs1c.domain.local N/A N/AY 486 >>> Quota Daemon on gfs1c.domain.local N/A N/AY 495 >>>Task Status of Volume myvolume >>>There are no active volume tasks >>> Original Message >>> On February 13, 2018 10:09 AM, Hari Gowtham hgowt...@redhat.com wrote: Hi, A part of the log won't be enough to debug the issue. Need the whole log messages till date. You can send it as attachments. Yes the quota.conf is a binary file. And I need the volume status output too. On Tue, Feb 13, 2018 at 1:56 PM, mabi m...@protonmail.ch wrote: >Hi Hari, > Sorry for not providing you more details from the start. Here below you > will > find all the relevant log entries and info. Regarding the quota.conf file > I > have found one for my volume but it is a binary file. Is it supposed to be > binary or text? > Regards, > M. > *** gluster volume info myvolume *** > Volume Name: myvolume > Type: Replicate > Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: gfs1a.domain.local:/data/myvolume/brick > Brick2: gfs1b.domain.local:/data/myvolume/brick > Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) > Options Reconfigured: > server.event-threads: 4 > client.event-threads: 4 > performance.readdir-ahead: on > nfs.disable: on > features.quota: on > features.inode-quota: on > features.quota-deem-statfs: on > transport.address-family: inet > *** /var/log/glusterfs/glusterd.log *** > [2018-02-13 08:16:09.929568] E [MSGID: 101042] > [compat.c:569:gf_umount_lazy] > 0-management: Lazy unmount of /var/run/gluster/myvolume_quota_list/ > [2018-02-13 08:16:28.596527] I [MSGID: 106499] > [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: > Received status volume req for volume myvolume > [2018-02-13 08:16:28.601097] I [MSGID: 106419
Re: [Gluster-users] Failed to get quota limits
Yes, I need the log files in that duration, the log rotated file after hitting the issue aren't necessary, but the ones before hitting the issues are needed (not just when you hit it, the ones even before you hit it). Yes, you have to do a stat from the client through fuse mount. On Tue, Feb 13, 2018 at 3:56 PM, mabi wrote: > Thank you for your answer. This problem seem to have started since last week, > so should I also send you the same log files but for last week? I think > logrotate rotates them on a weekly basis. > > The only two quota commands we use are the following: > > gluster volume quota myvolume limit-usage /directory 10GB > gluster volume quota myvolume list > > basically to set a new quota or to list the current quotas. The quota list > was working in the past yes but we already had a similar issue where the > quotas disappeared last August 2017: > > http://lists.gluster.org/pipermail/gluster-users/2017-August/031946.html > > In the mean time the only thing we did is to upgrade from 3.8 to 3.10. > > There are actually no errors to be seen using any gluster commands. The > "quota myvolume list" returns simply nothing. > > In order to lookup the directories should I run a "stat" on them? and if yes > should I do that on a client through the fuse mount? > > > Original Message > On February 13, 2018 10:58 AM, Hari Gowtham wrote: > >>The log provided are from 11th, you have seen the issue a while before >> that itself. >> >> The logs help us to know if something has actually went wrong. >> once something goes wrong the output might get affected and i need to know >> what >> went wrong. Which means i need the log from the beginning. >> >> and i need to know a few more things, >> Was the quota list command was working as expected at the beginning? >> If yes, what were the commands issued, before you noticed this problem. >> Is there any other error that you see other than this? >> >> And can you try looking up the directories the limits are set on and >> check if that fixes the error? >> >>> Original Message >>> On February 13, 2018 10:44 AM, mabi m...@protonmail.ch wrote: Hi Hari, Sure no problem, I will send you in a minute another mail where you can download all the relevant log files including the quota.conf binary file. Let me know if you need anything else. In the mean time here below is the output of a volume status. Best regards, M. Status of volume: myvolume Gluster process TCP Port RDMA Port Online Pid Brick gfs1a.domain.local:/data/myvolume /brick 49153 0 Y 3214 Brick gfs1b.domain.local:/data/myvolume /brick 49154 0 Y 3256 Brick gfs1c.domain.local:/srv/glusterf s/myvolume/brick 49153 0 Y 515 Self-heal Daemon on localhost N/A N/AY 3186 Quota Daemon on localhost N/A N/AY 3195 Self-heal Daemon on gfs1b.domain.local N/A N/AY 3217 Quota Daemon on gfs1b.domain.local N/A N/AY 3229 Self-heal Daemon on gfs1c.domain.local N/A N/AY 486 Quota Daemon on gfs1c.domain.local N/A N/AY 495 Task Status of Volume myvolume There are no active volume tasks Original Message On February 13, 2018 10:09 AM, Hari Gowtham hgowt...@redhat.com wrote: >Hi, > A part of the log won't be enough to debug the issue. > Need the whole log messages till date. > You can send it as attachments. > Yes the quota.conf is a binary file. > And I need the volume status output too. > On Tue, Feb 13, 2018 at 1:56 PM, mabi m...@protonmail.ch wrote: >>Hi Hari, >> Sorry for not providing you more details from the start. Here below you >> will >> find all the relevant log entries and info. Regarding the quota.conf >> file I >> have found one for my volume but it is a binary file. Is it supposed to >> be >> binary or text? >> Regards, >> M. >> *** gluster volume info myvolume *** >> Volume Name: myvolume >> Type: Replicate >> Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x (2 + 1) = 3 >> Transport-type: tcp >> Bricks: >> Brick1: gfs1a.domain.local:/data/myvolume/brick >> Brick2: gfs1b.domain.local:/data/myvolume/brick >> Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) >> Options Reconfigured: >> server.event-threads: 4 >> client.event-threads: 4 >> performance.readdir-ahead: on >> nfs.disable: on >> features.quota: on >> features.inode-quota: on >> features.quota-deem-statfs: on >> transpor
[Gluster-users] loads of broken symlinks in .glusterfs
Hi, A client had some issues with their glusterfs cluster and botched an update. Currently they have loads of broken symlinks in their bricks .glusterfs subfolders Like the second symlink one below lrwxrwxrwx 1 root root 55 Jul 14 2017 591c33db-e88b-4544-9372-bea4ec362360 -> ../../83/4a/834a8af8-95a4-4f8a-9810-4e4d829e87f9/394947 lrwxrwxrwx 1 root root 74 Jul 17 2017 591c89df-e5ed-4c5e-8ed1-643bbdf1ac3a -> ../../bb/e3/bbe35c18-467e-46a2-96d1-3f3cf07d30b2/omgevingsvisie-gelderland ./gfid-resolver.sh /data/home/brick1/ 591c33db-e88b-4544-9372-bea4ec362360 591c33db-e88b-4544-9372-bea4ec362360 Directory: /data/home/brick1/***/web_root/shared/private/uploads/image/file/39494 ./gfid-resolver.sh /data/home/brick1/ 591c89df-e5ed-4c5e-8ed1-643bbdf1ac3a 591c89df-e5ed-4c5e-8ed1-643bbdf1ac3a Directory: ./gfid-resolver.sh: line 46: cd: /data/home/brick1//.glusterfs/59/1c/../../bb/e3/bbe35c18-467e-46a2-96d1-3f3cf07d30b2: Too many levels of symbolic links /root/omgevingsvisie-gelderland The first is ok, the second link is broken. /data/home/brick1/.glusterfs/00/00 [root@storage01 00]# for l in $(find . -type l); do cd $(dirname $l); if [ ! -e "$(readlink $(basename $l))" ]; then echo $l; fi; cd - > /dev/null; done ./d3e7-bdaf-4c57-aaf8-e8d8906d85ce ./b8dd-6982-4dc5-80ee-0d2226b8a274 ./d173-3e12-41a2-973a-ca167f764b73 ./1420-a395-4f79-91db-690b009b8d3d ./9a6d-e885-4856-a9a1-d44badb4bef5 ./169f-1d61-4400-a13d-563df8dc78e1 ./9338-f7d9-4c33-8761-b0a7d0eaf6ef ./498f-3061-4c23-8633-410a21e54f60 ./e61f-eba8-4534-88f4-84c01c9bd698 ./9cb9-7d55-4558-93f3-f79ab4c7938d ./dd8c-ee79-47c4-9f7d-ef3569698907 ./403a-e9d8-4c76-9b62-72c396f34893 ./4ff2-a0f7-49d7-ac90-3eac37c6adba ./2d40-3ed7-4d9b-8b72-1e4f0568be3d ./dcce-6649-4446-ac4c-9eee16d5b009 ./c636-6d5a-4bd0-aeec-a9406af6f716 ./a5d9-57ac-416f-95c8-aa40577c2f99 ./36c1-9ec2-4a1c-846f-8b00b88a3718 ./3e8f-a9d8-4006-89ca-372800a814a7 ./5244-aeb7-4151-b8d5-3e8cc4861080 ./3c97-ea76-4c81-be46-9e943aefecce ./7ab9-c9a9-44b4-82d2-522a94270049 ./2ec6-ae91-4cae-ab99-db2a84deecaa ./778a-1489-4c43-bafc-bdcc134639dc Volume healing doesn't report any files to heal. Currently things are wonky as the client reports files missing that are actually there and this causes processes to fail. Also there are many of these errors in the client log: [2018-02-13 10:25:48.377536] W [MSGID: 114031] [client-rpc-fops.c:2151:client3_3_seek_cbk] 0-home-client-0: remote operation failed [No such device or address] [2018-02-13 10:25:48.386911] W [MSGID: 114031] [client-rpc-fops.c:2151:client3_3_seek_cbk] 0-home-client-0: remote operation failed [No such device or address] [2018-02-13 10:26:19.251306] W [MSGID: 114031] [client-rpc-fops.c:2151:client3_3_seek_cbk] 0-home-client-0: remote operation failed [No such device or address] Can I remove the orphaned links in .glusterfs? I haven't found a definitive answer yet and I don't want to cause more issues. Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts Tel: 053 20 30 270 i...@netbulae.euStaalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Failed to get quota limits
Were you able to set new limits after seeing this error? On Tue, Feb 13, 2018 at 4:19 PM, Hari Gowtham wrote: > Yes, I need the log files in that duration, the log rotated file after > hitting the > issue aren't necessary, but the ones before hitting the issues are needed > (not just when you hit it, the ones even before you hit it). > > Yes, you have to do a stat from the client through fuse mount. > > On Tue, Feb 13, 2018 at 3:56 PM, mabi wrote: >> Thank you for your answer. This problem seem to have started since last >> week, so should I also send you the same log files but for last week? I >> think logrotate rotates them on a weekly basis. >> >> The only two quota commands we use are the following: >> >> gluster volume quota myvolume limit-usage /directory 10GB >> gluster volume quota myvolume list >> >> basically to set a new quota or to list the current quotas. The quota list >> was working in the past yes but we already had a similar issue where the >> quotas disappeared last August 2017: >> >> http://lists.gluster.org/pipermail/gluster-users/2017-August/031946.html >> >> In the mean time the only thing we did is to upgrade from 3.8 to 3.10. >> >> There are actually no errors to be seen using any gluster commands. The >> "quota myvolume list" returns simply nothing. >> >> In order to lookup the directories should I run a "stat" on them? and if yes >> should I do that on a client through the fuse mount? >> >> >> Original Message >> On February 13, 2018 10:58 AM, Hari Gowtham wrote: >> >>>The log provided are from 11th, you have seen the issue a while before >>> that itself. >>> >>> The logs help us to know if something has actually went wrong. >>> once something goes wrong the output might get affected and i need to know >>> what >>> went wrong. Which means i need the log from the beginning. >>> >>> and i need to know a few more things, >>> Was the quota list command was working as expected at the beginning? >>> If yes, what were the commands issued, before you noticed this problem. >>> Is there any other error that you see other than this? >>> >>> And can you try looking up the directories the limits are set on and >>> check if that fixes the error? >>> Original Message On February 13, 2018 10:44 AM, mabi m...@protonmail.ch wrote: >Hi Hari, >Sure no problem, I will send you in a minute another mail where you can >download all the relevant log files including the quota.conf binary file. >Let me know if you need anything else. In the mean time here below is the >output of a volume status. >Best regards, > M. >Status of volume: myvolume > Gluster process TCP Port RDMA Port Online > Pid >Brick gfs1a.domain.local:/data/myvolume > /brick 49153 0 Y 3214 > Brick gfs1b.domain.local:/data/myvolume > /brick 49154 0 Y 3256 > Brick gfs1c.domain.local:/srv/glusterf > s/myvolume/brick 49153 0 Y 515 > Self-heal Daemon on localhost N/A N/AY > 3186 > Quota Daemon on localhost N/A N/AY > 3195 > Self-heal Daemon on gfs1b.domain.local N/A N/AY 3217 > Quota Daemon on gfs1b.domain.local N/A N/AY 3229 > Self-heal Daemon on gfs1c.domain.local N/A N/AY 486 > Quota Daemon on gfs1c.domain.local N/A N/AY 495 >Task Status of Volume myvolume >There are no active volume tasks > Original Message > On February 13, 2018 10:09 AM, Hari Gowtham hgowt...@redhat.com wrote: >>Hi, >> A part of the log won't be enough to debug the issue. >> Need the whole log messages till date. >> You can send it as attachments. >> Yes the quota.conf is a binary file. >> And I need the volume status output too. >> On Tue, Feb 13, 2018 at 1:56 PM, mabi m...@protonmail.ch wrote: >>>Hi Hari, >>> Sorry for not providing you more details from the start. Here below you >>> will >>> find all the relevant log entries and info. Regarding the quota.conf >>> file I >>> have found one for my volume but it is a binary file. Is it supposed to >>> be >>> binary or text? >>> Regards, >>> M. >>> *** gluster volume info myvolume *** >>> Volume Name: myvolume >>> Type: Replicate >>> Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5 >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 1 x (2 + 1) = 3 >>> Transport-type: tcp >>> Bricks: >>> Brick1: gfs1a.domain.local:/data/myvolume/brick >>> Brick2: gfs1b.domain.local:/data/myvolume/brick >>> Brick3: gfs1c.domain.local:/srv/glusterfs/myvolume/brick (arbiter) >>> Options Reconfigured:
[Gluster-users] Failover problems with gluster 3.8.8-1 (latest Debian stable)
I'm using gluster for a virt-store with 3x2 distributed/replicated servers for 16 qemu/kvm/libvirt virtual machines using image files stored in gluster and accessed via libgfapi. Eight of these disk images are standalone, while the other eight are qcow2 images which all share a single backing file. For the most part, this is all working very well. However, one of the gluster servers (azathoth) causes three of the standalone VMs and all 8 of the shared-backing-image VMs to fail if it goes down. Any of the other gluster servers can go down with no problems; only azathoth causes issues. In addition, the kvm hosts have the gluster volume fuse mounted and one of them (out of five) detects an error on the gluster volume and puts the fuse mount into read-only mode if azathoth goes down. libgfapi connections to the VM images continue to work normally from this host despite this and the other four kvm hosts are unaffected. It initially seemed relevant that I have the libgfapi URIs specified as gluster://azathoth/..., but I've tried changing them to make the initial connection via other gluster hosts and it had no effect on the problem. Losing azathoth still took them out. In addition to changing the mount URI, I've also manually run a heal and rebalance on the volume, enabled the bitrot daemons (then turned them back off a week later, since they reported no activity in that time), and copied one of the standalone images to a new file in case it was a problem with the file itself. As far as I can tell, none of these attempts changed anything. So I'm at a loss. Is this a known type of problem? If so, how do I fix it? If not, what's the next step to troubleshoot it? # gluster --version glusterfs 3.8.8 built on Jan 11 2017 14:07:11 Repository revision: git://git.gluster.com/glusterfs.git # gluster volume status Status of volume: palantir Gluster process TCP Port RDMA Port Online Pid -- Brick saruman:/var/local/brick0/data49154 0 Y 10690 Brick gandalf:/var/local/brick0/data49155 0 Y 18732 Brick azathoth:/var/local/brick0/data 49155 0 Y 9507 Brick yog-sothoth:/var/local/brick0/data49153 0 Y 39559 Brick cthulhu:/var/local/brick0/data49152 0 Y 2682 Brick mordiggian:/var/local/brick0/data 49152 0 Y 39479 Self-heal Daemon on localhost N/A N/AY 9614 Self-heal Daemon on saruman.lub.lu.se N/A N/AY 15016 Self-heal Daemon on cthulhu.lub.lu.se N/A N/AY 9756 Self-heal Daemon on gandalf.lub.lu.se N/A N/AY 5962 Self-heal Daemon on mordiggian.lub.lu.seN/A N/AY 8295 Self-heal Daemon on yog-sothoth.lub.lu.se N/A N/AY 7588 Task Status of Volume palantir -- Task : Rebalance ID : c38e11fe-fe1b-464d-b9f5-1398441cc229 Status : completed -- Dave Sherohman ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] loads of broken symlinks .glusterfs
Hi, A client had some issues with their glusterfs cluster and botched an update. Currently they have loads of broken symlinks in their bricks .glusterfs folder Like the second one below lrwxrwxrwx 1 root root 55 Jul 14 2017 591c33db-e88b-4544-9372-bea4ec362360 -> ../../83/4a/834a8af8-95a4-4f8a-9810-4e4d829e87f9/394947 lrwxrwxrwx 1 root root 74 Jul 17 2017 591c89df-e5ed-4c5e-8ed1-643bbdf1ac3a -> ../../bb/e3/bbe35c18-467e-46a2-96d1-3f3cf07d30b2/omgevingsvisie-gelderland ./gfid-resolver.sh /data/home/brick1/ 591c33db-e88b-4544-9372-bea4ec362360 591c33db-e88b-4544-9372-bea4ec362360 Directory: /data/home/brick1/vcjgdeventer/web_root/shared/private/uploads/image/file/39494 ./gfid-resolver.sh /data/home/brick1/ 591c89df-e5ed-4c5e-8ed1-643bbdf1ac3a 591c89df-e5ed-4c5e-8ed1-643bbdf1ac3a Directory: ./gfid-resolver.sh: line 46: cd: /data/home/brick1//.glusterfs/59/1c/../../bb/e3/bbe35c18-467e-46a2-96d1-3f3cf07d30b2: Too many levels of symbolic links /root/omgevingsvisie-gelderland The first is ok, the second link is broken. Volume healing doesn't report any files to heal. Currently things are wonky as the client reports files missing that are actually there and this causes processes to fail. Also there are many of these errors in the client log: [2018-02-13 10:25:48.377536] W [MSGID: 114031] [client-rpc-fops.c:2151:client3_3_seek_cbk] 0-home-client-0: remote operation failed [No such device or address] [2018-02-13 10:25:48.386911] W [MSGID: 114031] [client-rpc-fops.c:2151:client3_3_seek_cbk] 0-home-client-0: remote operation failed [No such device or address] [2018-02-13 10:26:19.251306] W [MSGID: 114031] [client-rpc-fops.c:2151:client3_3_seek_cbk] 0-home-client-0: remote operation failed [No such device or address] Can I remove the orphaned links in .glusterfs so things will heal back? Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts Tel: 053 20 30 270 i...@netbulae.euStaalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Failed to get quota limits
I tried to set the limits as you suggest by running the following command. $ sudo gluster volume quota myvolume limit-usage /directory 200GB volume quota : success but then when I list the quotas there is still nothing, so nothing really happened. I also tried to run stat on all directories which have a quota but nothing happened either. I will send you tomorrow all the other logfiles as requested. Original Message On February 13, 2018 12:20 PM, Hari Gowtham wrote: >Were you able to set new limits after seeing this error? > > On Tue, Feb 13, 2018 at 4:19 PM, Hari Gowtham hgowt...@redhat.com wrote: >>Yes, I need the log files in that duration, the log rotated file after >> hitting the >> issue aren't necessary, but the ones before hitting the issues are needed >> (not just when you hit it, the ones even before you hit it). >>Yes, you have to do a stat from the client through fuse mount. >>On Tue, Feb 13, 2018 at 3:56 PM, mabi m...@protonmail.ch wrote: >>>Thank you for your answer. This problem seem to have started since last >>>week, so should I also send you the same log files but for last week? I >>>think logrotate rotates them on a weekly basis. >>>The only two quota commands we use are the following: >>>gluster volume quota myvolume limit-usage /directory 10GB >>> gluster volume quota myvolume list >>>basically to set a new quota or to list the current quotas. The quota list >>>was working in the past yes but we already had a similar issue where the >>>quotas disappeared last August 2017: >>>http://lists.gluster.org/pipermail/gluster-users/2017-August/031946.html >>>In the mean time the only thing we did is to upgrade from 3.8 to 3.10. >>>There are actually no errors to be seen using any gluster commands. The >>>"quota myvolume list" returns simply nothing. >>>In order to lookup the directories should I run a "stat" on them? and if yes >>>should I do that on a client through the fuse mount? >>> Original Message >>> On February 13, 2018 10:58 AM, Hari Gowtham hgowt...@redhat.com wrote: The log provided are from 11th, you have seen the issue a while before that itself. The logs help us to know if something has actually went wrong. once something goes wrong the output might get affected and i need to know what went wrong. Which means i need the log from the beginning. and i need to know a few more things, Was the quota list command was working as expected at the beginning? If yes, what were the commands issued, before you noticed this problem. Is there any other error that you see other than this? And can you try looking up the directories the limits are set on and check if that fixes the error? > Original Message > On February 13, 2018 10:44 AM, mabi m...@protonmail.ch wrote: >>Hi Hari, >> Sure no problem, I will send you in a minute another mail where you can >> download all the relevant log files including the quota.conf binary >> file. Let me know if you need anything else. In the mean time here below >> is the output of a volume status. >> Best regards, >> M. >> Status of volume: myvolume >> Gluster process TCP Port RDMA Port Online >> Pid >> Brick gfs1a.domain.local:/data/myvolume >> /brick 49153 0 Y 3214 >> Brick gfs1b.domain.local:/data/myvolume >> /brick 49154 0 Y 3256 >> Brick gfs1c.domain.local:/srv/glusterf >> s/myvolume/brick 49153 0 Y 515 >> Self-heal Daemon on localhost N/A N/AY >> 3186 >> Quota Daemon on localhost N/A N/AY >> 3195 >> Self-heal Daemon on gfs1b.domain.local N/A N/AY 3217 >> Quota Daemon on gfs1b.domain.local N/A N/AY 3229 >> Self-heal Daemon on gfs1c.domain.local N/A N/AY 486 >> Quota Daemon on gfs1c.domain.local N/A N/AY 495 >> Task Status of Volume myvolume >> There are no active volume tasks >> Original Message >> On February 13, 2018 10:09 AM, Hari Gowtham hgowt...@redhat.com wrote: >>>Hi, >>> A part of the log won't be enough to debug the issue. >>> Need the whole log messages till date. >>> You can send it as attachments. >>> Yes the quota.conf is a binary file. >>> And I need the volume status output too. >>> On Tue, Feb 13, 2018 at 1:56 PM, mabi m...@protonmail.ch wrote: Hi Hari, Sorry for not providing you more details from the start. Here below you will find all the relevant log entries and info. Regarding the quota.conf file I have found one for my volume but it is a binary file. Is it supposed >>
