Thanks Mauro. On Thu, Mar 14, 2019 at 3:38 PM Mauro Tridici <[email protected]> wrote:
> Hi Raghavendra, > > I just changed the client option value to 8. > I will check the volume behaviour during the next hours. > > The GlusterFS version is 3.12.14. > > I will provide you the logs as soon as the activity load will be high. > Thank you, > Mauro > > On 14 Mar 2019, at 04:57, Raghavendra Gowdappa <[email protected]> > wrote: > > > > On Wed, Mar 13, 2019 at 3:55 PM Mauro Tridici <[email protected]> > wrote: > >> Hi Raghavendra, >> >> Yes, server.event-thread has been changed from 4 to 8. >> > > Was client.event-thread value too changed to 8? If not, I would like to > know the results of including this tuning too. Also, if possible, can you > get the output of following command from problematic clients and bricks > (during the duration when load tends to be high and ping-timer-expiry is > seen)? > > # top -bHd 3 > > This will help us to know CPU utilization of event-threads. > > And I forgot to ask, what version of Glusterfs are you using? > > During last days, I noticed that the error events are still here although >> they have been considerably reduced. >> >> So, I used grep command against the log files in order to provide you a >> global vision about the warning, error and critical events appeared today >> at 06:xx (may be useful I hope). >> I collected the info from s06 gluster server, but the behaviour is the >> the almost the same on the other gluster servers. >> >> *ERRORS: * >> *CWD: /var/log/glusterfs * >> *COMMAND: grep " E " *.log |grep "2019-03-13 06:"* >> >> (I can see a lot of this kind of message in the same period but I'm >> notifying you only one record for each type of error) >> >> glusterd.log:[2019-03-13 06:12:35.982863] E [MSGID: 101042] >> [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of >> /var/run/gluster/tier2_quota_list/ >> >> glustershd.log:[2019-03-13 06:14:28.666562] E >> [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a71ddcebb] (--> >> /lib64/libgfr >> pc.so.0(saved_frames_unwind+0x1de)[0x7f4a71ba1d9e] (--> >> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4a71ba1ebe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup >> +0x90)[0x7f4a71ba3640] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f4a71ba4130] ))))) >> 0-tier2-client-55: forced unwinding frame type(GlusterFS 3.3) >> op(INODELK(29)) >> called at 2019-03-13 06:14:14.858441 (xid=0x17fddb50) >> >> glustershd.log:[2019-03-13 06:17:48.883825] E >> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >> 192.168.0.55:49158 failed (Connection timed out); disco >> nnecting socket >> glustershd.log:[2019-03-13 06:19:58.931798] E >> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >> 192.168.0.55:49158 failed (Connection timed out); disco >> nnecting socket >> glustershd.log:[2019-03-13 06:22:08.979829] E >> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >> 192.168.0.55:49158 failed (Connection timed out); disco >> nnecting socket >> glustershd.log:[2019-03-13 06:22:36.226847] E [MSGID: 114031] >> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >> operation failed [Transport endpoint >> is not connected] >> glustershd.log:[2019-03-13 06:22:36.306669] E [MSGID: 114031] >> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >> operation failed [Transport endpoint >> is not connected] >> glustershd.log:[2019-03-13 06:22:36.385257] E [MSGID: 114031] >> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >> operation failed [Transport endpoint >> is not connected] >> >> *WARNINGS:* >> *CWD: /var/log/glusterfs * >> *COMMAND: grep " W " *.log |grep "2019-03-13 06:"* >> >> (I can see a lot of this kind of message in the same period but I'm >> notifying you only one record for each type of warnings) >> >> glustershd.log:[2019-03-13 06:14:28.666772] W [MSGID: 114031] >> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-tier2-client-55: remote >> operation failed. Path: <gfid:b6b35d >> 0f-f34d-4c25-bbe8-74bde0248d7e> (b6b35d0f-f34d-4c25-bbe8-74bde0248d7e). >> Key: (null) [Transport endpoint is not connected] >> >> glustershd.log:[2019-03-13 06:14:31.421576] W [MSGID: 122035] >> [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation >> with some subvolumes unavailable (2) >> >> glustershd.log:[2019-03-13 06:15:31.547417] W [MSGID: 122032] >> [ec-heald.c:266:ec_shd_index_sweep] 0-tier2-disperse-9: unable to get >> index-dir on tier2-client-55 [Operation >> now in progress] >> >> quota-mount-tier2.log:[2019-03-13 06:12:36.116277] W [MSGID: 101002] >> [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is >> deprecated, preferred is 'trans >> port.address-family', continuing with correction >> quota-mount-tier2.log:[2019-03-13 06:12:36.198430] W [MSGID: 101174] >> [graph.c:363:_log_if_unknown_option] 0-tier2-readdir-ahead: option >> 'parallel-readdir' is not recognized >> quota-mount-tier2.log:[2019-03-13 06:12:37.945007] W >> [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) >> [0x7f340892be25] -->/usr/sbin/glusterfs(gluste >> rfs_sigwaiter+0xe5) [0x55ef010164b5] >> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef0101632b] ) 0-: >> received signum (15), shutting down >> >> *CRITICALS:* >> *CWD: /var/log/glusterfs * >> *COMMAND: grep " C " *.log |grep "2019-03-13 06:"* >> >> no critical errors at 06:xx >> only one critical error during the day >> >> *[root@s06 glusterfs]# grep " C " *.log |grep "2019-03-13"* >> glustershd.log:[2019-03-13 02:21:29.126279] C >> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server >> 192.168.0.55:49158 has not responded in the last 42 seconds, >> disconnecting. >> >> >> Thank you very much for your help. >> Regards, >> Mauro >> >> On 12 Mar 2019, at 05:17, Raghavendra Gowdappa <[email protected]> >> wrote: >> >> Was the suggestion to increase server.event-thread values tried? If yes, >> what were the results? >> >> On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici <[email protected]> >> wrote: >> >>> Dear All, >>> >>> do you have any suggestions about the right way to "debug" this issue? >>> In attachment, the updated logs of “s06" gluster server. >>> >>> I noticed a lot of intermittent warning and error messages. >>> >>> Thank you in advance, >>> Mauro >>> >>> >>> >>> On 4 Mar 2019, at 18:45, Raghavendra Gowdappa <[email protected]> >>> wrote: >>> >>> >>> +Gluster Devel <[email protected]>, +Gluster-users >>> <[email protected]> >>> >>> I would like to point out another issue. Even if what I suggested >>> prevents disconnects, part of the solution would be only symptomatic >>> treatment and doesn't address the root cause of the problem. In most of the >>> ping-timer-expiry issues, the root cause is the increased load on bricks >>> and the inability of bricks to be responsive under high load. So, the >>> actual solution would be doing any or both of the following: >>> * identify the source of increased load and if possible throttle it. >>> Internal heal processes like self-heal, rebalance, quota heal are known to >>> pump traffic into bricks without much throttling (io-threads _might_ do >>> some throttling, but my understanding is its not sufficient). >>> * identify the reason for bricks to become unresponsive during load. >>> This may be fixable issues like not enough event-threads to read from >>> network or difficult to fix issues like fsync on backend fs freezing the >>> process or semi fixable issues (in code) like lock contention. >>> >>> So any genuine effort to fix ping-timer-issues (to be honest most of the >>> times they are not issues related to rpc/network) would involve performance >>> characterization of various subsystems on bricks and clients. Various >>> subsystems can include (but not necessarily limited to), underlying >>> OS/filesystem, glusterfs processes, CPU consumption etc >>> >>> regards, >>> Raghavendra >>> >>> On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici <[email protected]> >>> wrote: >>> >>>> Thank you, let’s try! >>>> I will inform you about the effects of the change. >>>> >>>> Regards, >>>> Mauro >>>> >>>> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa <[email protected]> >>>> wrote: >>>> >>>> >>>> >>>> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici <[email protected]> >>>> wrote: >>>> >>>>> Hi Raghavendra, >>>>> >>>>> thank you for your reply. >>>>> Yes, you are right. It is a problem that seems to happen randomly. >>>>> At this moment, server.event-threads value is 4. I will try to >>>>> increase this value to 8. Do you think that it could be a valid value ? >>>>> >>>> >>>> Yes. We can try with that. You should see at least frequency of >>>> ping-timer related disconnects reduce with this value (even if it doesn't >>>> eliminate the problem completely). >>>> >>>> >>>>> Regards, >>>>> Mauro >>>>> >>>>> >>>>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Mauro, >>>>>> >>>>>> It looks like some problem on s06. Are all your other nodes ok? Can >>>>>> you send us the gluster logs from this node? >>>>>> >>>>>> @Raghavendra G <[email protected]> , do you have any idea as >>>>>> to how this can be debugged? Maybe running top ? Or debug brick logs? >>>>>> >>>>> >>>>> If we can reproduce the problem, collecting tcpdump on both ends of >>>>> connection will help. But, one common problem is these bugs are >>>>> inconsistently reproducible and hence we may not be able to capture >>>>> tcpdump >>>>> at correct intervals. Other than that, we can try to collect some evidence >>>>> that poller threads were busy (waiting on locks). But, not sure what debug >>>>> data provides that information. >>>>> >>>>> From what I know, its difficult to collect evidence for this issue and >>>>> we could only reason about it. >>>>> >>>>> We can try a workaround though - try increasing server.event-threads >>>>> and see whether ping-timer expiry issues go away with an optimal value. If >>>>> that's the case, it kind of provides proof for our hypothesis. >>>>> >>>>> >>>>>> >>>>>> Regards, >>>>>> Nithya >>>>>> >>>>>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> some minutes ago I received this message from NAGIOS server >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ****** Nagios *****Notification Type: PROBLEMService: Brick - >>>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: CRITICALDate/Time: >>>>>>> Mon >>>>>>> Mar 4 10:25:33 CET 2019Additional Info:CHECK_NRPE STATE CRITICAL: Socket >>>>>>> timeout after 10 seconds.* >>>>>>> >>>>>>> I checked the network, RAM and CPUs usage on s06 node and everything >>>>>>> seems to be ok. >>>>>>> No bricks are in error state. In /var/log/messages, I detected again >>>>>>> a crash of “check_vol_utili” that I think it is a module used by NRPE >>>>>>> executable (that is the NAGIOS client). >>>>>>> >>>>>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general >>>>>>> protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in >>>>>>> libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>>>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of >>>>>>> user 0 killed by SIGSEGV - dumping core >>>>>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>>>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>>>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>>>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: >>>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory >>>>>>> ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify >>>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>>> 'report_uReport' exited with 1 >>>>>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of >>>>>>> user 0 killed by SIGABRT - dumping core >>>>>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>>>>> >>>>>>> Also, I noticed the following errors that I think are very critical: >>>>>>> >>>>>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: >>>>>>> server 192.168.0.55:49158 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>>>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>>>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: >>>>>>> server 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>>>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>>>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>>>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>>>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: >>>>>>> server 192.168.0.52:49158 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: >>>>>>> server >>>>>>> 192.168.0.51:49153 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: >>>>>>> server >>>>>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: >>>>>>> server >>>>>>> 192.168.0.51:49159 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: >>>>>>> server >>>>>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: >>>>>>> server >>>>>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: >>>>>>> server 192.168.0.54:49155 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: >>>>>>> server 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: >>>>>>> server 192.168.0.53:49159 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>>>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>>>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>>>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>>>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>>>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>>>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>>>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL >>>>>>> handshake with 192.168.1.56: 5 >>>>>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>>>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>>>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>>>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>>>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>>>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>>>>> >>>>>>> But, unfortunately, I don’t understand why it is happening. >>>>>>> Now, NAGIOS server shows that s06 status is ok: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ****** Nagios *****Notification Type: RECOVERYService: Brick - >>>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: OKDate/Time: Mon Mar >>>>>>> 4 >>>>>>> 10:35:23 CET 2019Additional Info:OK: Brick /gluster/mnt2/brick is up* >>>>>>> >>>>>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>>>>> /var/log/message file has been updated: >>>>>>> >>>>>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>>>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: >>>>>>> server 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: >>>>>>> server 192.168.0.54:49167 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>>>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>>>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>>>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>>>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client >>>>>>> 192.168.1.56, bailing out... >>>>>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>>>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>>>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>>>>> >>>>>>> Could you please help me to understand what it’s happening ? >>>>>>> Thank you in advance. >>>>>>> >>>>>>> Rergards, >>>>>>> Mauro >>>>>>> >>>>>>> >>>>>>> On 1 Mar 2019, at 12:17, Mauro Tridici <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> Thank you, Milind. >>>>>>> I executed the instructions you suggested: >>>>>>> >>>>>>> - grep “blocked for” /var/log/messages on s06 returns no output (no >>>>>>> “blocked” word is detected in messages file); >>>>>>> - in /var/log/messages file I can see this kind of error repeated >>>>>>> for a lot of times: >>>>>>> >>>>>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>>>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general >>>>>>> protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in >>>>>>> libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>>>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of >>>>>>> user 0 killed by SIGSEGV - dumping core >>>>>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>>>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: >>>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory >>>>>>> ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service >>>>>>> name='org.freedesktop.problems' (using servicehelper) >>>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated >>>>>>> service 'org.freedesktop.problems' >>>>>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify >>>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>>> 'report_uReport' exited with 1 >>>>>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>>>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>>>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>>>>> >>>>>>> - in /var/log/messages file I can see also 4 errors related to other >>>>>>> cluster servers: >>>>>>> >>>>>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>>>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>>>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: >>>>>>> server 192.168.0.51:49163 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>>>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>>>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: >>>>>>> server 192.168.0.52:49153 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: >>>>>>> server 192.168.0.52:49155 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>>>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>>>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: >>>>>>> server 192.168.0.51:49152 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>>>>> >>>>>>> No “blocked” word is in /var/log/messages files on other cluster >>>>>>> servers. >>>>>>> In attachment, the /var/log/messages file from s06 server. >>>>>>> >>>>>>> Thank you in advance, >>>>>>> Mauro >>>>>>> >>>>>>> <messages.zip> >>>>>>> >>>>>>> >>>>>>> On 1 Mar 2019, at 11:47, Milind Changire <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> The traces of very high disk activity on the servers are often found >>>>>>> in /var/log/messages >>>>>>> You might want to grep for "blocked for" in /var/log/messages on s06 >>>>>>> and correlate the timestamps to confirm the unresponsiveness as >>>>>>> reported in >>>>>>> gluster client logs. >>>>>>> In cases of high disk activity, although the operating system >>>>>>> continues to respond to ICMP pings, the processes writing to disks often >>>>>>> get blocked to a large flush to the disk which could span beyond 42 >>>>>>> seconds >>>>>>> and hence result in ping-timer-expiry logs. >>>>>>> >>>>>>> As a side note: >>>>>>> If you indeed find gluster processes being blocked in >>>>>>> /var/log/messages, you might want to tweak sysctl tunables called >>>>>>> vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller >>>>>>> value >>>>>>> than the existing. Please read up more on those tunables before touching >>>>>>> the settings. >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> in attachment the client log captured after changing >>>>>>>> network.ping-timeout option. >>>>>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>>>>> >>>>>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >>>>>>>> 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>>>>> [2019-03-01 09:23:36.078213] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 09:23:36.078432] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 09:23:36.092357] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 09:23:36.094146] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 10:06:24.708082] C >>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: >>>>>>>> server >>>>>>>> 192.168.0.56:49156 has not responded in the last 42 seconds, >>>>>>>> disconnecting. >>>>>>>> >>>>>>>> I don’t know why it happens, s06 server seems to be reachable. >>>>>>>> >>>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>>> Trying 192.168.0.56... >>>>>>>> Connected to 192.168.0.56. >>>>>>>> Escape character is '^]'. >>>>>>>> ^CConnection closed by foreign host. >>>>>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>>>>> 64 bytes from 192.168.0.56: icmp_seq=1 ttl=64 time=0.116 ms >>>>>>>> 64 bytes from 192.168.0.56: icmp_seq=2 ttl=64 time=0.101 ms >>>>>>>> >>>>>>>> --- 192.168.0.56 ping statistics --- >>>>>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>>>>> >>>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>>> Trying 192.168.0.56... >>>>>>>> Connected to 192.168.0.56. >>>>>>>> Escape character is '^]'. >>>>>>>> >>>>>>>> Thank you for your help, >>>>>>>> Mauro >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> thank you for the explanation. >>>>>>>> I just changed network.ping-timeout option to default value >>>>>>>> (network.ping-timeout=42). >>>>>>>> >>>>>>>> I will check the logs to see if the errors will appear again. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mauro >>>>>>>> >>>>>>>> On 1 Mar 2019, at 04:43, Milind Changire <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> network.ping-timeout should not be set to zero for non-glusterd >>>>>>>> clients. >>>>>>>> glusterd is a special case for which ping-timeout is set to zero >>>>>>>> via /etc/glusterfs/glusterd.vol >>>>>>>> >>>>>>>> Setting network.ping-timeout to zero disables arming of the ping >>>>>>>> timer for connections. This disables testing the connection for >>>>>>>> responsiveness and hence avoids proactive fail-over. >>>>>>>> >>>>>>>> Please reset network.ping-timeout to a non-zero positive value, eg. >>>>>>>> 42 >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>>>> >>>>>>>>> What is the effect of setting network.ping-timeout to 0 and should >>>>>>>>> it be set back to 42? >>>>>>>>> Regards, >>>>>>>>> Nithya >>>>>>>>> >>>>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Nithya, >>>>>>>>>> >>>>>>>>>> sorry for the late. >>>>>>>>>> network.ping-timeout has been set to 0 in order to try to solve >>>>>>>>>> some timeout problems, but it didn’t help. >>>>>>>>>> I can set it to the default value. >>>>>>>>>> >>>>>>>>>> Can I proceed with the change? >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Mauro >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Hi Mauro, >>>>>>>>>> >>>>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. >>>>>>>>>> Is there a particular reason why this was changed? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Nithya >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Xavi, >>>>>>>>>>> >>>>>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>>>>> >>>>>>>>>>> I will check the network and connectivity status using “ping” >>>>>>>>>>> and “telnet” as soon as the errors will come back again. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Mauro >>>>>>>>>>> >>>>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez < >>>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>>> >>>>>>>>>>> Hi Mauro, >>>>>>>>>>> >>>>>>>>>>> those errors say that the mount point is not connected to some >>>>>>>>>>> of the bricks while executing operations. I see references to 3rd >>>>>>>>>>> and 6th >>>>>>>>>>> bricks of several disperse sets, which seem to map to server s06. >>>>>>>>>>> For some >>>>>>>>>>> reason, gluster is having troubles connecting from the client >>>>>>>>>>> machine to >>>>>>>>>>> that particular server. At the end of the log I see that after long >>>>>>>>>>> time a >>>>>>>>>>> reconnect is done to both of them. However little after, other >>>>>>>>>>> bricks from >>>>>>>>>>> the s05 get disconnected and a reconnect times out. >>>>>>>>>>> >>>>>>>>>>> That's really odd. It seems like if server/communication is cut >>>>>>>>>>> to s06 for some time, then restored, and then the same happens to >>>>>>>>>>> the next >>>>>>>>>>> server. >>>>>>>>>>> >>>>>>>>>>> If the servers are really online and it's only a communication >>>>>>>>>>> issue, it explains why server memory and network has increased: if >>>>>>>>>>> the >>>>>>>>>>> problem only exists between the client and servers, any write made >>>>>>>>>>> by the >>>>>>>>>>> client will automatically mark the file as damaged, since some of >>>>>>>>>>> the >>>>>>>>>>> servers have not been updated. Since self-heal runs from the server >>>>>>>>>>> nodes, >>>>>>>>>>> they will probably be correctly connected to all bricks, which >>>>>>>>>>> allows them >>>>>>>>>>> to heal the just damaged file, which increases memory and network >>>>>>>>>>> usage. >>>>>>>>>>> >>>>>>>>>>> I guess you still have transport.listen-backlog set to 1024, >>>>>>>>>>> right ? >>>>>>>>>>> >>>>>>>>>>> Just to try to identify if the problem really comes from >>>>>>>>>>> network, can you check if you lose some pings from the client to >>>>>>>>>>> all of the >>>>>>>>>>> servers while you are seeing those errors in the log file ? >>>>>>>>>>> >>>>>>>>>>> You can also check if during those errors, you can telnet to the >>>>>>>>>>> port of the brick from the client. >>>>>>>>>>> >>>>>>>>>>> Xavi >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Nithya, >>>>>>>>>>>> >>>>>>>>>>>> “df -h” operation is not still slow, but no users are using the >>>>>>>>>>>> volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>>>>> >>>>>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>>>>> >>>>>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] >>>>>>>>>>>> [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing >>>>>>>>>>>> operation >>>>>>>>>>>> with some subvolumes unavailable (20) >>>>>>>>>>>> >>>>>>>>>>>> [2019-02-26 03:11:35.212603] E >>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] >>>>>>>>>>>> ))))) >>>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) >>>>>>>>>>>> op(WRITE(13)) >>>>>>>>>>>> called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>>>>> >>>>>>>>>>>> [2019-02-26 03:13:03.313831] E >>>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-50: >>>>>>>>>>>> connection to >>>>>>>>>>>> 192.168.0.56:49156 failed (Timeout della connessione); >>>>>>>>>>>> disconnecting socket >>>>>>>>>>>> >>>>>>>>>>>> It seems that some subvolumes are not available and >>>>>>>>>>>> 192.168.0.56 server (s06) is not reachable. >>>>>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>>>>> >>>>>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank you. >>>>>>>>>>>> Regards, >>>>>>>>>>>> Mauro >>>>>>>>>>>> >>>>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran < >>>>>>>>>>>> [email protected]> ha scritto: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I see a lot of EC messages in the log but they don't seem very >>>>>>>>>>>> serious. Xavi, can you take a look? >>>>>>>>>>>> >>>>>>>>>>>> The only errors I see are: >>>>>>>>>>>> [2019-02-25 10:58:45.519871] E >>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] >>>>>>>>>>>> ))))) >>>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) >>>>>>>>>>>> op(WRITE(13)) >>>>>>>>>>>> called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>>>> [2019-02-25 10:58:51.461493] E >>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] >>>>>>>>>>>> (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] >>>>>>>>>>>> ))))) >>>>>>>>>>>> 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) >>>>>>>>>>>> op(WRITE(13)) >>>>>>>>>>>> called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>>>> [2019-02-25 11:07:57.152874] E >>>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-70: >>>>>>>>>>>> connection to >>>>>>>>>>>> 192.168.0.55:49163 failed (Timeout della connessione); >>>>>>>>>>>> disconnecting socket >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Is the df -h operation still slow? If yes, can you take a >>>>>>>>>>>> tcpdump of the client while running df -h and send that across? >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Nithya >>>>>>>>>>>> >>>>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that >>>>>>>>>>>>> “df -h” command hanged for a while before returns the prompt. >>>>>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, >>>>>>>>>>>>> today, I see a lot of errors (please, take a look to the attached >>>>>>>>>>>>> file). >>>>>>>>>>>>> >>>>>>>>>>>>> On the client node, I detected an important RAM e NETWORK >>>>>>>>>>>>> usage. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Do you think that the errors have been caused by the client >>>>>>>>>>>>> resources usage? >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you in advance, >>>>>>>>>>>>> Mauro >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>> >>>> >>>> >>> >>> >> >> > >
_______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
