Re: [Gluster-users] Odd "Transport endpoint is not connected" when trying to gunzip a file

2022-06-21 Thread Pat Haley


Hi Strahil

I have tried a couple of tests of trying to gunzip the file with top 
running on the client (mseas) and on the brick server (mseas-data3) and 
with iotop running on the client (mseas).  I was not able to install 
iotop on the brick server yet (the external line is down).  I'll repeat 
when I fix that problem


I now can get one of two error messages when gunzip fails:

 * gzip:
   /projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz:
   File descriptor in bad state
 o a new error message
 * gzip:
   /projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz:
   Transport endpoint is not connected
 o the original error message

What I observed while waiting for gunzip to fail

 * top
 o no significant load (usually less than 0.1) on both machines.
 o zero IO-wait on both machines
 * iotop (only running on the client)
 o nothing related to gluster showing up in the display at all

I include below what I found in the log files again corresponding to 
these tests (and what I see in dmesg on the brick-server related to 
gluster, nothing showed up on the client)


Please let me know what I should try next.

Thanks

Pat


--
mseas-data3: dmesg | grep glust
--
many repeats of the following pairs of lines:

glusterfsd: page allocation failure. order:1, mode:0x20
Pid: 14245, comm: glusterfsd Not tainted 2.6.32-754.2.1.el6.x86_64 #1

--
mseas:messages
--
Jun 21 17:04:35 mseas gdata[155485]: [2022-06-21 21:04:35.638810] C 
[rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 
0-data-volume-client-2: server 172.16.1.113:49153 has not responded in 
the last 42 seconds, disconnecting.


Jun 21 17:21:04 mseas gdata[155485]: [2022-06-21 21:21:04.786083] C 
[rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 
0-data-volume-client-2: server 172.16.1.113:49153 has not responded in 
the last 42 seconds, disconnecting.


--
mseas:gdata.log
--
[2022-06-21 21:04:35.638810] C 
[rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 
0-data-volume-client-2: server 172.16.1.113:49153 has not responded in 
the last 42 seconds, disconnecting.
[2022-06-21 21:04:35.639261] E [rpc-clnt.c:362:saved_frames_unwind] (--> 
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x172)[0x7f84886a0202] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x7f848846c3e2] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f848846c4de] 
(--> 
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f848846dd2a] 
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f848846e538] 
) 0-data-volume-client-2: forced unwinding frame type(GlusterFS 3.3) 
op(READ(12)) called at 2022-06-21 21:03:29.735807 (xid=0xc05d54)
[2022-06-21 21:04:35.639494] E [rpc-clnt.c:362:saved_frames_unwind] (--> 
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x172)[0x7f84886a0202] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x7f848846c3e2] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f848846c4de] 
(--> 
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f848846dd2a] 
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f848846e538] 
) 0-data-volume-client-2: forced unwinding frame type(GF-DUMP) 
op(NULL(2)) called at 2022-06-21 21:03:53.633472 (xid=0xc05d55)



[2022-06-21 21:21:04.786083] C 
[rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 
0-data-volume-client-2: server 172.16.1.113:49153 has not responded in 
the last 42 seconds, disconnecting.
[2022-06-21 21:21:04.786732] E [rpc-clnt.c:362:saved_frames_unwind] (--> 
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x172)[0x7f84886a0202] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x7f848846c3e2] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f848846c4de] 
(--> 
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f848846dd2a] 
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f848846e538] 
) 0-data-volume-client-2: forced unwinding frame type(GlusterFS 3.3) 
op(READ(12)) called at 2022-06-21 21:19:52.634383 (xid=0xc05e31)
[2022-06-21 21:21:04.787172] E [rpc-clnt.c:362:saved_frames_unwind] (--> 
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x172)[0x7f84886a0202] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x7f848846c3e2] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f848846c4de] 
(--> 
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f848846dd2a] 
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f848846e538] 
) 0-data-volume-client-2: forced unwinding frame type(GF-DUMP) 
op(NULL(2)) called at 2022-06-21 21:20:22.780023 (xid=0xc05e32)


--
mseas-data3: 

Re: [Gluster-users] Odd "Transport endpoint is not connected" when trying to gunzip a file

2022-06-16 Thread Pat Haley


Hi Strahil,

I poked around our logs, and found this on the front-end (from the day & 
time of the last time we had the issue)



Jun 15 10:51:17 mseas gdata[155485]: [2022-06-15 14:51:17.263858] C 
[rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 
0-data-volume-client-2: server 172.16.1.113:49153 has not responded in 
the last 42 seconds, disconnecting.



This would indicate that the problem is related.  For us, however, I 
believe we can reproduce this issue at will (i.e. simply try to gunzip 
the same file). Unfortunately I have to go to a meeting now, but if you 
have some specific tests you'd like me to try, I can try them when I get 
back.


Thanks

Pat



On 6/16/22 3:07 PM, Strahil Nikolov wrote:

Pat,

Can you check the cpu and disk  performance when the volume reports 
the issue?



It seems that similar issue was reported in 
https://lists.gluster.org/pipermail/gluster-users/2019-March/035944.html 
but I don't see a clear solution.

Take a look in the thread and check if it matches your symptoms.


Best Regards,
Strahil Nikolov

On Thu, Jun 16, 2022 at 18:14, Pat Haley
 wrote:


Hi Strahil,

I poked around again and for brick 3 (where the file we were
testing resides)  I only found the same log file as was at the
bottom of my first Email:


---
mseas-data3:  bricks/export-sda-brick3.log
-
[2022-06-15 14:50:42.588143] I [MSGID: 115036]
[server.c:552:server_rpc_notify] 0-data-volume-server:
disconnecting connection from
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
[2022-06-15 14:50:42.588220] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd
cleanup on

/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC
[2022-06-15 14:50:42.588259] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd
cleanup on
/projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz
[2022-06-15 14:50:42.588288] I [MSGID: 101055]
[client_t.c:420:gf_client_unref] 0-data-volume-server: Shutting
down connection
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
[2022-06-15 14:50:53.605215] I [MSGID: 115029]
[server-handshake.c:690:server_setvolume] 0-data-volume-server:
accepted client from
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-29
(version: 3.7.11)
[2022-06-15 14:50:42.588247] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd
cleanup on

/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC

Thanks

Pat


On 6/15/22 6:47 PM, Strahil Nikolov wrote:
I agree. It will be very hard to debug.

Anything in the brick logs ?

I think it's pointless to mention that EL6 is dead and Gluster v3
is so old that it's worth considering a migration to a newer setup.

Best Regards,
Strahil Nikolov

On Wed, Jun 15, 2022 at 22:51, Yaniv Kaul
  wrote:




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-users mailing list
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users


-- 


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:pha...@mit.edu  

Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/  

77 Massachusetts Avenue
Cambridge, MA  02139-4301


--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:pha...@mit.edu
Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users


Re: [Gluster-users] Odd "Transport endpoint is not connected" when trying to gunzip a file

2022-06-16 Thread Pat Haley


Hi Strahil,

I poked around again and for brick 3 (where the file we were testing 
resides)  I only found the same log file as was at the bottom of my 
first Email:



---
mseas-data3:  bricks/export-sda-brick3.log
-
[2022-06-15 14:50:42.588143] I [MSGID: 115036] 
[server.c:552:server_rpc_notify] 0-data-volume-server: disconnecting 
connection from 
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
[2022-06-15 14:50:42.588220] I [MSGID: 115013] 
[server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on 
/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC
[2022-06-15 14:50:42.588259] I [MSGID: 115013] 
[server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on 
/projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz
[2022-06-15 14:50:42.588288] I [MSGID: 101055] 
[client_t.c:420:gf_client_unref] 0-data-volume-server: Shutting down 
connection 
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
[2022-06-15 14:50:53.605215] I [MSGID: 115029] 
[server-handshake.c:690:server_setvolume] 0-data-volume-server: accepted 
client from 
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-29 
(version: 3.7.11)
[2022-06-15 14:50:42.588247] I [MSGID: 115013] 
[server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on 
/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC


Thanks

Pat


On 6/15/22 6:47 PM, Strahil Nikolov wrote:

I agree. It will be very hard to debug.

Anything in the brick logs ?

I think it's pointless to mention that EL6 is dead and Gluster v3 is 
so old that it's worth considering a migration to a newer setup.


Best Regards,
Strahil Nikolov

On Wed, Jun 15, 2022 at 22:51, Yaniv Kaul
 wrote:




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users


--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:pha...@mit.edu
Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users


Re: [Gluster-users] Odd "Transport endpoint is not connected" when trying to gunzip a file

2022-06-15 Thread Yaniv Kaul
On Wed, Jun 15, 2022 at 6:28 PM Pat Haley  wrote:

>
> Hi,
>
> We have a cluster whose common storage is a gluster volume consisting of 5
> bricks residing on 3 servers.
>
>- Gluster volume machines
>   - mseas-data2:  CentOS release 6.8 (Final)
>   - mseas-data3:  CentOS release 6.10 (Final)
>   - mseas-data4:  CentOS Linux release 7.9.2009 (Core)
>- Client machines
>   - CentOS Linux release 7.9.2009 (Core)
>
> More details on the gluster volume are included below.
>
> We were recently trying to gunzip a file on the gluster volume and got  a
> "Transport endpoint is not connected" even though every test we try shows
> that gluster is fully up and running fine.  We traced the file to brick 3
> in the server mseas-data3.  We have included the relevant portions of the
> various log files on the client (mseas) where we were running the gunzip
> command and the server hosting the file (mseas-data3) below the gluster
> information
>
> What can you suggest we do to further debug and/or solve this issue?
>
> Thanks
> Pat
>
> 
> Gluster volume information
> 
>
> ---
> gluster volume info
> -
>
> Volume Name: data-volume
> Type: Distribute
> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> Status: Started
> Number of Bricks: 5
> Transport-type: tcp
> Bricks:
> Brick1: mseas-data2:/mnt/brick1
> Brick2: mseas-data2:/mnt/brick2
> Brick3: mseas-data3:/export/sda/brick3
> Brick4: mseas-data3:/export/sdc/brick4
> Brick5: mseas-data4:/export/brick5
> Options Reconfigured:
> diagnostics.client-log-level: ERROR
> network.inode-lru-limit: 5
> performance.md-cache-timeout: 60
> performance.open-behind: off
> disperse.eager-lock: off
> auth.allow: *
> server.allow-insecure: on
> nfs.exports-auth-enable: on
> diagnostics.brick-sys-log-level: WARNING
> performance.readdir-ahead: on
> nfs.disable: on
> nfs.export-volumes: off
> cluster.min-free-disk: 1%
>
> ---
> gluster volume status
> 
>
> Status of volume: data-volume
> Gluster process TCP Port  RDMA Port  Online
> Pid
>
> --
> Brick mseas-data2:/mnt/brick1   49154 0  Y
> 15978
> Brick mseas-data2:/mnt/brick2   49155 0  Y
> 15997
> Brick mseas-data3:/export/sda/brick349153 0  Y
> 14221
> Brick mseas-data3:/export/sdc/brick449154 0  Y
> 14240
> Brick mseas-data4:/export/brick549152 0  Y
> 50569
>
>
> ---
> gluster peer status
> -
>
> Number of Peers: 2
>
> Hostname: mseas-data3
> Uuid: b39d4deb-c291-437e-8013-09050c1fa9e3
> State: Peer in Cluster (Connected)
>
> Hostname: mseas-data4
> Uuid: 5c4d06eb-df89-4e5c-92e4-441fb401a9ef
> State: Peer in Cluster (Connected)
>
> ---
> glusterfs --version
> 
>
> glusterfs 3.7.11 built on Apr 18 2016 13:20:46
>

This is somewhat of an outdated version, I think it's best to upgrade (or
better - migrate?) to a newer version.
Y.

> Repository revision: git://git.gluster.com/glusterfs.git
> Copyright (c) 2006-2013 Red Hat, Inc. 
> 
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> It is licensed to you under your choice of the GNU Lesser
> General Public License, version 3 or any later version (LGPLv3
> or later), or the GNU General Public License, version 2 (GPLv2),
> in all cases as published by the Free Software Foundation.
>
> 
> Relevant sections from log files
> 
>
> ---
> mseas: gdata.log
> -
>
> [2022-06-15 14:51:17.263858] C
> [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-data-volume-client-2:
> server 172.16.1.113:49153 has not responded in the last 42 seconds,
> disconnecting.
> [2022-06-15 14:51:17.264522] E [rpc-clnt.c:362:saved_frames_unwind] (-->
> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x172)[0x7f84886a0202]
> (-->
> /usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x7f848846c3e2]
> (--> /usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f848846c4de]
> (-->
> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f848846dd2a]
> (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f848846e538]
> ) 0-data-volume-client-2: forced unwinding frame type(GlusterFS 3.3)
> op(READ(12)) called at 2022-06-15 14:49:52.113795 

[Gluster-users] Odd "Transport endpoint is not connected" when trying to gunzip a file

2022-06-15 Thread Pat Haley


Hi,

We have a cluster whose common storage is a gluster volume consisting of 
5 bricks residing on 3 servers.


 * Gluster volume machines
 o mseas-data2:  CentOS release 6.8 (Final)
 o mseas-data3:  CentOS release 6.10 (Final)
 o mseas-data4:  CentOS Linux release 7.9.2009 (Core)
 * Client machines
 o CentOS Linux release 7.9.2009 (Core)

More details on the gluster volume are included below.

We were recently trying to gunzip a file on the gluster volume and got  
a "Transport endpoint is not connected" even though every test we try 
shows that gluster is fully up and running fine.  We traced the file to 
brick 3 in the server mseas-data3.  We have included the relevant 
portions of the various log files on the client (mseas) where we were 
running the gunzip command and the server hosting the file (mseas-data3) 
below the gluster information


What can you suggest we do to further debug and/or solve this issue?

Thanks
Pat


Gluster volume information


---
gluster volume info
-

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Brick3: mseas-data3:/export/sda/brick3
Brick4: mseas-data3:/export/sdc/brick4
Brick5: mseas-data4:/export/brick5
Options Reconfigured:
diagnostics.client-log-level: ERROR
network.inode-lru-limit: 5
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
cluster.min-free-disk: 1%

---
gluster volume status


Status of volume: data-volume
Gluster process TCP Port  RDMA Port Online  Pid
--
Brick mseas-data2:/mnt/brick1   49154 0 Y   15978
Brick mseas-data2:/mnt/brick2   49155 0 Y   15997
Brick mseas-data3:/export/sda/brick3    49153 0 Y   14221
Brick mseas-data3:/export/sdc/brick4    49154 0 Y   14240
Brick mseas-data4:/export/brick5    49152 0 Y   50569


---
gluster peer status
-

Number of Peers: 2

Hostname: mseas-data3
Uuid: b39d4deb-c291-437e-8013-09050c1fa9e3
State: Peer in Cluster (Connected)

Hostname: mseas-data4
Uuid: 5c4d06eb-df89-4e5c-92e4-441fb401a9ef
State: Peer in Cluster (Connected)

---
glusterfs --version


glusterfs 3.7.11 built on Apr 18 2016 13:20:46
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. 
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.


Relevant sections from log files


---
mseas: gdata.log
-

[2022-06-15 14:51:17.263858] C 
[rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 
0-data-volume-client-2: server 172.16.1.113:49153 has not responded in 
the last 42 seconds, disconnecting.
[2022-06-15 14:51:17.264522] E [rpc-clnt.c:362:saved_frames_unwind] (--> 
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x172)[0x7f84886a0202] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x7f848846c3e2] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f848846c4de] 
(--> 
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f848846dd2a] 
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7f848846e538] 
) 0-data-volume-client-2: forced unwinding frame type(GlusterFS 3.3) 
op(READ(12)) called at 2022-06-15 14:49:52.113795 (xid=0xb4f49b)
[2022-06-15 14:51:17.264859] E [rpc-clnt.c:362:saved_frames_unwind] (--> 
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x172)[0x7f84886a0202] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1c2)[0x7f848846c3e2] 
(--> 
/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f848846c4de] 
(--> 
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7a)[0x7f848846dd2a] 
(-->