Hi Strahil,

I poked around our logs, and found this on the front-end (from the day & time of the last time we had the issue)


Jun 15 10:51:17 mseas gdata[155485]: [2022-06-15 14:51:17.263858] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-data-volume-client-2: server 172.16.1.113:49153 has not responded in the last 42 seconds, disconnecting.


This would indicate that the problem is related.  For us, however, I believe we can reproduce this issue at will (i.e. simply try to gunzip the same file). Unfortunately I have to go to a meeting now, but if you have some specific tests you'd like me to try, I can try them when I get back.

Thanks

Pat



On 6/16/22 3:07 PM, Strahil Nikolov wrote:
Pat,

Can you check the cpu and disk  performance when the volume reports the issue?


It seems that similar issue was reported in https://lists.gluster.org/pipermail/gluster-users/2019-March/035944.html but I don't see a clear solution.
Take a look in the thread and check if it matches your symptoms.


Best Regards,
Strahil Nikolov

    On Thu, Jun 16, 2022 at 18:14, Pat Haley
    <pha...@mit.edu> wrote:


    Hi Strahil,

    I poked around again and for brick 3 (where the file we were
    testing resides)  I only found the same log file as was at the
    bottom of my first Email:


    ---------------------------------------------------
    mseas-data3:  bricks/export-sda-brick3.log
    -----------------------------------------
    [2022-06-15 14:50:42.588143] I [MSGID: 115036]
    [server.c:552:server_rpc_notify] 0-data-volume-server:
    disconnecting connection from
    mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
    [2022-06-15 14:50:42.588220] I [MSGID: 115013]
    [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd
    cleanup on
    
/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC
    [2022-06-15 14:50:42.588259] I [MSGID: 115013]
    [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd
    cleanup on
    /projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz
    [2022-06-15 14:50:42.588288] I [MSGID: 101055]
    [client_t.c:420:gf_client_unref] 0-data-volume-server: Shutting
    down connection
    mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
    [2022-06-15 14:50:53.605215] I [MSGID: 115029]
    [server-handshake.c:690:server_setvolume] 0-data-volume-server:
    accepted client from
    mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-29
    (version: 3.7.11)
    [2022-06-15 14:50:42.588247] I [MSGID: 115013]
    [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd
    cleanup on
    
/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC

    Thanks

    Pat


    On 6/15/22 6:47 PM, Strahil Nikolov wrote:
    I agree. It will be very hard to debug.

    Anything in the brick logs ?

    I think it's pointless to mention that EL6 is dead and Gluster v3
    is so old that it's worth considering a migration to a newer setup.

    Best Regards,
    Strahil Nikolov

        On Wed, Jun 15, 2022 at 22:51, Yaniv Kaul
        <yk...@redhat.com> <mailto:yk...@redhat.com> wrote:
        ________



        Community Meeting Calendar:

        Schedule -
        Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
        Bridge: https://meet.google.com/cpu-eiue-hvk
        <https://meet.google.com/cpu-eiue-hvk>
        Gluster-users mailing list
        Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
        https://lists.gluster.org/mailman/listinfo/gluster-users
        <https://lists.gluster.org/mailman/listinfo/gluster-users>

--
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
    Pat Haley                          Email:pha...@mit.edu  
<mailto:pha...@mit.edu>
    Center for Ocean Engineering       Phone:  (617) 253-6824
    Dept. of Mechanical Engineering    Fax:    (617) 253-8125
    MIT, Room 5-213http://web.mit.edu/phaley/www/  
<http://web.mit.edu/phaley/www/>
    77 Massachusetts Avenue
    Cambridge, MA  02139-4301

--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:pha...@mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Reply via email to