Re: [Gluster-users] KVM lockups on Gluster 4.1.1

Wed, 10 Oct 2018 02:59:04 -0700 

Hi,
Sorry for the delay, should have gotten to this earlier. We uncovered the issue in our internal QE testing and it is a regression. Details/patch is available in BZ 1637802.  I'll back-port it to the release branches once it gets merged.  Shout out to Pranith for helping with the RCA! 
Regards,
Ravi

On 10/02/2018 10:39 AM, Dmitry Melekhov wrote:

01.10.2018 23:09, Danny Lee пишет:
Ran into this issue too with 4.1.5 with an arbiter setup.  Also could not run a statedump due to "Segmentation fault".
Tried with 3.12.13 and had issues with locked files as well.  We were able to do a statedump and found that some of our files were "BLOCKED" (xlator.features.locks.vol-locks.inode).  Attached part of statedump.
Also tried clearing the locks using clear-locks, which did remove the lock, but as soon as I tried to cat the file, it got locked again and the cat process hung. 

I created issue in bugzilla, can't find it though :-(
Looks like there is no activity after I sent all logs...
On Wed, Aug 29, 2018, 3:13 AM Dmitry Melekhov <[email protected] <mailto:[email protected]>> wrote: 

    28.08.2018 10:43, Amar Tumballi пишет:



    On Tue, Aug 28, 2018 at 11:24 AM, Dmitry Melekhov <[email protected]
    <mailto:[email protected]>> wrote:

        Hello!


        Yesterday we hit something like this on 4.1.2

        Centos 7.5.


        Volume is replicated - two bricks and one arbiter.


        We rebooted arbiter, waited for heal end,  and tried to live
        migrate VM to another node ( we run VMs on gluster nodes ):


        [2018-08-27 09:56:22.085411] I [MSGID: 115029]
        [server-handshake.c:763:server_setvolume] 0-pool-server:
        accepted client from
        
CTX_ID:b55f4a90-e241-48ce-bd4d-268c8a956f4a-GRAPH_ID:0-PID:8887-HOST:son-PC_NAME:pool-
        client-6-RECON_NO:-0 (version: 4.1.2)
        [2018-08-27 09:56:22.107609] I [MSGID: 115036]
        [server.c:483:server_rpc_notify] 0-pool-server:
        disconnecting connection from
        
CTX_ID:b55f4a90-e241-48ce-bd4d-268c8a956f4a-GRAPH_ID:0-PID:8887-HOST:son-PC_NAME:pool-
        client-6-RECON_NO:-0
        [2018-08-27 09:56:22.107747] I [MSGID: 101055]
        [client_t.c:444:gf_client_unref] 0-pool-server: Shutting
        down connection
        
CTX_ID:b55f4a90-e241-48ce-bd4d-268c8a956f4a-GRAPH_ID:0-PID:8887-HOST:son-PC_NAME:pool-clien
        t-6-RECON_NO:-0
        [2018-08-27 09:58:37.905829] I [MSGID: 115036]
        [server.c:483:server_rpc_notify] 0-pool-server:
        disconnecting connection from
        
CTX_ID:c3eb6cfc-2ef9-470a-89d1-a87170d00da5-GRAPH_ID:0-PID:30292-HOST:father-PC_NAME:p
        ool-client-6-RECON_NO:-0
        [2018-08-27 09:58:37.905926] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=28c831d8bc550000}
        [2018-08-27 09:58:37.905959] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=2870a7d6bc550000}
        [2018-08-27 09:58:37.905979] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=2880a7d6bc550000}
        [2018-08-27 09:58:37.905997] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=28f031d8bc550000}
        [2018-08-27 09:58:37.906016] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=28b07dd5bc550000}
        [2018-08-27 09:58:37.906034] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=28e0a7d6bc550000}
        [2018-08-27 09:58:37.906056] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=28b845d8bc550000}
        [2018-08-27 09:58:37.906079] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=2858a7d8bc550000}
        [2018-08-27 09:58:37.906098] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=2868a8d7bc550000}
        [2018-08-27 09:58:37.906121] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=28f80bd7bc550000}
        ...

        [2018-08-27 09:58:37.907375] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=28a8cdd6bc550000}
        [2018-08-27 09:58:37.907393] W
        [inodelk.c:610:pl_inodelk_log_cleanup] 0-pool-server:
        releasing lock on 12172afe-f0a4-4e10-bc0f-c5e4e0d9f318 held
        by {client=0x7ffb58035bc0, pid=30292 lk-owner=2880cdd6bc550000}
        [2018-08-27 09:58:37.907476] I
        [socket.c:3837:socket_submit_reply] 0-tcp.pool-server: not
        connected (priv->connected = -1)
        [2018-08-27 09:58:37.907520] E
        [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed
        to submit message (XID: 0xcb88cb, Program: GlusterFS 4.x v1,
        ProgVers: 400, Proc: 30) to rpc-transport (tcp.pool-server)
        [2018-08-27 09:58:37.910727] E
        [server.c:137:server_submit_reply]
        (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x20084)
        [0x7ffb64379084]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x605
        ba) [0x7ffb5fddf5ba]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce)
        [0x7ffb5fd89fce] ) 0-: Reply submission failed
        [2018-08-27 09:58:37.910814] E
        [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed
        to submit message (XID: 0xcb88ce, Program: GlusterFS 4.x v1,
        ProgVers: 400, Proc: 30) to rpc-transport (tcp.pool-server)
        [2018-08-27 09:58:37.910861] E
        [server.c:137:server_submit_reply]
        (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x20084)
        [0x7ffb64379084]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x605
        ba) [0x7ffb5fddf5ba]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce)
        [0x7ffb5fd89fce] ) 0-: Reply submission failed
        [2018-08-27 09:58:37.910904] E
        [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed
        to submit message (XID: 0xcb88cf, Program: GlusterFS 4.x v1,
        ProgVers: 400, Proc: 30) to rpc-transport (tcp.pool-server)
        [2018-08-27 09:58:37.910940] E
        [server.c:137:server_submit_reply]
        (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x20084)
        [0x7ffb64379084]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x605
        ba) [0x7ffb5fddf5ba]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce)
        [0x7ffb5fd89fce] ) 0-: Reply submission failed
        [2018-08-27 09:58:37.910979] E
        [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed
        to submit message (XID: 0xcb88d1, Program: GlusterFS 4.x v1,
        ProgVers: 400, Proc: 30) to rpc-transport (tcp.pool-server)
        [2018-08-27 09:58:37.911012] E
        [server.c:137:server_submit_reply]
        (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x20084)
        [0x7ffb64379084]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x605
        ba) [0x7ffb5fddf5ba]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce)
        [0x7ffb5fd89fce] ) 0-: Reply submission failed
        [2018-08-27 09:58:37.911050] E
        [rpcsvc.c:1378:rpcsvc_submit_generic] 0-rpc-service: failed
        to submit message (XID: 0xcb88d8, Program: GlusterFS 4.x v1,
        ProgVers: 400, Proc: 30) to rpc-transport (tcp.pool-server)
        [2018-08-27 09:58:37.911083] E
        [server.c:137:server_submit_reply]
        (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x20084)
        [0x7ffb64379084]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x605
        ba) [0x7ffb5fddf5ba]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce)
        [0x7ffb5fd89fce] ) 0-: Reply submission failed
        [2018-08-27 09:58:37.916217] E
        [server.c:137:server_submit_reply]
        (-->/usr/lib64/glusterfs/4.1.2/xlator/debug/io-stats.so(+0x20084)
        [0x7ffb64379084]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0x605
        ba) [0x7ffb5fddf5ba]
        -->/usr/lib64/glusterfs/4.1.2/xlator/protocol/server.so(+0xafce)
        [0x7ffb5fd89fce] ) 0-: Reply submission failed
        [2018-08-27 09:58:37.916520] I [MSGID: 115013]
        [server-helpers.c:286:do_fd_cleanup] 0-pool-server: fd
        cleanup on /balamak.img


        after this I/O on  /balamak.img was blocked.


        Only solution we found was to reboot all 3 nodes.


        Is there any bug report in bugzilla we can add logs?


    Not aware of such bugs!

        Is it possible to turn of these locks?


    Not sure, will get back on this one!



    btw, found this link
    https://docs.gluster.org/en/v3/Troubleshooting/troubleshooting-filelocks/

    tried on another (test) cluster:

     [root@marduk ~]# gluster volume statedump pool
    Segmentation fault (core dumped)


    4.1.2 too...

    something is wrong here.



        Thank you!




        _______________________________________________
        Gluster-users mailing list
        [email protected] <mailto:[email protected]>
        https://lists.gluster.org/mailman/listinfo/gluster-users
-- Amar Tumballi (amarts) 


    _______________________________________________
    Gluster-users mailing list
    [email protected] <mailto:[email protected]>
    https://lists.gluster.org/mailman/listinfo/gluster-users





_______________________________________________
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users



_______________________________________________
Gluster-users mailing list
[email protected]
https://lists.gluster.org/mailman/listinfo/gluster-users

Reply via email to