I've been seeing the same thing happen, and in our case, it's because of running a script that checks gluster from time to time ( https://github.com/jtopjian/scripts/blob/master/gluster/gluster-status.sh in our case).
Do you have a job that runs and periodically checks for gluster health? Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | @ArtemR <http://twitter.com/ArtemR> On Fri, Feb 14, 2020 at 3:10 AM Christian Reiss <em...@christian-reiss.de> wrote: > Hey folks, > > my logs are constantly (every few secs, continuously) swamped with > > [2020-02-14 11:05:20.258542] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-0: > Connected to ssd_storage-client-0, attached to remote volume > '/gluster_bricks/node01.company.com/gluster'. > [2020-02-14 11:05:20.258559] I [MSGID: 108005] > [afr-common.c:5280:__afr_handle_child_up_event] > 0-ssd_storage-replicate-0: Subvolume 'ssd_storage-client-0' came back > up; going online. > [2020-02-14 11:05:20.258920] I [rpc-clnt.c:1963:rpc_clnt_reconfig] > 0-ssd_storage-client-2: changing port to 49152 (from 0) > [2020-02-14 11:05:20.259132] I [socket.c:864:__socket_shutdown] > 0-ssd_storage-client-2: intentional socket shutdown(11) > [2020-02-14 11:05:20.260010] I [MSGID: 114057] > [client-handshake.c:1376:select_server_supported_programs] > 0-ssd_storage-client-1: Using Program GlusterFS 4.x v1, Num (1298437), > Version (400) > [2020-02-14 11:05:20.261077] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-1: > Connected to ssd_storage-client-1, attached to remote volume > '/gluster_bricks/node02.company.com/gluster'. > [2020-02-14 11:05:20.261089] I [MSGID: 108002] > [afr-common.c:5647:afr_notify] 0-ssd_storage-replicate-0: Client-quorum > is met > [2020-02-14 11:05:20.262005] I [MSGID: 114057] > [client-handshake.c:1376:select_server_supported_programs] > 0-ssd_storage-client-2: Using Program GlusterFS 4.x v1, Num (1298437), > Version (400) > [2020-02-14 11:05:20.262685] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-2: > Connected to ssd_storage-client-2, attached to remote volume > '/gluster_bricks/node03.company.com/gluster'. > [2020-02-14 11:05:20.263909] I [MSGID: 108031] > [afr-common.c:2580:afr_local_discovery_cbk] 0-ssd_storage-replicate-0: > selecting local read_child ssd_storage-client-0 > [2020-02-14 11:05:20.264124] I [MSGID: 104041] > [glfs-resolve.c:954:__glfs_active_subvol] 0-ssd_storage: switched to > graph 6e6f6465-3031-2e64-632d-6475732e6461 (0) > [2020-02-14 11:05:22.407851] I [MSGID: 114007] > [client.c:2478:client_check_remote_host] 0-ssd_storage-snapd-client: > Remote host is not set. Assuming the volfile server as remote host > [Invalid argument] > [2020-02-14 11:05:22.409711] I [MSGID: 104045] [glfs-master.c:80:notify] > 0-gfapi: New graph 6e6f6465-3031-2e64-632d-6475732e6461 (0) coming up > [2020-02-14 11:05:22.409738] I [MSGID: 114020] [client.c:2436:notify] > 0-ssd_storage-client-0: parent translators are ready, attempting connect > on transport > [2020-02-14 11:05:22.412949] I [MSGID: 114020] [client.c:2436:notify] > 0-ssd_storage-client-1: parent translators are ready, attempting connect > on transport > [2020-02-14 11:05:22.413130] I [rpc-clnt.c:1963:rpc_clnt_reconfig] > 0-ssd_storage-client-0: changing port to 49152 (from 0) > [2020-02-14 11:05:22.413154] I [socket.c:864:__socket_shutdown] > 0-ssd_storage-client-0: intentional socket shutdown(10) > [2020-02-14 11:05:22.415534] I [MSGID: 114020] [client.c:2436:notify] > 0-ssd_storage-client-2: parent translators are ready, attempting connect > on transport > [2020-02-14 11:05:22.417836] I [MSGID: 114057] > [client-handshake.c:1376:select_server_supported_programs] > 0-ssd_storage-client-0: Using Program GlusterFS 4.x v1, Num (1298437), > Version (400) > [2020-02-14 11:05:22.418036] I [rpc-clnt.c:1963:rpc_clnt_reconfig] > 0-ssd_storage-client-1: changing port to 49152 (from 0) > [2020-02-14 11:05:22.418095] I [socket.c:864:__socket_shutdown] > 0-ssd_storage-client-1: intentional socket shutdown(12) > [2020-02-14 11:05:22.420029] I [MSGID: 114020] [client.c:2436:notify] > 0-ssd_storage-snapd-client: parent translators are ready, attempting > connect on transport > [2020-02-14 11:05:22.420533] E [MSGID: 101075] > [common-utils.c:505:gf_resolve_ip6] 0-resolver: getaddrinfo failed > (family:2) (Name or service not known) > [2020-02-14 11:05:22.420545] E > [name.c:266:af_inet_client_get_remote_sockaddr] > 0-ssd_storage-snapd-client: DNS resolution failed on host > /var/run/glusterd.socket > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume ssd_storage-client-0 > 2: type protocol/client > 3: option opversion 70000 > 4: option clnt-lk-version 1 > 5: option volfile-checksum 0 > 6: option volfile-key ssd_storage > 7: option client-version 7.0 > 8: option process-name gfapi.glfsheal > 9: option process-uuid > > CTX_ID:50cec79e-6028-4e6f-b8ed-dda9db36b2d0-GRAPH_ID:0-PID:24926-HOST:node01.company.com-PC_NAME:ssd_storage-client-0-RECON_NO:-0 > 10: option fops-version 1298437 > 11: option ping-timeout 42 > 12: option remote-host node01.company.com > 13: option remote-subvolume /gluster_bricks/ > node01.company.com/gluster > 14: option transport-type socket > 15: option transport.address-family inet > 16: option username 96bcf4d4-932f-4654-86c3-470a081d5021 > 17: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec > 18: option transport.socket.ssl-enabled off > 19: option transport.tcp-user-timeout 0 > 20: option transport.socket.keepalive-time 20 > 21: option transport.socket.keepalive-interval 2 > 22: option transport.socket.keepalive-count 9 > 23: option send-gids true > 24: end-volume > 25: > 26: volume ssd_storage-client-1 > 27: type protocol/client > 28: option ping-timeout 42 > 29: option remote-host node02.company.com > 30: option remote-subvolume /gluster_bricks/ > node02.company.com/gluster > 31: option transport-type socket > 32: option transport.address-family inet > 33: option username 96bcf4d4-932f-4654-86c3-470a081d5021 > 34: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec > 35: option transport.socket.ssl-enabled off > 36: option transport.tcp-user-timeout 0 > 37: option transport.socket.keepalive-time 20 > 38: option transport.socket.keepalive-interval 2 > 39: option transport.socket.keepalive-count 9 > 40: option send-gids true > 41: end-volume > 42: > 43: volume ssd_storage-client-2 > 44: type protocol/client > 45: option ping-timeout 42 > 46: option remote-host node03.company.com > 47: option remote-subvolume /gluster_bricks/ > node03.company.com/gluster > 48: option transport-type socket > 49: option transport.address-family inet > 50: option username 96bcf4d4-932f-4654-86c3-470a081d5021 > 51: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec > 52: option transport.socket.ssl-enabled off > 53: option transport.tcp-user-timeout 0 > 54: option transport.socket.keepalive-time 20 > 55: option transport.socket.keepalive-interval 2 > 56: option transport.socket.keepalive-count 9 > 57: option send-gids true > 58: end-volume > 59: > 60: volume ssd_storage-replicate-0 > 61: type cluster/replicate > 62: option background-self-heal-count 0 > 63: option afr-pending-xattr > ssd_storage-client-0,ssd_storage-client-1,ssd_storage-client-2 > 64: option metadata-self-heal on > 65: option data-self-heal on > 66: option entry-self-heal on > 67: option data-self-heal-algorithm full > 68: option use-compound-fops off > 69: subvolumes ssd_storage-client-0 ssd_storage-client-1 > ssd_storage-client-2 > 70: end-volume > 71: > 72: volume ssd_storage-dht > 73: type cluster/distribute > 74: option readdir-optimize on > 75: option lock-migration off > 76: option force-migration off > 77: subvolumes ssd_storage-replicate-0 > 78: end-volume > 79: > 80: volume ssd_storage-utime > 81: type features/utime > 82: option noatime on > 83: subvolumes ssd_storage-dht > 84: end-volume > 85: > 86: volume ssd_storage-write-behind > 87: type performance/write-behind > 88: subvolumes ssd_storage-utime > 89: end-volume > 90: > 91: volume ssd_storage-read-ahead > 92: type performance/read-ahead > 93: subvolumes ssd_storage-write-behind > 94: end-volume > 95: > 96: volume ssd_storage-readdir-ahead > 97: type performance/readdir-ahead > 98: option parallel-readdir off > 99: option rda-request-size 131072 > 100: option rda-cache-limit 10MB > 101: subvolumes ssd_storage-read-ahead > 102: end-volume > 103: > 104: volume ssd_storage-io-cache > 105: type performance/io-cache > 106: subvolumes ssd_storage-readdir-ahead > 107: end-volume > 108: > 109: volume ssd_storage-open-behind > 110: type performance/open-behind > 111: subvolumes ssd_storage-io-cache > 112: end-volume > 113: > 114: volume ssd_storage-quick-read > 115: type performance/quick-read > 116: subvolumes ssd_storage-open-behind > 117: end-volume > 118: > 119: volume ssd_storage-md-cache > 120: type performance/md-cache > 121: subvolumes ssd_storage-quick-read > 122: end-volume > 123: > 124: volume ssd_storage-snapd-client > 125: type protocol/client > 126: option remote-host /var/run/glusterd.socket > 127: option ping-timeout 42 > 128: option remote-subvolume snapd-ssd_storage > 129: option transport-type socket > 130: option transport.address-family inet > 131: option username 96bcf4d4-932f-4654-86c3-470a081d5021 > 132: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec > 133: option transport.socket.ssl-enabled off > 134: option transport.tcp-user-timeout 0 > 135: option transport.socket.keepalive-time 20 > 136: option transport.socket.keepalive-interval 2 > 137: option transport.socket.keepalive-count 9 > 138: option send-gids true > 139: end-volume > 140: > 141: volume ssd_storage-snapview-client > 142: type features/snapview-client > 143: option snapshot-directory .snaps > 144: option show-snapshot-directory on > 145: subvolumes ssd_storage-md-cache ssd_storage-snapd-client > 146: end-volume > 147: > 148: volume ssd_storage > 149: type debug/io-stats > 150: option log-level INFO > 151: option threads 16 > 152: option latency-measurement off > 153: option count-fop-hits off > 154: option global-threading off > 155: subvolumes ssd_storage-snapview-client > 156: end-volume > 157: > 158: volume meta-autoload > 159: type meta > 160: subvolumes ssd_storage > 161: end-volume > 162: > > +------------------------------------------------------------------------------+ > [2020-02-14 11:05:22.421366] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-0: > Connected to ssd_storage-client-0, attached to remote volume > '/gluster_bricks/node01.company.com/gluster'. > [2020-02-14 11:05:22.421379] I [MSGID: 108005] > [afr-common.c:5280:__afr_handle_child_up_event] > 0-ssd_storage-replicate-0: Subvolume 'ssd_storage-client-0' came back > up; going online. > [2020-02-14 11:05:22.421669] I [rpc-clnt.c:1963:rpc_clnt_reconfig] > 0-ssd_storage-client-2: changing port to 49152 (from 0) > [2020-02-14 11:05:22.421686] I [socket.c:864:__socket_shutdown] > 0-ssd_storage-client-2: intentional socket shutdown(11) > [2020-02-14 11:05:22.422460] I [MSGID: 114057] > [client-handshake.c:1376:select_server_supported_programs] > 0-ssd_storage-client-1: Using Program GlusterFS 4.x v1, Num (1298437), > Version (400) > [2020-02-14 11:05:22.423377] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-1: > Connected to ssd_storage-client-1, attached to remote volume > '/gluster_bricks/node02.company.com/gluster'. > [2020-02-14 11:05:22.423391] I [MSGID: 108002] > [afr-common.c:5647:afr_notify] 0-ssd_storage-replicate-0: Client-quorum > is met > [2020-02-14 11:05:22.424586] I [MSGID: 114057] > [client-handshake.c:1376:select_server_supported_programs] > 0-ssd_storage-client-2: Using Program GlusterFS 4.x v1, Num (1298437), > Version (400) > [2020-02-14 11:05:22.425323] I [MSGID: 114046] > [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-2: > Connected to ssd_storage-client-2, attached to remote volume > '/gluster_bricks/node03.company.com/gluster'. > [2020-02-14 11:05:22.426613] I [MSGID: 108031] > [afr-common.c:2580:afr_local_discovery_cbk] 0-ssd_storage-replicate-0: > selecting local read_child ssd_storage-client-0 > [2020-02-14 11:05:22.426758] I [MSGID: 104041] > [glfs-resolve.c:954:__glfs_active_subvol] 0-ssd_storage: switched to > graph 6e6f6465-3031-2e64-632d-6475732e6461 (0) > > > Can you guys make any sense out of this? 5 unsynced entries remain. > > -- > with kind regards, > mit freundlichen Gruessen, > > Christian Reiss > > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
