I am also seeing a bunch of these errors in the logs. Do they really need to be Info logs?
<snip> [2018-11-14 06:01:18.597029] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 382 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.597640] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 385 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.598114] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 344 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.598737] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 343 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.599172] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 384 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.599789] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 386 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.600237] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 387 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.579405] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 324 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.563771] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 187 (errno:11:Resource temporarily unavailable); returning ENODATA [2018-11-14 06:01:18.560416] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 161 (errno:0:Success); returning ENODATA [2018-11-14 06:01:18.564826] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 194 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.561503] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 306 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.567985] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 164 (errno:0:Success); returning ENODATA [2018-11-14 06:01:18.566924] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 198 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.562567] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 346 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:18.580451] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 363 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:23.165797] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 346 (errno:11:Resource temporarily unavailable); returning ENODATA [2018-11-14 06:01:23.166019] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 207 (errno:0:Success); returning ENODATA [2018-11-14 06:01:23.166905] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 197 (errno:0:Success); returning ENODATA [2018-11-14 06:01:23.167541] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 273 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:23.167904] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 275 (errno:61:No data available); returning ENODATA [2018-11-14 06:01:23.168502] I [socket.c:693:__socket_rwv] 0-tcp.patchy-vol01-server: EOF on socket 277 (errno:61:No data available); returning ENODATA </snip> [nbalacha@dhcp35-62 bricks]$ grep "EOF on socket" d-backends-vol01-brick* |wc -l 1580 Regards, Nithya On 14 November 2018 at 21:03, Shyam Ranganathan <[email protected]> wrote: > On 11/14/2018 10:04 AM, Nithya Balachandran wrote: > > Hi Mohit, > > > > The regression run in the subject has failed because a brick has crashed > in > > > > bug-1432542-mpx-restart-crash.t > > > > > > *06:03:38* 1 test(s) generated core > > *06:03:38* ./tests/bugs/core/bug-1432542-mpx-restart-crash.t > > *06:03:38* > > > > > > The brick process has crashed in posix_fs_health_check as this->priv > > contains garbage. It looks like it might have been freed already. Can > > you take a look at it? > > Sounds like another incarnation of: > https://bugzilla.redhat.com/show_bug.cgi?id=1636570 > > @mohit, any further clues? > > > > > > > > > (gdb) bt > > #0 0x00007f4019ea1f19 in vfprintf () from ./lib64/libc.so.6 > > #1 0x00007f4019eccf49 in vsnprintf () from ./lib64/libc.so.6 > > #2 0x00007f401b87705a in gf_vasprintf (string_ptr=0x7f3e81ff99f0, > > format=0x7f400df32f40 "op=%s;path=%s;error=%s;brick=%s:%s timeout is > > %d", arg=0x7f3e81ff99f8) > > at > > /home/jenkins/root/workspace/centos7-regression/ > libglusterfs/src/mem-pool.c:234 > > #3 0x00007f401b8de6e2 in _gf_event > > (event=EVENT_POSIX_HEALTH_CHECK_FAILED, fmt=0x7f400df32f40 > > "op=%s;path=%s;error=%s;brick=%s:%s timeout is %d") > > at > > /home/jenkins/root/workspace/centos7-regression/ > libglusterfs/src/events.c:89 > > #4 0x00007f400def07f9 in posix_fs_health_check (this=0x7f3fd78b7840) at > > /home/jenkins/root/workspace/centos7-regression/xlators/ > storage/posix/src/posix-helpers.c:1960 > > #5 0x00007f400def0926 in posix_health_check_thread_proc > > (data=0x7f3fd78b7840) > > at > > /home/jenkins/root/workspace/centos7-regression/xlators/ > storage/posix/src/posix-helpers.c:2005 > > #6 0x00007f401a68ae25 in start_thread () from ./lib64/libpthread.so.0 > > #7 0x00007f4019f53bad in clone () from ./lib64/libc.so.6 > > (gdb) f 4 > > #4 0x00007f400def07f9 in posix_fs_health_check (this=0x7f3fd78b7840) at > > /home/jenkins/root/workspace/centos7-regression/xlators/ > storage/posix/src/posix-helpers.c:1960 > > 1960 gf_event(EVENT_POSIX_HEALTH_CHECK_FAILED, > > (gdb) l > > 1955 sys_close(fd); > > 1956 } > > 1957 if (ret && file_path[0]) { > > 1958 gf_msg(this->name, GF_LOG_WARNING, errno, > > P_MSG_HEALTHCHECK_FAILED, > > 1959 "%s() on %s returned", op, file_path); > > 1960 gf_event(EVENT_POSIX_HEALTH_CHECK_FAILED, > > 1961 "op=%s;path=%s;error=%s;brick=%s:%s timeout is > %d", op, > > 1962 file_path, strerror(op_errno), priv->hostname, > > priv->base_path, > > 1963 timeout); > > 1964 } > > (gdb) p pri->hostname > > No symbol "pri" in current context. > > *(gdb) p priv->hostname* > > *$14 = 0xa200 <error: Cannot access memory at address 0xa200>* > > *(gdb) p priv->base_path* > > *$15 = 0x7f3ddeadc0de00 <error: Cannot access memory at address > > 0x7f3ddeadc0de00>* > > (gdb) > > > > > > > > Thanks, > > Nithya > > > > > > _______________________________________________ > > Gluster-devel mailing list > > [email protected] > > https://lists.gluster.org/mailman/listinfo/gluster-devel > > >
_______________________________________________ Gluster-devel mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-devel
