[Gluster-devel] Was using the file system, and then apparently lost connection GF_EVENT_POLLERR

Sun, 01 Jul 2007 04:11:21 -0700 


everything was working fine, and then i lost access to my file system.
I looked at logs and it appears I lost connectivity to my server bricks,
but i could ping them and telnet to port 6996
They are all on the same subnet, and no network problems and no sort of
glusterfs recovery occurred.

Script started on Sun 01 Jul 2007 02:54:02 AM PDT
/tmp/1 is my glusterfs client mountpoint



[EMAIL PROTECTED] 1]# ls
ls: cannot open directory .: Transport endpoint is not connected
]0;[EMAIL PROTECTED]:/tmp/1 [EMAIL PROTECTED] 1]# cd /usr/local/var/log/glusterfs/ 2007-06-30 08:54:55 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-06-30 08:54:56 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-06-30 08:54:56 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-06-30 08:55:01 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-06-30 09:20:07 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-06-30 09:20:08 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-06-30 09:20:08 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-06-30 09:20:15 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-06-30 09:20:53 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-06-30 09:24:28 E [protocol.c:262:gf_block_unserialize_transport] libglusterfs/protocol: full_read of header failed: peer (0.0.0.0) 2007-06-30 09:24:28 C [tcp.c:81:tcp_disconnect] glfsd115-250GB-truecrypt: connection disconnected 2007-06-30 09:24:28 E [client-protocol.c:327:client_protocol_xfer] glfsd115-250GB-truecrypt: transport_submit failed 2007-06-30 09:24:31 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:24:34 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:24:39 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:24:47 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:25:00 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:25:21 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:25:55 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:26:50 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:28:19 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:30:43 E [tcp-client.c:171:tcp_connect] glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection refused) 2007-06-30 09:32:10 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-06-30 09:32:31 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (11), printing backtrace 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(gf_print_trace+0x2d) [0x84d94d] 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xc33420] 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(transport_notify+0x37) [0x84ee47] 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9) [0x84f969] 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(poll_iteration+0x1d) [0x84ef1d] 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x8049e70] 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/libc.so.6(__libc_start_main+0xe0) [0x125f70] 2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x8049a41] 2007-06-30 09:35:53 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-06-30 09:49:02 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-06-30 10:15:14 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-07-01 01:54:43 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:43 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:44 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:44 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:50 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENT_POLLERR 2007-07-01 02:52:06 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (11), printing backtrace 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(gf_print_trace+0x2d) [0xada94d] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xf53420] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/glusterfs/1.3.0-pre5/xlator/cluster/afr.so [0xf91225] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(default_lookup+0x85) [0xada795] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(default_lookup+0x85) [0xada795] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x804d3d6] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libfuse.so.2 [0x8ded61] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libfuse.so.2 [0x8df7a3] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libfuse.so.2(fuse_session_process+0x26) [0x8e0fa6] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x804a740] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(transport_notify+0x37) [0xadbe47] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9) [0xadc969] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libglusterfs.so.0(poll_iteration+0x1d) [0xadbf1d] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x8049e70] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/libc.so.6(__libc_start_main+0xe0) [0x58af70] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x8049a41] ]0;[EMAIL PROTECTED]:/usr/local/var/log/glusterfs [EMAIL PROTECTED] glusterfs]# vi !$ 
vi glusterfs.log
[?1049h[?1h=[?12;25h[?12l[?25h[?25l"glusterfs.log"
"glusterfs.log" 103L, 13037C2007-06-30 04:29:08 E [xlator.c:115:xlator_set_type] libglusterfs/xlator: dlopen(/usr/local/llib/glusterfs/1.3.0-pre5/xlator/performance/stat-prefetch.so): /usr/local/lib/glusterfs/1.3.0--pre5/xlator/performance/stat-prefetch.so: cannot open shared object file: No such file or dirrectory 2007-06-30 04:41:27 C [unify.c:4379:init] bricks: namespace option not specified, Exiting 2007-06-30 04:41:27 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (11), pprinting backtrace 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(gf_print_trace+0x2d) [0xef994d] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xc8d420] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(default_notify+0x5b) [0xef7bbb] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(default_notify+0x5b) [0xef7bbb] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(default_notify+0x5b) [0xef7bbb] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(xlator_tree_init+0x53) [0xef6c73] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x8044d90d] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbfuse.so.2 [0xe86bef] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbfuse.so.2 [0xe867a3] 2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/lib@ [?12l[?25h[?25l2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(default_lookup+0x85) [0xada795] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x8044d3d6] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbfuse.so.2 [0x8ded61] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbfuse.so.2 [0x8df7a3] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbfuse.so.2(fuse_session_process+0x26) [0x8e0fa6] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x8044a740] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(transport_notify+0x37) [0xadbe47] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(sys_epoll_iteration+0xd9) [0xadc969] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(poll_iteration+0x1d) [0xadbf1d] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x80449e70] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/libc.so.6(__llibc_start_main+0xe0) [0x58af70] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [glusterfs] [0x80449a41] ~ [?12l[?25h[?25l2007-07-01 01:54:44 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: not in progress - trace: Network is unreachable 2007-07-01 01:54:50 E [fuse-bridge.c:1910:fuse_transport_notify] glusterfs-fuse: got GF_EVENTT_POLLERR 2007-07-01 02:52:06 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (11), pprinting backtrace 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(gf_print_trace+0x2d) [0xada94d] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xf53420] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/gluusterfs/1.3.0-pre5/xlator/cluster/afr.so [0xf91225] 2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/local/lib/libbglusterfs.so.0(default_lookup+0x85) [0xada795] 


[?12l[?25h

[?25l:[?12l[?25hq
[EMAIL PROTECTED]:/usr/local/var/log/glusterfs [
[EMAIL PROTECTED] glusterfs]# ping device128
PING device128 (192.168.1.128) 56(84) bytes of data.
64 bytes from 192.168.1.128: icmp_seq=1 ttl=64 time=2.49 ms
64 bytes from 192.168.1.128: icmp_seq=2 ttl=64 time=0.354 ms

--- device128 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.354/1.425/2.497/1.072 ms
]0;[EMAIL PROTECTED]:/usr/local/var/log/glusterfs [EMAIL PROTECTED] glusterfs]# telnet device128 6996 
Trying 192.168.1.128...
Connected to device128.
Escape character is '^]'.

^]

telnet> quit
Connection closed.
]0;[EMAIL PROTECTED]:/usr/local/var/log/glusterfs [EMAIL PROTECTED] glusterfs]#
Script done on Sun 01 Jul 2007 02:55:32 AM PDT

_________________________________________________________________
Like puzzles? Play free games & earn great prizes. Play Clink now. http://club.live.com/clink.aspx?icid=clink_hotmailtextlink2 



_______________________________________________
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel

Reply via email to