On 07/09/2012 09:58 AM, Liu Yuan wrote:
> Got an weird segfault,
>
> (gdb) where
> #0 0x0000000000411936 in do_process_work (work=0xd13c70) at ops.c:992
> #1 0x000000000040ed05 in worker_routine (arg=0xd12a20) at work.c:171
> #2 0x00007f43f992c971 in start_thread (arg=<value optimized out>) at
> pthread_create.c:304
> #3 0x00007f43f8eeef3d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #4 0x0000000000000000 in ?? ()
>
> sheep.log:
> ...
> Jul 09 09:47:23 [main] client_handler(764) connection seems to be dead
> Jul 09 09:47:23 [main] clear_client(703) refcnt:0, fd:14, ::1:43328
> Jul 09 09:47:23 [main] destroy_client(672) connection from: ::1:43328
> Jul 09 09:47:23 [main] cdrv_cpg_deliver(448) 5
> Jul 09 09:47:23 [main] sd_notify_handler(851) size: 96, from: IPv4
> ip:127.0.0.1 port:7000
> Jul 09 09:47:23 [main] client_tx_handler(663) connection from: 13, ::1:43330
> Jul 09 09:47:23 [main] client_handler(764) connection seems to be dead
> Jul 09 09:47:23 [main] clear_client(703) refcnt:0, fd:13, ::1:43330
> Jul 09 09:47:23 [main] destroy_client(672) connection from: ::1:43330
> Jul 09 09:47:23 [main] listen_handler(819) accepted a new connection: 13
> Jul 09 09:47:23 [main] listen_handler(819) accepted a new connection: 14
> Jul 09 09:47:23 [block] do_process_work(990) 80, 0 , 32579 <--- XXX
> Jul 09 09:47:23 [main] client_rx_handler(577) connection from: 14, ::1:43337
> Jul 09 09:47:23 [main] queue_request(323) 2
> Jul 09 09:47:23 [main] crash_handler(408) sheep pid 5326 exited
> unexpectedly.
>
> Thanks,
> Yuan
>
Yet another segfault.
#0 __libc_free (mem=0x7f3301864000) at malloc.c:3709
3709 malloc.c: No such file or directory.
in malloc.c
(gdb) where
#0 __libc_free (mem=0x7f3301864000) at malloc.c:3709
#1 0x00000000004090a1 in free_request (req=0x7f32fc000a00) at sdnet.c:474
#2 0x00000000004098bd in client_tx_handler (ci=0x7f32fc0143c0) at
sdnet.c:656
#3 0x0000000000409d32 in client_handler (fd=14, events=4,
data=0x7f32fc0143c0) at sdnet.c:760
#4 0x000000000041e470 in event_loop (timeout=-1) at event.c:179
#5 0x0000000000404376 in main (argc=7, argv=0x7fff9f1566a8) at sheep.c:275
Jul 09 10:37:17 [main] create_cluster(1092) use corosync cluster driver as default
Jul 09 10:37:17 [main] create_cluster(1121) zone id = 0
Jul 09 10:37:17 [main] send_join_request(961) IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] main(272) sheepdog daemon (version 0.3.0_482_g7d18217_dirty) started
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:1, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] sd_check_join_cb(894) IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 0
Jul 09 10:37:17 [main] sockfd_cache_add_group(242) 1
Jul 09 10:37:17 [main] sd_join_handler(1001) join Sheepdog cluster
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:2, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:3, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] sd_check_join_cb(918) 127.0.0.1:7001: ret = 0x0, cluster_status = 0x2
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [1] IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 1
Jul 09 10:37:17 [main] sockfd_cache_add(271) 127.0.0.1:7001, count 2
Jul 09 10:37:17 [main] sd_check_join_cb(918) 127.0.0.1:7002: ret = 0x0, cluster_status = 0x2
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:4, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7002
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [1] IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] sd_join_handler(992) [2] IPv4 ip:127.0.0.1 port:7002
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 1
Jul 09 10:37:17 [main] sockfd_cache_add(271) 127.0.0.1:7002, count 3
Jul 09 10:37:17 [main] sd_check_join_cb(918) 127.0.0.1:7003: ret = 0x0, cluster_status = 0x2
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:5, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7003
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [1] IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] sd_join_handler(992) [2] IPv4 ip:127.0.0.1 port:7002
Jul 09 10:37:17 [main] sd_join_handler(992) [3] IPv4 ip:127.0.0.1 port:7003
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 1
Jul 09 10:37:17 [main] sockfd_cache_add(271) 127.0.0.1:7003, count 4
Jul 09 10:37:17 [main] sd_check_join_cb(918) 127.0.0.1:7004: ret = 0x0, cluster_status = 0x2
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:6, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7004
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [1] IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] sd_join_handler(992) [2] IPv4 ip:127.0.0.1 port:7002
Jul 09 10:37:17 [main] sd_join_handler(992) [3] IPv4 ip:127.0.0.1 port:7003
Jul 09 10:37:17 [main] sd_join_handler(992) [4] IPv4 ip:127.0.0.1 port:7004
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 1
Jul 09 10:37:17 [main] sockfd_cache_add(271) 127.0.0.1:7004, count 5
Jul 09 10:37:17 [main] sd_check_join_cb(918) 127.0.0.1:7006: ret = 0x0, cluster_status = 0x2
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7006
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [1] IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] sd_join_handler(992) [2] IPv4 ip:127.0.0.1 port:7002
Jul 09 10:37:17 [main] sd_join_handler(992) [3] IPv4 ip:127.0.0.1 port:7003
Jul 09 10:37:17 [main] sd_join_handler(992) [4] IPv4 ip:127.0.0.1 port:7004
Jul 09 10:37:17 [main] sd_join_handler(992) [5] IPv4 ip:127.0.0.1 port:7006
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 1
Jul 09 10:37:17 [main] sockfd_cache_add(271) 127.0.0.1:7006, count 6
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:7, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] sd_check_join_cb(918) 127.0.0.1:7007: ret = 0x0, cluster_status = 0x2
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7007
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [1] IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] sd_join_handler(992) [2] IPv4 ip:127.0.0.1 port:7002
Jul 09 10:37:17 [main] sd_join_handler(992) [3] IPv4 ip:127.0.0.1 port:7003
Jul 09 10:37:17 [main] sd_join_handler(992) [4] IPv4 ip:127.0.0.1 port:7004
Jul 09 10:37:17 [main] sd_join_handler(992) [5] IPv4 ip:127.0.0.1 port:7006
Jul 09 10:37:17 [main] sd_join_handler(992) [6] IPv4 ip:127.0.0.1 port:7007
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 1
Jul 09 10:37:17 [main] sockfd_cache_add(271) 127.0.0.1:7007, count 7
Jul 09 10:37:17 [main] cdrv_cpg_confchg(562) mem:8, joined:1, left:0
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 0
Jul 09 10:37:17 [main] sd_check_join_cb(918) 127.0.0.1:7005: ret = 0x0, cluster_status = 0x2
Jul 09 10:37:17 [main] cdrv_cpg_deliver(448) 1
Jul 09 10:37:17 [main] sd_join_handler(990) join IPv4 ip:127.0.0.1 port:7005
Jul 09 10:37:17 [main] sd_join_handler(992) [0] IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:17 [main] sd_join_handler(992) [1] IPv4 ip:127.0.0.1 port:7001
Jul 09 10:37:17 [main] sd_join_handler(992) [2] IPv4 ip:127.0.0.1 port:7002
Jul 09 10:37:17 [main] sd_join_handler(992) [3] IPv4 ip:127.0.0.1 port:7003
Jul 09 10:37:17 [main] sd_join_handler(992) [4] IPv4 ip:127.0.0.1 port:7004
Jul 09 10:37:17 [main] sd_join_handler(992) [5] IPv4 ip:127.0.0.1 port:7006
Jul 09 10:37:17 [main] sd_join_handler(992) [6] IPv4 ip:127.0.0.1 port:7007
Jul 09 10:37:17 [main] sd_join_handler(992) [7] IPv4 ip:127.0.0.1 port:7005
Jul 09 10:37:17 [main] update_cluster_info(778) status = 2, epoch = 0, finished: 1
Jul 09 10:37:17 [main] sockfd_cache_add(271) 127.0.0.1:7005, count 8
Jul 09 10:37:20 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:20 [main] client_rx_handler(577) connection from: 13, ::1:45690
Jul 09 10:37:20 [main] queue_request(323) 84
Jul 09 10:37:20 [main] queue_cluster_request(338) 0x108d100 84
Jul 09 10:37:20 [main] cdrv_cpg_deliver(448) 3
Jul 09 10:37:20 [main] sd_notify_handler(851) size: 101, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:20 [main] farm_format(732) try get a clean store
Jul 09 10:37:20 [main] trunk_reset(402) clean
Jul 09 10:37:20 [main] farm_init(333) use farm store driver
Jul 09 10:37:20 [main] init_sys_vdi_bitmap(306) found the working directory /home/tailai.ly/sheepdog/store/0/obj/
Jul 09 10:37:21 [main] update_epoch_log(55) update epoch: 1, 8
Jul 09 10:37:22 [main] have_enough_zones(89) flags 1, nr_zones 8, copies 3
Jul 09 10:37:22 [main] client_tx_handler(663) connection from: 13, ::1:45690
Jul 09 10:37:22 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:22 [main] clear_client(703) refcnt:0, fd:13, ::1:45690
Jul 09 10:37:22 [main] destroy_client(672) connection from: ::1:45690
Jul 09 10:37:23 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:23 [main] client_rx_handler(577) connection from: 13, ::1:45691
Jul 09 10:37:23 [main] queue_request(323) 82
Jul 09 10:37:23 [io 1] do_process_work(990) 82, 0 , 1
Jul 09 10:37:23 [main] client_tx_handler(663) connection from: 13, ::1:45691
Jul 09 10:37:23 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:23 [main] clear_client(703) refcnt:0, fd:13, ::1:45691
Jul 09 10:37:23 [main] destroy_client(672) connection from: ::1:45691
Jul 09 10:37:23 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:23 [main] client_rx_handler(577) connection from: 13, ::1:45692
Jul 09 10:37:23 [main] queue_request(323) 11
Jul 09 10:37:23 [main] queue_cluster_request(338) 0x108dbb0 11
Jul 09 10:37:23 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:23 [block] do_process_work(990) 11, 6400000 , 1
Jul 09 10:37:23 [block] do_lookup_vdi(209) looking for test0 (fd34af)
Jul 09 10:37:23 [block] add_vdi(302) creating new vdi test0: size 104857600, vid fd34af, base 0, cur 0
Jul 09 10:37:23 [main] queue_request(323) 1
Jul 09 10:37:23 [gateway 2] do_process_work(990) 1, 80fd34af00000000 , 1
Jul 09 10:37:23 [gateway 2] do_gateway_write_obj(231) 80fd34af00000000
Jul 09 10:37:23 [gateway 2] sockfd_cache_get(290) create connection 127.0.0.1:7001 idx 0
Jul 09 10:37:23 [gateway 2] connect_to(256) 15, 127.0.0.1:7001
Jul 09 10:37:23 [gateway 2] sockfd_cache_get(290) create connection 127.0.0.1:7005 idx 0
Jul 09 10:37:23 [gateway 2] connect_to(256) 16, 127.0.0.1:7005
Jul 09 10:37:23 [gateway 2] do_gateway_write_obj(288) nr_sent 2, err 0
Jul 09 10:37:23 [gateway 2] wait_forward_write(171) 1, revents 1
Jul 09 10:37:23 [gateway 2] sockfd_cache_put(312) 127.0.0.1:7005 idx 0
Jul 09 10:37:23 [gateway 2] write_info_update(112) 2, 1
Jul 09 10:37:23 [gateway 2] wait_forward_write(171) 0, revents 1
Jul 09 10:37:23 [gateway 2] sockfd_cache_put(312) 127.0.0.1:7001 idx 0
Jul 09 10:37:23 [gateway 2] write_info_update(112) 1, 0
Jul 09 10:37:23 [main] cdrv_cpg_deliver(448) 5
Jul 09 10:37:23 [main] sd_notify_handler(851) size: 352, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:23 [main] post_cluster_new_vdi(132) done 0 16594095
Jul 09 10:37:23 [main] client_tx_handler(663) connection from: 13, ::1:45692
Jul 09 10:37:23 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:23 [main] clear_client(703) refcnt:0, fd:13, ::1:45692
Jul 09 10:37:23 [main] destroy_client(672) connection from: ::1:45692
Jul 09 10:37:24 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:24 [main] client_rx_handler(577) connection from: 13, ::1:45695
Jul 09 10:37:24 [main] queue_request(323) 82
Jul 09 10:37:24 [io 3] do_process_work(990) 82, 0 , 1
Jul 09 10:37:24 [main] client_tx_handler(663) connection from: 13, ::1:45695
Jul 09 10:37:24 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:24 [main] clear_client(703) refcnt:0, fd:13, ::1:45695
Jul 09 10:37:24 [main] destroy_client(672) connection from: ::1:45695
Jul 09 10:37:24 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:24 [main] client_rx_handler(577) connection from: 13, ::1:45696
Jul 09 10:37:24 [main] queue_request(323) 11
Jul 09 10:37:24 [main] queue_cluster_request(338) 0x108dbb0 11
Jul 09 10:37:24 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:24 [block] do_process_work(990) 11, 6400000 , 1
Jul 09 10:37:24 [block] do_lookup_vdi(209) looking for test1 (fd32fc)
Jul 09 10:37:24 [block] add_vdi(302) creating new vdi test1: size 104857600, vid fd32fc, base 0, cur 0
Jul 09 10:37:24 [main] queue_request(323) 1
Jul 09 10:37:24 [gateway 4] do_process_work(990) 1, 80fd32fc00000000 , 1
Jul 09 10:37:24 [gateway 4] do_gateway_write_obj(231) 80fd32fc00000000
Jul 09 10:37:24 [gateway 4] sockfd_cache_get(285) 127.0.0.1:7005, idx 0
Jul 09 10:37:24 [gateway 4] sockfd_cache_get(290) create connection 127.0.0.1:7006 idx 0
Jul 09 10:37:24 [gateway 4] connect_to(256) 17, 127.0.0.1:7006
Jul 09 10:37:25 [gateway 4] do_gateway_write_obj(288) nr_sent 2, err 0
Jul 09 10:37:25 [gateway 4] wait_forward_write(171) 1, revents 1
Jul 09 10:37:25 [gateway 4] sockfd_cache_put(312) 127.0.0.1:7006 idx 0
Jul 09 10:37:25 [gateway 4] write_info_update(112) 2, 1
Jul 09 10:37:25 [gateway 4] wait_forward_write(171) 0, revents 1
Jul 09 10:37:25 [gateway 4] sockfd_cache_put(312) 127.0.0.1:7005 idx 0
Jul 09 10:37:25 [gateway 4] write_info_update(112) 1, 0
Jul 09 10:37:25 [main] cdrv_cpg_deliver(448) 5
Jul 09 10:37:25 [main] sd_notify_handler(851) size: 352, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:25 [main] post_cluster_new_vdi(132) done 0 16593660
Jul 09 10:37:25 [main] client_tx_handler(663) connection from: 13, ::1:45696
Jul 09 10:37:25 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:25 [main] clear_client(703) refcnt:0, fd:13, ::1:45696
Jul 09 10:37:25 [main] destroy_client(672) connection from: ::1:45696
Jul 09 10:37:26 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:26 [main] client_rx_handler(577) connection from: 13, ::1:45698
Jul 09 10:37:26 [main] queue_request(323) 82
Jul 09 10:37:26 [io 5] do_process_work(990) 82, 0 , 1
Jul 09 10:37:26 [main] client_tx_handler(663) connection from: 13, ::1:45698
Jul 09 10:37:26 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:26 [main] clear_client(703) refcnt:0, fd:13, ::1:45698
Jul 09 10:37:26 [main] destroy_client(672) connection from: ::1:45698
Jul 09 10:37:26 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:26 [main] client_rx_handler(577) connection from: 13, ::1:45699
Jul 09 10:37:26 [main] queue_request(323) 11
Jul 09 10:37:26 [main] queue_cluster_request(338) 0x108dbb0 11
Jul 09 10:37:26 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:26 [block] do_process_work(990) 11, 6400000 , 1
Jul 09 10:37:26 [block] do_lookup_vdi(209) looking for test2 (fd3815)
Jul 09 10:37:26 [block] add_vdi(302) creating new vdi test2: size 104857600, vid fd3815, base 0, cur 0
Jul 09 10:37:26 [main] queue_request(323) 1
Jul 09 10:37:26 [gateway 6] do_process_work(990) 1, 80fd381500000000 , 1
Jul 09 10:37:26 [gateway 6] do_gateway_write_obj(231) 80fd381500000000
Jul 09 10:37:26 [gateway 6] sockfd_cache_get(285) 127.0.0.1:7005, idx 0
Jul 09 10:37:26 [gateway 6] sockfd_cache_get(285) 127.0.0.1:7001, idx 0
Jul 09 10:37:26 [gateway 6] sockfd_cache_get(290) create connection 127.0.0.1:7002 idx 0
Jul 09 10:37:26 [gateway 6] connect_to(256) 18, 127.0.0.1:7002
Jul 09 10:37:26 [gateway 6] do_gateway_write_obj(288) nr_sent 3, err 0
Jul 09 10:37:27 [gateway 6] wait_forward_write(171) 2, revents 1
Jul 09 10:37:27 [gateway 6] sockfd_cache_put(312) 127.0.0.1:7002 idx 0
Jul 09 10:37:27 [gateway 6] write_info_update(112) 3, 2
Jul 09 10:37:27 [gateway 6] wait_forward_write(171) 1, revents 1
Jul 09 10:37:27 [gateway 6] sockfd_cache_put(312) 127.0.0.1:7001 idx 0
Jul 09 10:37:27 [gateway 6] write_info_update(112) 2, 1
Jul 09 10:37:27 [gateway 6] wait_forward_write(171) 0, revents 1
Jul 09 10:37:27 [gateway 6] sockfd_cache_put(312) 127.0.0.1:7005 idx 0
Jul 09 10:37:27 [gateway 6] write_info_update(112) 1, 0
Jul 09 10:37:27 [main] cdrv_cpg_deliver(448) 5
Jul 09 10:37:27 [main] sd_notify_handler(851) size: 352, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:27 [main] post_cluster_new_vdi(132) done 0 16594965
Jul 09 10:37:27 [main] client_tx_handler(663) connection from: 13, ::1:45699
Jul 09 10:37:27 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:27 [main] clear_client(703) refcnt:0, fd:13, ::1:45699
Jul 09 10:37:27 [main] destroy_client(672) connection from: ::1:45699
Jul 09 10:37:28 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:28 [main] client_rx_handler(577) connection from: 13, ::1:45701
Jul 09 10:37:28 [main] queue_request(323) 82
Jul 09 10:37:28 [io 7] do_process_work(990) 82, 0 , 1
Jul 09 10:37:28 [main] client_tx_handler(663) connection from: 13, ::1:45701
Jul 09 10:37:28 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:28 [main] clear_client(703) refcnt:0, fd:13, ::1:45701
Jul 09 10:37:28 [main] destroy_client(672) connection from: ::1:45701
Jul 09 10:37:28 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:28 [main] client_rx_handler(577) connection from: 13, ::1:45702
Jul 09 10:37:28 [main] queue_request(323) 11
Jul 09 10:37:28 [main] queue_cluster_request(338) 0x108dbb0 11
Jul 09 10:37:28 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:28 [block] do_process_work(990) 11, 6400000 , 1
Jul 09 10:37:28 [block] do_lookup_vdi(209) looking for test3 (fd3662)
Jul 09 10:37:28 [block] add_vdi(302) creating new vdi test3: size 104857600, vid fd3662, base 0, cur 0
Jul 09 10:37:28 [main] queue_request(323) 1
Jul 09 10:37:28 [gateway 8] do_process_work(990) 1, 80fd366200000000 , 1
Jul 09 10:37:28 [gateway 8] do_gateway_write_obj(231) 80fd366200000000
Jul 09 10:37:28 [gateway 8] sockfd_cache_get(290) create connection 127.0.0.1:7003 idx 0
Jul 09 10:37:28 [gateway 8] connect_to(256) 19, 127.0.0.1:7003
Jul 09 10:37:28 [gateway 8] sockfd_cache_get(285) 127.0.0.1:7002, idx 0
Jul 09 10:37:28 [gateway 8] sockfd_cache_get(285) 127.0.0.1:7005, idx 0
Jul 09 10:37:28 [gateway 8] do_gateway_write_obj(288) nr_sent 3, err 0
Jul 09 10:37:29 [gateway 8] wait_forward_write(171) 1, revents 1
Jul 09 10:37:29 [gateway 8] sockfd_cache_put(312) 127.0.0.1:7002 idx 0
Jul 09 10:37:29 [gateway 8] write_info_update(112) 3, 1
Jul 09 10:37:29 [gateway 8] wait_forward_write(171) 0, revents 1
Jul 09 10:37:29 [gateway 8] sockfd_cache_put(312) 127.0.0.1:7003 idx 0
Jul 09 10:37:29 [gateway 8] write_info_update(112) 2, 0
Jul 09 10:37:29 [gateway 8] wait_forward_write(171) 0, revents 1
Jul 09 10:37:29 [gateway 8] sockfd_cache_put(312) 127.0.0.1:7005 idx 0
Jul 09 10:37:29 [gateway 8] write_info_update(112) 1, 0
Jul 09 10:37:29 [main] cdrv_cpg_deliver(448) 5
Jul 09 10:37:29 [main] sd_notify_handler(851) size: 352, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:29 [main] post_cluster_new_vdi(132) done 0 16594530
Jul 09 10:37:29 [main] client_tx_handler(663) connection from: 13, ::1:45702
Jul 09 10:37:29 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:29 [main] clear_client(703) refcnt:0, fd:13, ::1:45702
Jul 09 10:37:29 [main] destroy_client(672) connection from: ::1:45702
Jul 09 10:37:30 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:30 [main] client_rx_handler(577) connection from: 13, ::1:45704
Jul 09 10:37:30 [main] queue_request(323) 82
Jul 09 10:37:30 [io 9] do_process_work(990) 82, 0 , 1
Jul 09 10:37:30 [main] client_tx_handler(663) connection from: 13, ::1:45704
Jul 09 10:37:30 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:30 [main] clear_client(703) refcnt:0, fd:13, ::1:45704
Jul 09 10:37:30 [main] destroy_client(672) connection from: ::1:45704
Jul 09 10:37:30 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:30 [main] client_rx_handler(577) connection from: 13, ::1:45705
Jul 09 10:37:30 [main] queue_request(323) 11
Jul 09 10:37:30 [main] queue_cluster_request(338) 0x108dbb0 11
Jul 09 10:37:30 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:30 [block] do_process_work(990) 11, 6400000 , 1
Jul 09 10:37:30 [block] do_lookup_vdi(209) looking for test4 (fd2de3)
Jul 09 10:37:30 [block] add_vdi(302) creating new vdi test4: size 104857600, vid fd2de3, base 0, cur 0
Jul 09 10:37:30 [main] queue_request(323) 1
Jul 09 10:37:30 [gateway 10] do_process_work(990) 1, 80fd2de300000000 , 1
Jul 09 10:37:30 [gateway 10] do_gateway_write_obj(231) 80fd2de300000000
Jul 09 10:37:30 [gateway 10] sockfd_cache_get(285) 127.0.0.1:7003, idx 0
Jul 09 10:37:30 [gateway 10] sockfd_cache_get(285) 127.0.0.1:7006, idx 0
Jul 09 10:37:30 [gateway 10] sockfd_cache_get(290) create connection 127.0.0.1:7007 idx 0
Jul 09 10:37:30 [gateway 10] connect_to(256) 20, 127.0.0.1:7007
Jul 09 10:37:30 [gateway 10] do_gateway_write_obj(288) nr_sent 3, err 0
Jul 09 10:37:31 [gateway 10] wait_forward_write(171) 2, revents 1
Jul 09 10:37:31 [gateway 10] sockfd_cache_put(312) 127.0.0.1:7007 idx 0
Jul 09 10:37:31 [gateway 10] write_info_update(112) 3, 2
Jul 09 10:37:31 [gateway 10] wait_forward_write(171) 0, revents 1
Jul 09 10:37:31 [gateway 10] sockfd_cache_put(312) 127.0.0.1:7003 idx 0
Jul 09 10:37:31 [gateway 10] write_info_update(112) 2, 0
Jul 09 10:37:31 [gateway 10] wait_forward_write(171) 0, revents 1
Jul 09 10:37:31 [gateway 10] sockfd_cache_put(312) 127.0.0.1:7006 idx 0
Jul 09 10:37:31 [gateway 10] write_info_update(112) 1, 0
Jul 09 10:37:31 [main] cdrv_cpg_deliver(448) 5
Jul 09 10:37:31 [main] sd_notify_handler(851) size: 352, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:31 [main] post_cluster_new_vdi(132) done 0 16592355
Jul 09 10:37:31 [main] client_tx_handler(663) connection from: 13, ::1:45705
Jul 09 10:37:31 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:31 [main] clear_client(703) refcnt:0, fd:13, ::1:45705
Jul 09 10:37:31 [main] destroy_client(672) connection from: ::1:45705
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 13, ::1:45707
Jul 09 10:37:32 [main] queue_request(323) 82
Jul 09 10:37:32 [io 11] do_process_work(990) 82, 0 , 1
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 14
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 13, ::1:45707
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 14, ::1:45708
Jul 09 10:37:32 [main] queue_request(323) 82
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 21
Jul 09 10:37:32 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:32 [main] clear_client(703) refcnt:0, fd:13, ::1:45707
Jul 09 10:37:32 [main] destroy_client(672) connection from: ::1:45707
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 13, ::1:45710
Jul 09 10:37:32 [main] queue_request(323) 12
Jul 09 10:37:32 [main] queue_cluster_request(338) 0x1094010 12
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 21, ::1:45709
Jul 09 10:37:32 [main] queue_request(323) 82
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 22
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 22, ::1:45711
Jul 09 10:37:32 [main] queue_request(323) 82
Jul 09 10:37:32 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:32 [io 12] do_process_work(990) 82, 0 , 1
Jul 09 10:37:32 [block] do_process_work(990) 12, 0 , 1
Jul 09 10:37:32 [block] do_lookup_vdi(209) looking for test2 (fd3815)
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 14, ::1:45708
Jul 09 10:37:32 [main] queue_request(323) 2
Jul 09 10:37:32 [io 13] do_process_work(990) 82, 0 , 1
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 21, ::1:45709
Jul 09 10:37:32 [gateway 14] do_process_work(990) 2, 80fd381500000000 , 1
Jul 09 10:37:32 [gateway 14] sockfd_cache_get(285) 127.0.0.1:7005, idx 0
Jul 09 10:37:32 [gateway 14] sockfd_cache_put(312) 127.0.0.1:7005 idx 0
Jul 09 10:37:32 [io 15] do_process_work(990) 82, 0 , 1
Jul 09 10:37:32 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:32 [main] clear_client(703) refcnt:0, fd:14, ::1:45708
Jul 09 10:37:32 [main] destroy_client(672) connection from: ::1:45708
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 14
Jul 09 10:37:32 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:32 [main] clear_client(703) refcnt:0, fd:21, ::1:45709
Jul 09 10:37:32 [main] destroy_client(672) connection from: ::1:45709
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 21
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 14, ::1:45712
Jul 09 10:37:32 [main] queue_request(323) 12
Jul 09 10:37:32 [main] queue_cluster_request(338) 0x7f32fc000a00 12
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 22, ::1:45711
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 21, ::1:45713
Jul 09 10:37:32 [main] queue_request(323) 12
Jul 09 10:37:32 [main] queue_cluster_request(338) 0x7f32fc000b60 12
Jul 09 10:37:32 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:32 [block] do_process_work(990) 12, 0 , 1
Jul 09 10:37:32 [block] do_lookup_vdi(209) looking for test2 (fd3815)
Jul 09 10:37:32 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:32 [main] clear_client(703) refcnt:0, fd:22, ::1:45711
Jul 09 10:37:32 [main] destroy_client(672) connection from: ::1:45711
Jul 09 10:37:32 [main] queue_request(323) 2
Jul 09 10:37:32 [main] cdrv_cpg_deliver(448) 5
Jul 09 10:37:32 [main] sd_notify_handler(851) size: 96, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 22
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 13, ::1:45710
Jul 09 10:37:32 [gateway 16] do_process_work(990) 2, 80fd381500000000 , 1
Jul 09 10:37:32 [gateway 16] sockfd_cache_get(285) 127.0.0.1:7002, idx 0
Jul 09 10:37:32 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:32 [main] clear_client(703) refcnt:0, fd:13, ::1:45710
Jul 09 10:37:32 [main] destroy_client(672) connection from: ::1:45710
Jul 09 10:37:32 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:32 [gateway 16] sockfd_cache_put(312) 127.0.0.1:7002 idx 0
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 22, ::1:45714
Jul 09 10:37:32 [main] queue_request(323) 12
Jul 09 10:37:32 [main] queue_cluster_request(338) 0x7f32fc000df0 12
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 13
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 13, ::1:45715
Jul 09 10:37:32 [main] queue_request(323) 2
Jul 09 10:37:32 [gateway 17] do_process_work(990) 2, 80fd381500000000 , 1
Jul 09 10:37:32 [gateway 17] sockfd_cache_get(285) 127.0.0.1:7005, idx 0
Jul 09 10:37:32 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:32 [block] do_process_work(990) 12, 0 , 1
Jul 09 10:37:32 [block] do_lookup_vdi(209) looking for test3 (fd3662)
Jul 09 10:37:32 [main] queue_request(323) 2
Jul 09 10:37:32 [gateway 18] do_process_work(990) 2, 80fd366200000000 , 1
Jul 09 10:37:32 [gateway 18] sockfd_cache_get(285) 127.0.0.1:7003, idx 0
Jul 09 10:37:32 [main] cdrv_cpg_deliver(448) 5
Jul 09 10:37:32 [main] sd_notify_handler(851) size: 96, from: IPv4 ip:127.0.0.1 port:7000
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 14, ::1:45712
Jul 09 10:37:32 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:32 [main] clear_client(703) refcnt:0, fd:14, ::1:45712
Jul 09 10:37:32 [main] destroy_client(672) connection from: ::1:45712
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 14
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 14, ::1:45716
Jul 09 10:37:32 [main] queue_request(323) 2
Jul 09 10:37:32 [gateway 18] sockfd_cache_put(312) 127.0.0.1:7003 idx 0
Jul 09 10:37:32 [block] do_process_work(997) failed: 2, 80fd381500000000 , 1, 8
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 23
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 23, ::1:45717
Jul 09 10:37:32 [main] queue_request(323) 82
Jul 09 10:37:32 [gateway 19] do_process_work(990) 2, 80fd381500000000 , 1
Jul 09 10:37:32 [gateway 19] sockfd_cache_get(290) create connection 127.0.0.1:7005 idx 1
Jul 09 10:37:32 [gateway 19] connect_to(256) 24, 127.0.0.1:7005
Jul 09 10:37:32 [io 20] do_process_work(990) 82, 0 , 1
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 23, ::1:45717
Jul 09 10:37:32 [main] client_handler(764) connection seems to be dead
Jul 09 10:37:32 [main] clear_client(703) refcnt:0, fd:23, ::1:45717
Jul 09 10:37:32 [main] destroy_client(672) connection from: ::1:45717
Jul 09 10:37:32 [main] listen_handler(819) accepted a new connection: 23
Jul 09 10:37:32 [main] client_rx_handler(577) connection from: 23, ::1:45719
Jul 09 10:37:32 [main] queue_request(323) 12
Jul 09 10:37:32 [main] queue_cluster_request(338) 0x7f32fc014620 12
Jul 09 10:37:32 [main] cdrv_cpg_deliver(448) 4
Jul 09 10:37:32 [gateway 19] sockfd_cache_put(312) 127.0.0.1:7005 idx 1
Jul 09 10:37:32 [gateway 17] sockfd_cache_put(312) 127.0.0.1:7005 idx 0
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 13, ::1:45715
Jul 09 10:37:32 [main] client_tx_handler(663) connection from: 14, ::1:45716
Jul 09 10:37:32 [main] crash_handler(408) sheep pid 10613 exited unexpectedly.
--
sheepdog mailing list
[email protected]
http://lists.wpkg.org/mailman/listinfo/sheepdog
