6 nodes sheepdog cluster (v0.9.1) When I kill one node, I get the following error message. When I restart the close node, I get the same error message.
Jan 20 11:17:33 INFO [main] recover_object_main(905) object recovery progress 97% Jan 20 11:17:33 ALERT [rw 22948] get_vdi_copy_number(110) copy number for 3f4db1 not found, set 6 Jan 20 11:17:33 ALERT [rw 22948] get_vdi_copy_number(110) copy number for 3f4db1 not found, set 6 Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.172:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22948] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.174:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.165:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.173:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] recover_replication_object(411) can not recover oid 803f4db000000000 Jan 20 11:17:33 ERROR [rw 22947] recover_object_work(575) failed to recover object 803f4db000000000 Jan 20 11:17:33 ALERT [rw 22926] get_vdi_copy_number(110) copy number for 4167a5 not found, set 6 Jan 20 11:17:33 ALERT [rw 22926] get_vdi_copy_number(110) copy number for 4167a5 not found, set 6 Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.174:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.172:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22948] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.172:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22948] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.176:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.165:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.173:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22948] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.173:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.176:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] recover_replication_object(411) can not recover oid 804167a500000000 Jan 20 11:17:33 ERROR [rw 22926] recover_object_work(575) failed to recover object 804167a500000000 Jan 20 11:17:33 INFO [main] recover_object_main(905) object recovery progress 98% Jan 20 11:17:33 ALERT [rw 22947] get_vdi_copy_number(110) copy number for b65ff6 not found, set 6 Jan 20 11:17:33 ALERT [rw 22947] get_vdi_copy_number(110) copy number for b65ff6 not found, set 6 Jan 20 11:17:33 ERROR [rw 22948] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.165:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22948] recover_replication_object(411) can not recover oid 803f4db100000000 Jan 20 11:17:33 ERROR [rw 22948] recover_object_work(575) failed to recover object 803f4db100000000 Jan 20 11:17:33 ALERT [rw 22926] get_vdi_copy_number(110) copy number for de4900 not found, set 6 Jan 20 11:17:33 ALERT [rw 22926] get_vdi_copy_number(110) copy number for de4900 not found, set 6 Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.165:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.172:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.174:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.172:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.173:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.176:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.176:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.173:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.165:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22947] recover_replication_object(411) can not recover oid 80b65ff600000000 Jan 20 11:17:33 ERROR [rw 22947] recover_object_work(575) failed to recover object 80b65ff600000000 Jan 20 11:17:33 INFO [main] recover_object_main(905) object recovery progress 99% Jan 20 11:17:33 ERROR [rw 22926] sheep_exec_req(1170) failed Network error between sheep, remote address: 103.24.1.174:7000, op name: READ_PEER Jan 20 11:17:33 ERROR [rw 22926] recover_replication_object(411) can not recover oid 80de490000000000 Jan 20 11:17:33 ERROR [rw 22926] recover_object_work(575) failed to recover object 80de490000000000 Jan 20 11:17:38 NOTICE [main] cluster_recovery_completion(714) all nodes are recovered, epoch 20 -- sheepdog mailing list [email protected] https://lists.wpkg.org/mailman/listinfo/sheepdog -- sheepdog mailing list [email protected] https://lists.wpkg.org/mailman/listinfo/sheepdog
