Hi,
I figured that the reason for this issue is that:
1.Precondition:There are only gateway nodes left in the cluster. So the gateway
nodes kept the latest epoch. But as gateway node doesn't count in nr_zones,
nr_zones equal to 0.
2.The nodes in cluster was restarted, and start to recover from the latest
epoch, in which the nr_zones is 0. In function recover_object_from replica, it
will skip all the recover process(as nr_copies equals to 0), but return
success(as ret was initialled as success).
static int recover_object_from_replica()
{
int nr_copies, ret = SD_RES_SUCCESS, start = 0;
...
nr_copies = get_obj_copy_number(oid, old->nr_zones);
for (int i = 0; i < nr_copies; i++) {
...
}
for (int i = 0; i < nr_copies; i++) {
...
}
if (fully_replicated && ret != SD_RES_SUCCESS)
ret = SD_RES_STALE_OBJ;
return ret;
}
3.The node thought it got the latest obj successfully and delete the old
version in .stale. but actually there was no obj recovered.
In this case the latest epoch should be the latest one with at least one node
who is not in gateway mode.
Is that possible to let the gateway mode node run without tracking epoch info?
In this way the latest epoch will always includes an non-gateway node.
Thanks,
Yang
-----原始邮件-----
发件人: "张扬" <[email protected]>
发送时间: 2014-11-29 17:44:31 (星期六)
收件人: [email protected]
抄送:
主题: Fixing method for data wipe bug in the recovery
Hi,
We recently encounter the bug which was reported here:
https://bugs.launchpad.net/sheepdog-project/+bug/1327037
I'm not sure if this is the root cause, but by changing the initial value of
ret in function recover_object_from_replica(in recovery.c)
ret = SD_RES_SUCCESS
to
ret = SD_RES_NO_OBJ (actually anything but SD_RES_SUCCESS), i'm able to avoid
the lose of data.
The reason is that, when the node try to recover it self to match the newest
epoch, the nr_copies in function recover_object_from_replica is zero, so it
return ret == SD_RES_SUCCESS while it actually skipped all the recover function
in the loop. I don't know why nr_copies is zero though.
Thanks,
Yang--
sheepdog mailing list
[email protected]
http://lists.wpkg.org/mailman/listinfo/sheepdog
