The OSD is supposed to stay down if any of the networks are missing. Ceph is a CP system in CAP parlance; there's no such thing as a CA system. ;)
What version of Ceph are you testing right now? -Greg On Mon, Sep 14, 2015 at 1:02 AM, [email protected] <[email protected]> wrote: > Thanks for your explaination~ > you mean the origin design leads to this result?In that network > situation,flapping must appear and can not terminate except that someone(such > as manager) force marking osd out,isn't it? > I also want to ask about whether ceph community has a plan to solve this > problem or not? > > Thanks~ > > -----邮件原件----- > 发件人: Chen, Xiaoxi [mailto:[email protected]] > 发送时间: 2015年9月14日 15:21 > 收件人: huang jun; zhaomingyue 09440 (RD) > 抄送: [email protected] > 主题: RE: 答复: 2 replications,flapping can not stop for a very long time > > This is kind of unsolvable problem, in CAP , we choose Consistency and > Availability, thus we had to lose Partition tolerance. > > There are three networks here , mon<-> osd, osd<-public->osd, osd<- cluster-> > osd. If some of the networks are reachable but some are not, likely the > flipping will happen. > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of huang jun > Sent: Sunday, September 13, 2015 5:46 PM > To: [email protected] > Cc: [email protected] > Subject: Re: 答复: 2 replications,flapping can not stop for a very long time > > 2015-09-13 14:07 GMT+08:00 [email protected] <[email protected]>: >> hi, do you set both public_network and cluster_network, but just cut off the >> cluster_network? >> And do you have not only one osd on the same host? >> =============================yes,public network+cluster network,and I >> cut off the cluster network; 2 node ,each node has serveral osds; >> >> If so, maybe you can not get stable, now the osd have peers in the prev and >> next osd id, they can exchange ping message. >> you cut off the cluster_network, the outbox peer osds can not detect the >> ping, they reports the osd failure to MON, and MON gather enough reporters >> and reports, then the osd will be marked down. >> =============================when osd recv a new map and it is marked >> down,it think MON wrongly mark me down,what will it do,join the cluster >> again or other actions?can you give me some more detailed explanation? > > It will send a boot message to MON, and will be marked UP by MON. >> >> But the osd can reports to MON bc the public_network is ok, MON thinks the >> osd wronly marked down, mark it to UP. >> =============================you mean that MON recv message ONE TIME from >> this osd then it will mark this osd up? >> > >> So flapping happens again and again. >> ============================= I tried 3 replications,(public network + >> cluster network,3 node,each node has serveral osds),although it will >> occur flapping,but after serveral minutes it will be stable, compared with 2 >> replications situation, I wait for the same intervals,the cluster can not be >> stable; so I'm confused about the machnism that how monitor can decide which >> osd is actually down? >> > It's weird, if you cut off the cluster_network, the osds in other node can > not get the ping messages, and naturally think the osd is failed. > >> thanks >> >> -----邮件原件----- >> 发件人: huang jun [mailto:[email protected]] >> 发送时间: 2015年9月13日 10:39 >> 收件人: zhaomingyue 09440 (RD) >> 抄送: [email protected] >> 主题: Re: 2 replications,flapping can not stop for a very long time >> >> hi, do you set both public_network and cluster_network, but just cut off the >> cluster_network? >> And do you have not only one osd on the same host? >> If so, maybe you can not get stable, now the osd have peers in the prev and >> next osd id, they can exchange ping message. >> you cut off the cluster_network, the outbox peer osds can not detect the >> ping, they reports the osd failure to MON, and MON gather enough reporters >> and reports, then the osd will be marked down. >> But the osd can reports to MON bc the public_network is ok, MON thinks the >> osd wronly marked down, mark it to UP. >> So flapping happens again and again. >> >> 2015-09-12 20:26 GMT+08:00 [email protected] <[email protected]>: >>> >>> Hi, >>> I'm testing reliability of ceph recently, and I have met the flapping >>> problem. >>> I have 2 replications, and cut off the cluster network ,now flapping can >>> not stop,I have wait more than 30min, but status of osds are still not >>> stable; >>> I want to know about when monitor recv reports from osds ,how it can >>> mark one osd down? >>> (reports && reporter && grace) need to satisfied some conditions, how >>> to calculate the grace? >>> and how long will the flapping stop?Does the flapping must be stopped by >>> configure,such as configure an osd lost? >>> Can someone help me ? >>> Thanks~ >>> --------------------------------------------------------------------- >>> - >>> --------------------------------------------------------------- >>> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出 >>> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 >>> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 >>> 邮件！ >>> This e-mail and its attachments contain confidential information from >>> H3C, which is intended only for the person or entity whose address is >>> listed above. Any use of the information contained herein in any way >>> (including, but not limited to, total or partial disclosure, >>> reproduction, or dissemination) by persons other than the intended >>> recipient(s) is prohibited. If you receive this e-mail in error, >>> please notify the sender by phone or email immediately and delete it! >> >> >> >> -- >> thanks >> huangjun > > > > -- > thanks > huangjun > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the > body of a message to [email protected] More majordomo info at > http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
