Hi, Please take a look on the new PR for ZK-2959: https://github.com/apache/zookeeper/pull/500 If there are no further comments, I can commit it.
Thanks, Alex On Fri, Apr 6, 2018 at 11:33 AM, Alexander Shraer <shra...@gmail.com> wrote: > Hi, > > The bug described in ZOOKEEPER-2959 > <https://issues.apache.org/jira/browse/ZOOKEEPER-2959> is that > getEpochToPropose an waitForEpochAck do not distinguish between followers > and observers. > This can cause a candidate leader's acceptedEpoch to be updated with only > support from observers. Same for waitForEpochAck - passing this method > allows the candidate leader to update the currentEpoch. The latter helps > this server to win FLE elections continuously, and the former > (acceptedEpoch) > causes anyone trying to connect to the server to think that it has more > up-to-date data and trucate their logs to match. > > > Alex > > On Fri, Apr 6, 2018 at 10:04 AM, Fangmin Lv <lvfang...@gmail.com> wrote: > >> Hi Alex, >> >> Can you give more details about the data loss scenario in Jira >> ZOOKEEPER-2959 <https://issues.apache.org/jira/browse/ZOOKEEPER-2959>? >> As far as I know, the leader will ignore the observers' ACK in >> waitForNewLeaderAck, so it will not start serve traffic until it received >> the actual quorum ACK, if it doesn't have enough followers support before >> timeout, it will quit leading and it's learners will re-sync with new >> leader. >> >> Thanks, >> Fangmin >> >> On Thu, Apr 5, 2018 at 12:57 PM, Alexander Shraer <shra...@gmail.com> >> wrote: >> >>> Btw we actually observed the described issue (data loss), thankfully in a >>> test environment. So I thought this is important to share with the >>> community. >>> >>> Unfortunately I don’t have time to run a new ZK release for this, so I’m >>> not going to -1 your candidate, but we are actively working on a fix (ie >>> a >>> test at this point) and I can commit that as soon as we have that. >>> >>> It may be worth while to delay the release by a few more days, but it’s >>> totally up to you since you’re running it. >>> >>> Cheers >>> Alex >>> On Thu, Apr 5, 2018 at 12:47 PM Andor Molnar <an...@cloudera.com> wrote: >>> >>> > Got that. I still believe it's a completely valid issue which has to be >>> > addressed, but it's not a showstopper. I'm afraid we're not going to >>> > convince each other, so it's probably Abe's call if he want to create >>> > another release candidate for the fix. >>> > >>> > I reviewed the code on github and I think it just needs to be covered >>> with >>> > a unit test to be complete. >>> > >>> > Regards, >>> > Andor >>> > >>> > >>> > >>> > On Thu, Apr 5, 2018 at 9:05 PM, Alexander Shraer <shra...@gmail.com> >>> > wrote: >>> > >>> > > Yes sort of, FLE is finished, then enough observer's messages reach >>> the >>> > > leader before participant's messages do. >>> > > Whether its rare depends on the number of observers and >>> participants. For >>> > > example with very few participants and many observers >>> > > your chance of hitting this are quite high. >>> > > >>> > > Alex >>> > > >>> > > On Thu, Apr 5, 2018 at 11:44 AM, Andor Molnar <an...@cloudera.com> >>> > wrote: >>> > > >>> > > > Maybe I'm missing something here, but this looks like a rare edge >>> case >>> > to >>> > > > me. Participants must finish the leader election successfully and >>> right >>> > > > after enough followers should fail to send epoch to the leader, so >>> > > > observers can take it over. >>> > > > >>> > > > Is that description accurate? >>> > > > >>> > > > Andor >>> > > > >>> > > > >>> > > > On Thu, Apr 5, 2018 at 7:35 PM, Alexander Shraer < >>> shra...@gmail.com> >>> > > > wrote: >>> > > > >>> > > > > To clarify - in a deployment with observers this bug can >>> potentially >>> > > > cause >>> > > > > data loss. A server could be elected leader based just on the >>> support >>> > > of >>> > > > > observers, even if this servers data is stale wrt other >>> followers. >>> > > > > >>> > > > > It is certainly a blocker, just not sure if for 3.4.11 or 3.4.12. >>> > > > > >>> > > > > >>> > > > > Alex >>> > > > > On Thu, Apr 5, 2018 at 10:29 AM Andor Molnar <an...@cloudera.com >>> > >>> > > wrote: >>> > > > > >>> > > > > > I don't think it's a blocker. >>> > > > > > The jira and PR has been open since last December and 3.4.11 >>> has >>> > > > released >>> > > > > > without it. >>> > > > > > >>> > > > > > Although this bug is also important to fix, I believe it's more >>> > > > important >>> > > > > > to release a fix for the regression we've found in 3.4.11 asap. >>> > > > > > >>> > > > > > Abe, any thoughts? >>> > > > > > >>> > > > > > Regards, >>> > > > > > Andor >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > On Thu, Apr 5, 2018 at 7:00 PM, Alexander Shraer < >>> > shra...@gmail.com> >>> > > > > > wrote: >>> > > > > > >>> > > > > > > Sorry for coming in at the last moment. I'm not sure when the >>> > next >>> > > > 3.4 >>> > > > > > > release is scheduled, so just wanted to mention this bug, >>> > > > > > > which I believe is a blocker for either this or next release: >>> > > > > > > https://issues.apache.org/jira/browse/ZOOKEEPER-2959 >>> > > > > > > >>> > > > > > > Best, >>> > > > > > > Alex >>> > > > > > > >>> > > > > > > On Thu, Apr 5, 2018 at 9:09 AM, Ted Yu <yuzhih...@gmail.com> >>> > > wrote: >>> > > > > > > >>> > > > > > > > Can the vote be closed ? >>> > > > > > > > >>> > > > > > > > It seems we have enough +1's >>> > > > > > > > >>> > > > > > > > Thanks >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >> >> >
