> The case in which no more entries are added once a bookie is replaced is a special case
This is the case I referred. Example: Ensemble (A, B, C, D), write quorum size is 3, ack quorum size is 2. 0 ~ N entries are added. But C is slow. so timeout of entry 0 happened. A new bookie E is kicked in to replace C. so the ensemble would be changed 0=>(A, B, E, D). so auto-replication would treated 0 ~ N is underreplicated. Auto-replication would be scheduled for it, although C might eventually already have 0 ~ N entries. > bookies are temporarily slow > In general, ops should remove slow bookies from the system if they are consistently slow. That depends on how fast ops could be notified with such slow bookie. If when ops noticed a slow bookie, did it already make effects to the system especially the metadata store and is the time too late? Also please note, when a slow bookie happened, it might not just cause few ledgers, which might cause those ledgers who are adding entries to that slow bookie. So I think we need Jiannan and Fangmin consider this case, since their case would have lots of ledgers adding entries not a case that few ledgers adding lots of entries. > I can see that there could be some updates we can save in some special cases, but I'm not convinced that it would be huge savings as you're implying. It depends on the number of ledgers writing to a slow bookie at the time interval before it was back to normal or removed. In normal cases, I don't have any concern. But when scale (the number of ledgers) goes up, I would keep the concern. In general, I think avoiding replacing the slow bookie doesn't volatile the contract provided by BookKeeper. On Tue, Dec 11, 2012 at 11:35 PM, Flavio Junqueira <[email protected]>wrote: > When we replace a bookie, it won't do any work for previous entries, but > it will do for future entries. The case in which no more entries are added > once a bookie is replaced is a special case. I'm not sure if this is what > you're referring to. > > In general, ops should remove slow bookies from the system if they are > consistently slow. This quorum mechanism we are talking about is useful > when bookies are temporarily slow or until we have enough time to detect > and remove faulty bookies. > > I can see that there could be some updates we can save in some special > cases, but I'm not convinced that it would be huge savings as you're > implying. I believe we are really talking about exceptional cases here, not > regular cases. > > -Flavio > > On Dec 12, 2012, at 7:48 AM, Sijie Guo wrote: > > >> On the other hand, this is what the application asked us to do, so bk > > should fulfill its commitment. > > > > The key point here is client replaced the slow bookie with a brand new > > bookie but not adding any entry to the new bookie for those ack'ed > entries. > > it already doesn't fulfill the commitment, so why do we need to this > step, > > which would introduce one metadata update. for some systems, it would be > a > > headache if there are lots of ledgers on a slow bookie. > > > > Even worse, for a slow bookie, it might actually store that entry but > just > > respond slowly. so the entry is eventually stored at that bookie > > contributing to the write quorum. But client replaced it with a brand new > > bookie without adding any entries, which reduce the number of replicas of > > that entry and increment the work of auto-replication, doesn't it? > > > > from my perspective, if we could reduce an unnecessary metadata access > for > > a ledger, we could save a lot of traffic to metadata storage for those > > applications using huge number of ledgers. Especially it would be worse > if > > the metadata store is not optimized for writes. > > > > -Sijie > > > > > > On Tue, Dec 11, 2012 at 10:19 PM, Flavio Junqueira <[email protected] > >wrote: > > > >> In my understanding, the semantics we provide is closer to your option > b), > >> but not quite the way you put it. The contract is that upon an > >> acknowledgement to a request to add an entry, we guarantee that we have > at > >> least ackQuorumSize copies stored in bookies. We send copies to more > >> bookies (writeQuorumSize) to avoid slow bookies as you say. > >> > >> It is correct that if there aren't enough bookies we will throw an > >> exception, and you imply that it is perhaps unnecessary, since the > contract > >> is not to store on writeQuorumSize bookies anyway. On the other hand, > this > >> is what the application asked us to do, so bk should fulfill its > >> commitment. In the case there aren't enough bookies and the application > >> gets an exception, it closes the ledger and create a new one with fewer > >> bookies. > >> > >> -Flavio > >> > >> On Dec 12, 2012, at 6:47 AM, Sijie Guo wrote: > >> > >>> Hello guys, > >>> > >>> AckSet is introduced recently to resolve slow bookie issue. But the > >> semantic of 'writeQuorumSize' is not so clearly. What kind of goal we > >> should achieve for 'writeQuorumSize': > >>> > >>> a) an entry is written to all 'writeQuorumSize' bookies eventually > >>> b) just guarantee issuing 'writeQuorumSize' bookies and at least > >> 'ackQuorumSize' bookies acked. other bookies we don't care about it. > >>> > >>> semantic a): > >>> > >>> if we want a), currently it doesn't work. for example, writing ensemble > >> (A, B, C, D), writeQuorumSize is 3, ackQuorumSize is 2 and C is a slow > >> bookie. > >>> > >>> 1. client adding entries 0. and entry 0 is acked. > >>> 2. at time T, C is timeout (due to it is slow of failed). so the adding > >> entry 0 at C would failed with BookieHandleNotAvailable. It would try > to > >> pickup a new bookie and unsetSuccessAndSentWriteRequest to new bookie. > But > >> nothing would be executed since pendingAddOps queue is empty. so nothing > >> would be added in the new bookie. > >>> > >>> private void unsetSuccessAndSendWriteRequest(final int bookieIndex) > { > >>> for (PendingAddOp pendingAddOp : pendingAddOps) { > >>> pendingAddOp.unsetSuccessAndSendWriteRequest(bookieIndex); > >>> } > >>> } > >>> > >>> If we want to achieve semantic a), it would be difficult to handle > >> client failure case. > >>> > >>> semantic b): > >>> > >>> if we just want achieve b), so do we need to pickup a new bookie > replace > >> the slow bookie? Also even a new bookie is replaced, no entry would be > >> added again to it, since pendingAddOps is empty. > >>> > >>> I raised this question, is because if we pickup a new bookie replace > the > >> slow bookie, it might cause closing ledger due to > >> NotEnoughBookiesException. It is easy to produce the case using existing > >> test case. Please see the attached file. > >>> > >>> if we allow b), so maybe we don't need handleBookieFailure when a late > >> response arrived at the client after its entry has been acked. > >>> > >>> -Sijie > >> > >> > >
