This may not be an issue with the master as we moved to immutable metadata,
but older releases have an issue with not decrementing blockAddCompletions.
Sijie/Ivan/Sam can you validate this?
blockAddCompletions gets incremented while initiating the ensemble change
in handleBookieFailure(). Based on how we handle the bookie failure this
should get decremented. Until this comes back to zero there wont be any
addEntryCompletions are sent to the application.
I found two areas where this counter may not get decremented. (Look for
"BUG" in the code snippet below)
1. 1.
2.
3. updateMetadataIfPossible -> This should decrement blockAddCompletions
if returning TRUE
4. {
5. if (metadata.isNewerThan(newMeta)) {
6. rereadMetadata(this); ==> This is a loop back to reattempt which
comes back to here.
7. return true;
8. }
9. if (metadata.isConflictWith(newMeta)) {
10. return false;
11. }
12. // update znode version
13. metadata.setVersion(newMeta.getVersion());
14. // merge ensemble infos from new meta except last ensemble
15. // since they might be modified by recovery tool.
16. metadata.mergeEnsembles(newMeta.getEnsembles());
17. writeLedgerConfig(new ChangeEnsembleCb(ensembleInfo,
curBlockAddCompletions,
18. ensembleChangeIdx));
19. return true;
20. ====> BUG: We are NOT decrementing blockAddCompletions
21. }
22.
23. 2.
24. resolveConflict()
25. {
26. int diff = newMeta.getEnsembles().size() - metadata.getEnsembles().
size();
27. if (0 != diff) {
28. if (-1 == diff) {
29. // Case 1: metadata is changed by other ones (e.g. Recovery)
30. return updateMetadataIfPossible(newMeta);
31. }
32. return false;
33. }
34. if (!areFailedBookiesReplaced(newMeta, ensembleInfo)) {
35. if (areFailedBookiesReplaced(metadata, ensembleInfo)) {
36. return updateMetadataIfPossible(newMeta);
37. }
38. else??
39. ===>BUG: There is no else case here. If there can be an else case
here, we are returning TRUE but not decrementing blockAddCompletions.
40. } else {
41. int newBlockAddCompletions = blockAddCompletions.decrementAndGet();
42. unsetSuccessAndSendWriteRequest(ensembleInfo.replacedBookies);
43. }
44. return true;
45. }
@Sam Just
<https://gus.lightning.force.com/lightning/r/005B0000000SS00IAG/view> let
us discuss this tomorrow.
--
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi
