Let me try to bump up the prio on getting this out soon.
Blockers were:1) waiting on getting this proved out in production - no longer an issue; we've been running this for weeks now 2) getting approval from management to release the code - also no longer an issue; the approved this a while ago 3) cleaning up the code to remove dependencies on our other code - this still needs to be done, but won't take long 4) getting this into shape for inclusion in the apache source tree (see: http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201008.mbox/%[email protected]%3e) - this will actually take a while
If I tackle #3, I can at least put out an "unofficial version" for people to play with.
Tackling #4 will take longer though. (Perhaps if someone wants to volunteer to help, this could get sped up.)
Will email back soon with a link to grab the unofficial RC. DR On 12/02/2010 12:54 PM, Claudio Martella wrote:
Mhm, that's smart, i like it. looking forward to see your published code. On 12/2/10 6:51 PM, David Rosenstrauch wrote:Yes. E.g., current state is: 2122691627-2122691654,2122691754-2122691786,2122719881-2122719915,2122720271-2122720368,2122720488-2122720510,2122721282-2122721368,2122721369-2122721391,2122733020-2122733095,2122733212-2122733330,2122733489-2122733546,2122733932-2122734000,2122734001-2122734013,2122734256-2122734300,2122738380-2122738419,2122738420-2122738516,2122738949-2122739091,2122739207-2122739212,2122739298-2122739324,2122739960-2122740122,2122740641-2122740651,2122740769-2122740789,2122744412-2122744449,2122745502-2122745554,2122745668-2122745734,2122745735-2122745801,2122745939-2122745959,2122746069-2122746105,2122746811-2122746845,2122746965-2122747044,2122747452-2122747477,2122747580-2122747602,2122747603-2122747656,2122747774-2122747844,2122747963-2122748009,2122748864-2122748935,2122749312-2122749331,2122749545-2122749577,2122749956-2122750030,2122750281-2122750592,2122750593-2122750593,2122753024-2122753079,2122753080-2122753084,2122753211-2122753214,2122753343-2122753455,2122753782-21227538
11,
2122754275-2122754352,2122755379-2122755527,2122755528-2122755611,2122755801-2122755849,2122755850-2122755889,2122756248-2122756275,2122756493-2122756586,2122756685-2122756737,2122756738-2122756861,2122756862-2122756887,2122757251-2122757360,2122757593-2122757715,2122757716-2122757736,2122757867-2122757941,2122758063-2122758079,2122758347-2122758359,2122758576-2122758651,2122758914-2122759041,2122759175-2122759191,2122759292-2122759319,2122759320-2122759334,2122759467-2122759524,2122759747-2122759765,2122759871-2122759952,2122759953-2122759991,2122760096-2122760191,2122760192-2122760199,2122760200-2122760227,2122760547-2122760601,2122760700-2122760701,2122760702-2122760713,2122760714-2122760830,2122761222-2122761316,2122761416-2122761439,2122762805-2122762987,2122763112-2122763148,2122763269-2122763336,2122764382-2122764445,2122764446-2122764463,2122764587-2122764613,2122764846-2122765138,2122765511-2122765511,2122765738-2122765744,2122766347-2122766398,2122766520-21227665
44,
2122766653-2122766756,2122766757-2122766820,2122767104-2122767148,2122767149-2122767185,2122767311-2122767359,2122767360-2122767398,2122767533-2122767596,2122767597-2122767680,2122767924-2122767939,2122768199-2122768340,2122768849-2122768927,2122769064-2122769314,2122769545-2122769587,2122770173-2122770362,2122770497-2122770512,2122770621-2122770664,2122770972-2122771104,2122771213-2122771311,2122771312-2122771531,2122771655-2122771673,2122771785-2122771806,2122771807-2122771853,2122771854-2122771889,2122771890-2122771917,2122777516-2122777676,2122777778-2122777789,2122779043-2122779098,2122779206-2122779405,2122779670-2122779698,2122779808-2122779872,2122779873-2122779875,2122780679-2122780755,2122780869-2122780957,2122780958-2122781066,2122781185-2122781339,2122781463-2122781484,2122781485-2122781504,2122781739-2122781740,2122781863-2122781878,2122781879-2122782019,2122782121-2122782244,2122783688-2122783812,2122783916-2122783936,2122784051-2122784122,2122784260-21227842
95,
2122784296-2122784530,2122784678-2122784812,2122784911-2122784913,2122785039-2122785220,2122785324-2122785412,2122785525-2122785662,2122786051-2122786341,2122786457-2122786503,2122786504-2122786787,2122787039-2122787093,2122787237-2122787431,2122787669-2122787812,2122787925-2122788059,2122788176-2122788295,2122788516-2122788579,2122788681-2122788812,2122788813-2122788914,2122789037-2122789229,2122789230-2122789459,2122789460-2122789502,2122791353-2122791552,2122791553-2122791614,2122791725-2122791824,2122791825-2122791949,2122792069-2122792213,2122792330-2122792341,2122792342-2122792383,2122792507-2122792851,2122793085-2122793089,2122793090-2122793224,2122793225-2122793347,2122793460-2122793516,2122793517-2122793643,2122793745-2122793782,2122793783-2122793949,2122793950-2122794070,2122794071-2122794085,2122794086-2122794245,2122794571-2122794662,2122794910-2122795296,2122795432-2122795701,2122795825-2122795891,2122796002-2122796074,2122796180-2122796269,2122796500-21227969
28,
2122797169-2122797296,2122797297-2122797544,2122797545-2122797548,2122797549-2122797626,2122799193-2122799247,2122799248-2122799406,2122799529-2122799658,2122799659-2122799703,2122799822-2122800053,2122800179-2122800309,2122800310-2122800427,2122800678-2122800703,2122800859-2122801458,2122801586-2122801679,2122801680-2122801703,2122801844-2122801857,2122801858-2122801916,2122802042-2122802098,2122802218-2122802703,2122802800-2122802938,2122803202-2122803333,2122803470-2122803703,2122803704-2122804012,2122804116-2122804148,2122804149-2122804316,2122804317-2122804436,2122804545-2122804561,2122805924-2122806012,2122806135-2122806667,2122806786-2122807012,2122807137-2122807159,2122807280-2122807791,2122807897-2122808012,2122808138-2122808389,2122809758-2122810047,2122810161-2122810240,2122810359-2122810880,2122811003-2122811165,2122811166-2122811240,2122811355-2122811381,2122811507-2122812119,2122813558-2122814389,2122814514-2122814568,2122815795-2122816389,2122816525-21228168
22,
2122819142-2122819389,2122819390-2122819484,2122819485-2122819820,2122819821-2122819937,2122820042-2122820134,2122821337-2122821937,2122821938-2122822223,2122823559-2122824464,2122824465-4000000000 (~5K characters) DR On 12/02/2010 12:40 PM, Claudio Martella wrote:so you keep a list of unused ID ranges in the ZK node's data? On 12/2/10 6:21 PM, David Rosenstrauch wrote:I don't, for several reasons: a) We request ID's frequently enough that old ones do eventually get used up. b) Even with a largely fragmented ID space, the largest we've seen the bytes contents of that ZK node get up to is 5-6KBytes. So no real worries about either storage space or network I/O when reading/writing to/from the ZK node. DR On 12/02/2010 10:58 AM, Claudio Martella wrote:I like Ted's idea too. David, how do you handle re-compaction of your fragmented ID space? On 12/2/10 4:55 PM, David Rosenstrauch wrote:On 12/02/2010 10:47 AM, Ted Dunning wrote:I would recommend that you increment the counter by 100 or 1000 and then increment a local counter over the implied range. This will drive the amortized ZK overhead down to tens of microseconds which should be good for almost any application. Your final ids will still be almost entirely contiguous. You could implement a fancier counter in ZK that remembers returned chunks for re-use to get perfect contiguity if you really wanted that.This is what our library does. You request chunks of, say, 1000 ID's, and then push back any remaining unused ID's in the chunk you took. DR
