Here you go https://gist.github.com/vinothchandar/18feedfa84650e3efdc0
On Fri, Mar 25, 2016 at 8:32 AM, kishore g <[email protected]> wrote: > Can you point me to your code. fork/patch? > > On Fri, Mar 25, 2016 at 5:26 AM, Vinoth Chandar <[email protected]> wrote: > >> Hi Kishore, >> >> Printed out more information and trimmed the test down to 1 resource with >> 2 partitions, and I bring up 8 servers in parallel. >> >> Below is the paste of my logging output + annotations. >> >> >>> Computing partition assignment >> >>>> NodeShift for countLog-2a 0 is 5, index 5 >> >>>> NodeShift for countLog-2a 1 is 5, index 6 >> >> VC: So this part seems fine. We pick nodes at index 5 & 6 instead of 0, 1 >> >> >>>> Preferred Assignment: {countLog-2a_0|0=########## >> name=localhost-server-6 >> preferred:0 >> nonpreferred:0, countLog-2a_1|0=########## >> name=localhost-server-7 >> preferred:0 >> nonpreferred:0} >> >> VC: This translates to server-6/server-7 (since I named them starting 1) >> >> >>>> Existing Preferred Assignment: {} >> >>>> Existing Non Preferred Assignment: {} >> >>>> Orphaned: [countLog-2a_0|0, countLog-2a_1|0] >> >>> Final State Map :{0=ONLINE} >> >>>> Final ZK record : countLog-2a, >> {}{countLog-2a_0={localhost-server-1=ONLINE}, >> countLog-2a_1={localhost-server-1=ONLINE}}{countLog-2a_0=[localhost-server-1], >> countLog-2a_1=[localhost-server-1]} >> >> VC: But the final effect still seems to be assigning the partitions to >> servers 1 & 2 (first two). >> >> Any ideas on where to start poking? >> >> >> Thanks >> Vinoth >> >> On Tue, Mar 15, 2016 at 5:52 PM, Vinoth Chandar <[email protected]> wrote: >> >>> Hi Kishore, >>> >>> I think the changes I made are exercised when computing the preferred >>> assignment, later when the reconciliation happens with existing >>> assignment/orphaned partitions etc, I think it does not take effect. >>> >>> The effective assignment I saw was all partitions (2 per resource) were >>> assigned to first 2 servers. I started to dig into the above mentioned >>> parts of the code, will report back tmrw when I pick this back up. >>> >>> Thanks, >>> Vinoth >>> >>> _____________________________ >>> From: kishore g <[email protected]> >>> Sent: Tuesday, March 15, 2016 2:01 PM >>> Subject: Re: Balancing out skews in FULL_AUTO mode with built-in >>> rebalancer >>> To: <[email protected]> >>> >>> >>> >>> 1) I am guessing it gets overriden by other logic in >>> computePartitionAssignment(..), the end assignment is still skewed. >>> >>> What is the logic you are referring to? >>> >>> Can you print the assignment count for your use case? >>> >>> >>> thanks, >>> Kishore G >>> >>> On Tue, Mar 15, 2016 at 1:45 PM, Vinoth Chandar <[email protected]> >>> wrote: >>> >>>> Hi guys, >>>> >>>> We are hitting a fairly known issue where we have 100s of resource with >>>> < 8 resources spreading across 10 servers and the built-in assignment >>>> always assigns partitions from first to last, resulting in heavy skew for a >>>> few nodes. >>>> >>>> Chatted with Kishore offline and made a patch as here >>>> <https://gist.github.com/vinothchandar/e8837df301501f85e257>.Tested >>>> with 5 resources with 2 partitions each across 8 servers, logging out the >>>> nodeShift & ultimate index picked does indicate that we choose servers >>>> other than the first two, which is good >>>> >>>> But >>>> 1) I am guessing it gets overriden by other logic in >>>> computePartitionAssignment(..), the end assignment is still skewed. >>>> 2) Even with murmur hash, there is some skew on the nodeshift, which >>>> needs to ironed out. >>>> >>>> I will keep chipping at this.. Any feedback appreciated >>>> >>>> Thanks >>>> Vinoth >>>> >>> >>> >>> >>> >> >
