capistrant commented on PR #16335: URL: https://github.com/apache/druid/pull/16335#issuecomment-2079691221
> Thanks for adding the new balancer strategy, @capistrant ! > > > The overarching goal of the balancer is to drive a cluster's storage distribution towards uniform disk utilization % across historical servers. > > Do you think the current strategies fail to achieve this in certain cases? It would be nice to know some real world scenarios where the new strategy does a better job of balancing servers as compared to the current strategies. From my experience running a large high segment count (over 6 million), high datasource count (over 12k) cluster with great variance in segment size (kb to gb) - the cost balancer creates a highly skewed environment when it comes to disk allocation. Maybe if cpu constraints were not a problem and we could balance many segments in short order - the algorithm would work its way towards having less storage skew. But in our environment even with a 90 cores on coordinator dedicated to balancing, we can not balance meaningful number of segments while also keeping coordination within reason. We had servers with nearly 90% utilization and others with under 50% with little to no progress towards balance over months and months of uptime. Perhaps, an alternative for us could have been cachingCost, if that meaningfully drops cpu. But we had experienced odd behavior that we couldn't pinpoint in the past when testing it in lower envs, and thus abandoned the idea of using it in prod until exhaustive testing could be done. That left us with the random balancer, which by nature leads to skew in an environment like ours where segment sizes vary so much. We did really like the performance in terms of how many segments we could balance though. And this was the trigger for us to write this balancer that uses simple weighting to avoid the pitfalls of random for us, while still allowing us to move many segments in a short amount of time. > It would also be nice to have some tests based on the `CoordinatorSimulationBaseTest` that demonstrates how this strategy performs in a typical setup. I will look into this shortly, thanks for bringing it to my attention! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
