On Wed, Jan 27, 2010 at 12:02 PM, Ryan King <r...@twitter.com> wrote: > On Wed, Jan 27, 2010 at 11:49 AM, Jonathan Ellis <jbel...@gmail.com> wrote: >> On Wed, Jan 27, 2010 at 1:48 PM, Stu Hood <stu.h...@rackspace.com> wrote: >>>> The HH code currently tries to send the hints to nodes other than the >>>> natural endpoints. If small-scale performance is a problem, we could >>>> make the natural endpoints be responsible for the hints. This reduces >>>> durability a bit, but might be a decent tradeoff. >>> The other interesting benefit is that the hint would not need to store the >>> actual content of the change, since the natural endpoints will already be >>> storing copies. The hints would just need to encode the fact that a given >>> (key,name1[,name2]) changed. >> >> Right, I think that's what Ryan was getting at. > > Indeed. Like I said, this change only helps with the "surprising > effects on a small cluster" problem, but if that's enough, perhaps we > should do it.
So, after having some more experience with HH, I've reformed my opinion. I think we have 3 options: 1. Make the natural endpoints responsible for the hints. 2. Make a random node responsible for hints. 3. Get rid of HH. #1 reduces the "surprising effects in a small cluster" problem by adding a marginal amount of resource demands to nodes that already have the data we need. #2 will spread the load out. We had a node die last week and decided to leave it down so that we could learn about the effects of this situation. We eventually ended up killing the next node on the ring with all the hints (I think there some improvements to this in 0.6, but I don't know if they'll be enough). So, even on a large cluster (ours is currently 45 nodes), HH can have surprising effects on nodes that neighbor a node that's down. Picking either a random node or using the coordinator node for the hint would spread the load out. #3 is, I think, the right answer. It make our system simpler and it makes the behavior in failure conditions more predictable and safe. What do you all think? -ryan
