Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12158 )
Change subject: KUDU-2348: Pick a random replica in RemoteTablet.java ...................................................................... Patch Set 9: Thinking about this a bit, I wonder what behavior we really want. Today, every client inserts the servers into a hashmap, and then would return the last server in hashmap iteration order. In other words, we end up ranking the servers by something like hashcode%num_hashmap_buckets. Given that num_hashmap_buckets is likely constant across all tablets (these hashtables almost always have 3 elements so we likely have the default of 16 buckets, the ranking function is more or less consistent across all clients and all tablets. The major problem this causes is this: in a cluster without locality (eg fully remote), whichever servers have high hashcode%16 are going to get significantly more read load than those with low hashcode%16. I wrote a little simulation here: https://gist.github.com/b3de552784da4afa29a2f1f66673b187 Running this script results in a load distribution like: ts_idx % of load ----------------- 0 10.1 2 9.2 27 8.0 12 8.0 10 7.8 13 6.5 11 6.2 22 6.2 24 6.0 20 5.0 23 4.0 15 3.6 18 2.9 3 2.9 21 2.9 16 2.1 4 1.8 25 1.2 6 1.1 5 1.1 9 1.0 1 1.0 7 0.4 19 0.3 26 0.3 29 0.2 28 0.0 8 0.0 14 0.0 17 0.0 It seems that this patch will change the behavior so that the server preference is randomized and dependent on the client, which solves the issue, but also means that, for a given tablet, load will be spread evenly across the replicas if there are multiple clients. Depending on the workload, that may be good or bad -- in many cases you would prefer _not_ to spread the load, so that you can make more efficient use of cache memory. The spreading of load is then accomplished by partitioning rather than replication. Anyone have thoughts on how we might express this preference through the API? A separate concern with the particular implementation is that pid may have a lot of correlation across machines, particularly if the client is running inside Docker containers or set to start at boot. AFAIK pids are sequentially assigned, so within Docker containers you would expect all clients to end up with identical pids. If we need a randomized id for a process I think it's better to use Java's random number generation to get one and assign it in a static intializer. -- To view, visit http://gerrit.cloudera.org:8080/12158 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d70e45d4c9532bb32223c1dddd0936b4ff8fd99 Gerrit-Change-Number: 12158 Gerrit-PatchSet: 9 Gerrit-Owner: Yifan Zhang <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Will Berkeley <[email protected]> Gerrit-Reviewer: Yifan Zhang <[email protected]> Gerrit-Comment-Date: Wed, 29 May 2019 16:33:22 +0000 Gerrit-HasComments: No
