Hi, I've made a fairly major update to KIP-179 to propose APIs for setting throttled rates and throttled replicas with the ability to remove these automatically at the end of reassignment.
I'd be grateful for your feedback: https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+ReassignPartitionsCommand+to+use+AdminClient Thanks, Tom On 2 October 2017 at 13:15, Tom Bentley <t.j.bent...@gmail.com> wrote: > One question I have is about whether/how to scope throttling to a > reassignment. Currently throttles are only loosely associated with > reassignment: You can start a reassignment without any throttling, add > throttling to an in-flight reassignment, and remember/forget to remove > throttling after the reassignment is complete. There's is great flexibility > in that, but also the risk that you forget the remove the throttle(s). > > Just adding an API for setting the throttled rate makes this situation > worse: While it's nice to be able to auto-remove the throttles rate what > about the config for the throttled replicas? Also you might add a throttle > thinking a reassignment is in-flight, but it has in fact just finished: > Those throttles will now hang around until reset or the end of the next > reassignment. For these reasons it would be good if the throttle were more > directly scoped to the reassignment. > > On the other hand, taking LinkedIn's Cruise Control as an example, there > they seem to modify the reassignment znode directly and incrementally and > so there is no notion of "the reassignment". Reassignments will be running > continuously, with partitions added before all of the current partitions > have completed. If there is no meaningful cluster-wide "reassignment" then > it would be better to remove remove the throttle by changing the list of > replicas as each replica catches up. > > I'm interested in any use cases people can share on this, as I'd like the > throttle API to be useful for a broad range of use cases, rather than being > too narrowly focussed on what's needed by the existing CLI tools. > > Thanks, > > Tom > > > > > On 28 September 2017 at 17:22, Tom Bentley <t.j.bent...@gmail.com> wrote: > >> I'm starting to think about KIP-179 again. In order to have more >> manageably-scoped KIPs and PRs I think it might be worth factoring-out the >> throttling part into a separate KIP. Wdyt? >> >> Keeping the throttling discussion in this thread for the moment... >> >> The throttling behaviour is currently spread across the >> `(leader|follower).replication.throttled.replicas` topic config and the >> `(leader|follower).replication.throttled.rate` dynamic broker config. >> It's not really clear to me exactly what "removing the throttle" is >> supposed to mean. I mean we could reset the rate to Long.MAV_VALUE or we >> could change the list of replicas to an empty list. The >> ReassignPartitionsCommand does both, but there is some small utility in >> leaving the rate, but clearing the list, if you've discovered the "right" >> rate for your cluster/workload and to want it to be sticky for next time. >> Does any one do this in practice? >> >> With regards to throttling, it would be >>>> worth thinking about a way where the throttling configs can be >>>> automatically removed without the user having to re-run the tool. >>>> >>> >>> Isn't that just a matter of updating the topic configs for >>> (leader|follower).replication.throttled.replicas at the same time we >>> remove the reassignment znode? That leaves open the question about whether >>> to reset the rates at the same time. >>> >> >> Thinking some more about my "update the configs at the same time we >> remove the reassignment znode" suggestion. The reassignment znode is >> persistent, so the reassignment will survive a zookeeper restart. If there >> was a flag for the auto-removal of the throttle it would likewise need to >> be persistent. Otherwise a ZK restart would remember the reassignment, but >> forget about the preference for auto removal of throttles. So, we would use >> a persistent znode (a child of the reassignment path, perhaps) to store a >> flag for throttle removal. >> >> Thoughts? >> >> Cheers, >> >> Tom >> > >
