Hi Colin,
Can you give us more details on why you don't want this to be
part of the Kafka core. You are proposing KIP-500 which will take away
zookeeper and
writing this interim tools to change the zookeeper metadata doesn't make
sense to me. As George pointed out there are several benefits having it in
the system itself
instead of asking users to hack bunch of json files to deal with outage
scenario.
Thanks,
Harsha
On Fri, Sep 6, 2019 at 4:36 PM George Li
wrote:
> Hi Colin,
>
> Thanks for the feedback. The "separate set of metadata about blacklists"
> in KIP-491 is just the list of broker ids. Usually 1 or 2 or a couple in
> the cluster. Should be easier than keeping json files? e.g. what if we
> first blacklist broker_id_1, then another broker_id_2 has issues, and we
> need to write out another json file to restore later (and in which order)?
> Using blacklist, we can just add the broker_id_2 to the existing one. and
> remove whatever broker_id returning to good state without worrying how(the
> ordering of putting the broker to blacklist) to restore.
>
> For topic level config, the blacklist will be tied to
> topic/partition(e.g. Configs:
> topic.preferred.leader.blacklist=0:101,102;1:103where 0 & 1 is the
> partition#, 101,102,103 are the blacklist broker_ids), and easier to
> update/remove, no need for external json files?
>
>
> Thanks,
> George
>
> On Friday, September 6, 2019, 02:20:33 PM PDT, Colin McCabe <
> cmcc...@apache.org> wrote:
>
> One possibility would be writing a new command-line tool that would
> deprioritize a given replica using the new KIP-455 API. Then it could
> write out a JSON files containing the old priorities, which could be
> restored when (or if) we needed to do so. This seems like it might be
> simpler and easier to maintain than a separate set of metadata about
> blacklists.
>
> best,
> Colin
>
>
> On Fri, Sep 6, 2019, at 11:58, George Li wrote:
> > Hi,
> >
> > Just want to ping and bubble up the discussion of KIP-491.
> >
> > On a large scale of Kafka clusters with thousands of brokers in many
> > clusters. Frequent hardware failures are common, although the
> > reassignments to change the preferred leaders is a workaround, it
> > incurs unnecessary additional work than the proposed preferred leader
> > blacklist in KIP-491, and hard to scale.
> >
> > I am wondering whether others using Kafka in a big scale running into
> > same problem.
> >
> >
> > Satish,
> >
> > Regarding your previous question about whether there is use-case for
> > TopicLevel preferred leader "blacklist", I thought about one
> > use-case: to improve rebalance/reassignment, the large partition will
> > usually cause performance/stability issues, planning to change the say
> > the New Replica will start with Leader's latest offset(this way the
> > replica is almost instantly in the ISR and reassignment completed), and
> > put this partition's NewReplica into Preferred Leader "Blacklist" at
> > the Topic Level config for that partition. After sometime(retention
> > time), this new replica has caught up and ready to serve traffic,
> > update/remove the TopicConfig for this partition's preferred leader
> > blacklist.
> >
> > I will update the KIP-491 later for this use case of Topic Level config
> > for Preferred Leader Blacklist.
> >
> >
> > Thanks,
> > George
> >
> >On Wednesday, August 7, 2019, 07:43:55 PM PDT, George Li
> > wrote:
> >
> > Hi Colin,
> >
> > > In your example, I think we're comparing apples and oranges. You
> started by outlining a scenario where "an empty broker... comes up...
> [without] any > leadership[s]." But then you criticize using reassignment
> to switch the order of preferred replicas because it "would not actually
> switch the leader > automatically." If the empty broker doesn't have any
> leaderships, there is nothing to be switched, right?
> >
> > Let me explained in details of this particular use case example for
> > comparing apples to apples.
> >
> > Let's say a healthy broker hosting 3000 partitions, and of which 1000
> > are the preferred leaders (leader count is 1000). There is a hardware
> > failure (disk/memory, etc.), and kafka process crashed. We swap this
> > host with another host but keep the same broker.id, when this new
> > broker coming up, it has no historical data, and we manage to have the
> > current last offsets of all partitions set in
> > the replication-offset-checkpoint (if we don't set them, it could cause
> > crazy ReplicaFetcher pulling of historical data from other brokers and
> > cause cluster high latency and other instabilities), so when Kafka is
> > brought up, it is quickly catching up as followers in the ISR. Note,
> > we have auto.leader.rebalance.enable disabled, so it's not serving any
> > traffic as leaders (leader count = 0), even there are 1000 partitions
> > that this broker is the Preferred Leader.
> >
> > We need to make this broker not serving traffic for a few hours or days
> >
