Hey Aditya,
This is a great! A couple of comments:
1. Leaving the file config in place is definitely the least disturbance.
But let's really think about getting rid of the files and just have one
config mechanism. There is always a tendency to make everything pluggable
which so often just leads to two mediocre solutions. Can we do the exercise
of trying to consider fully getting rid of file config and seeing what goes
wrong?
2. Do we need to model defaults? The current approach is that if you have a
global config x it is overridden for a topic xyz by /topics/xyz/x, and I
think this could be extended to /brokers/0/x. I think this is simpler. We
need to specify the precedence for these overrides, e.g. if you override at
the broker and topic level I think the topic level takes precedence.
3. I recommend we have the producer and consumer config just be an override
under client.id. The override is by client id and we can have separate
properties for controlling quotas for producers and consumers.
4. Some configs can be changed just by updating the reference, others may
require some action. An example of this is if you want to disable log
compaction (assuming we wanted to make that dynamic) we need to call
shutdown() on the cleaner. I think it may be required to register a
listener callback that gets called when the config changes.
5. For handling the reference can you explain your plan a bit? Currently we
have an immutable KafkaConfig object with a bunch of vals. That or
individual values in there get injected all over the code base. I was
thinking something like this:
a. We retain the KafkaConfig object as an immutable object just as today.
b. It is no longer legit to grab values out fo that config if they are
changeable.
c. Instead of making KafkaConfig itself mutable we make KafkaConfiguration
which has a single volatile reference to the current KafkaConfig.
KafkaConfiguration is what gets passed into various components. So to
access a config you do something like config.instance.myValue. When the
config changes the config manager updates this reference.
d. The KafkaConfiguration is the thing that allows doing the
configuration.onChange("my.config", callback)
-Jay
On Tue, Apr 28, 2015 at 3:57 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:
> Hey everyone,
>
> Wrote up a KIP to update topic, client and broker configs dynamically via
> Zookeeper.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
>
> Please read and provide feedback.
>
> Thanks,
> Aditya
>
> PS: I've intentionally kept this discussion separate from KIP-5 since I'm
> not sure if that is actively being worked on and I wanted to start with a
> clean slate.
>
