Hey Jay,
Thanks for the feedback.
1. We can certainly discuss what it means to remove the file configuration as a
thought exercise. However, is this something we want to do for real? IMO, we
can remove file configuration by having all configs stored in zookeeper. The
flow can be:
- Broker starts and reads all the configs from ZK. (overrides)
- Apply them on top of the defaults that are hardcoded within the broker. This
should simulate file based config behavior as it is currently.
- Potentially, we can write back all the merged configs to zookeeper (defaults
+ overrides). This means that the entire config of that broker is in ZK.
Thoughts?
2. Good point. All overridden configs (topic, client level) will have a
corresponding broker config that serves as a default. It should be sufficient
to change that broker config dynamically and that effectively means that the
default has been changed. The overrides on a per topic/client basis still take
precedence. So yeah, I don't think we need to model defaults explicitly. Using
an example to be sure we are on the same page, lets say we wanted to increase
the log retention time for all topics without having to create a separate
override for each topic, we could simply change the "log.retention.time" under
"/brokers/<broker_id>" to the desired value and that should change the default
log retention for everyone (apart from the explicitly overridden ones on a
per-topic basis).
3. I thought it was cleaner to simply separate the producer and consumer
configs but I guess if they present the same clientId, they are essentially the
"same" client. I'll follow your suggestion.
4. Interesting that you mention this. I actually thought about having callbacks
but I left it out of the initial proposal since I wanted to keep it relatively
simple. The only configs we can change by checking references are the ones we
check frequently while processing requests or (or something periodic). I shall
incorporate this on the KIP.
5. What you are proposing sounds good. Initially, I was planning to push down
everything to KafkaConfig by not having immutable vals within. Having a wrapper
(KafkaConfiguration) like you suggest is probably cleaner.
One implementation detail. There don't appear to be any concerns wrt the client
based config section (and the topic config already exists). Are there any
concerns if we keep implementation of the per-client config piece and
generalizing the code in TopicConfigManager separate from the broker config
section? Client configs are an immediate requirement to operationalize quotas
(perhaps can be used to manage authorization also for security). The broker
side changes to mark configs dynamic, implement callbacks etc.. can be
implemented as a followup task since it will take longer to identity which
configs can be made dynamic and actually doing the work to make them so. I
think that once we have reasonable agreement on the overall picture, we can
implement these things piece by piece.
Thanks,
Aditya
________________________________________
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Friday, May 01, 2015 12:53 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-21 Configuration Management
Hey Aditya,
This is a great! A couple of comments:
1. Leaving the file config in place is definitely the least disturbance.
But let's really think about getting rid of the files and just have one
config mechanism. There is always a tendency to make everything pluggable
which so often just leads to two mediocre solutions. Can we do the exercise
of trying to consider fully getting rid of file config and seeing what goes
wrong?
2. Do we need to model defaults? The current approach is that if you have a
global config x it is overridden for a topic xyz by /topics/xyz/x, and I
think this could be extended to /brokers/0/x. I think this is simpler. We
need to specify the precedence for these overrides, e.g. if you override at
the broker and topic level I think the topic level takes precedence.
3. I recommend we have the producer and consumer config just be an override
under client.id. The override is by client id and we can have separate
properties for controlling quotas for producers and consumers.
4. Some configs can be changed just by updating the reference, others may
require some action. An example of this is if you want to disable log
compaction (assuming we wanted to make that dynamic) we need to call
shutdown() on the cleaner. I think it may be required to register a
listener callback that gets called when the config changes.
5. For handling the reference can you explain your plan a bit? Currently we
have an immutable KafkaConfig object with a bunch of vals. That or
individual values in there get injected all over the code base. I was
thinking something like this:
a. We retain the KafkaConfig object as an immutable object just as today.
b. It is no longer legit to grab values out fo that config if they are
changeable.
c. Instead of making KafkaConfig itself mutable we make KafkaConfiguration
which has a single volatile reference to the current KafkaConfig.
KafkaConfiguration is what gets passed into various components. So to
access a config you do something like config.instance.myValue. When the
config changes the config manager updates this reference.
d. The KafkaConfiguration is the thing that allows doing the
configuration.onChange("my.config", callback)
-Jay
On Tue, Apr 28, 2015 at 3:57 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:
> Hey everyone,
>
> Wrote up a KIP to update topic, client and broker configs dynamically via
> Zookeeper.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
>
> Please read and provide feedback.
>
> Thanks,
> Aditya
>
> PS: I've intentionally kept this discussion separate from KIP-5 since I'm
> not sure if that is actively being worked on and I wanted to start with a
> clean slate.
>
