Hi David,

Thanks for the detailed feedback.

I agree with your points. Most of the list in this KIP is configuration
that the client does not use, so the heartbeat and share fetch responses
would be distributing a group resource rather than carrying something the
member acts on. That is also where most of the complexity comes from (the
token, the per-key cache and its lifetime rules), and it bypasses
DESCRIBE_CONFIGS for values the member does not need. I don't think that
is worth it for logging.

On concrete use cases: I'm not aware of a report from a production system.
While checking your point that the effective value only differs when an
administrator narrows the broker bounds, I found another way it can
happen. A group config override is validated by the controller against
the controller's own group bounds, but it is clamped by the broker against
the broker's bounds. With a separate controller that has no group.consumer.*
settings (default 45000-60000) and a broker with
group.consumer.min.session.timeout.ms=20000,
group.consumer.session.timeout.ms=25000 and
group.consumer.max.session.timeout.ms=30000, on trunk at 601492157a:

- --alter consumer.session.timeout.ms=45000 succeeded, and the broker
  immediately logged that the value is capped to 30000;
- DescribeConfigs still returned 45000, and session expiry about 28
  seconds after SIGKILL was consistent with the capped timeout;
- 25000, which the broker would accept, was rejected with "must be in the
  range 45000 to 60000".

So when the two sets of bounds differ, an override can be out of effect
from the moment it is set, without anyone changing the bounds, and when
they don't overlap, every accepted override is clamped. I'm not sure
whether operators are expected to set the group bounds on the controllers
as well. Is this known? If not, I can file a ticket with the
reproduction. This reinforces the need to expose effective values through
DescribeConfigs, but does not by itself justify distributing them through
the heartbeat or fetch APIs.

I'd be glad to work on DescribeConfigs returning the effective values with
your help. Some questions to scope it:

- Should the value be the effective one, with the stored override still
  visible so that a clamped override can be told apart, for example as
  the DYNAMIC_GROUP_CONFIG synonym?
- The admin client sends a group config describe to any broker, and the
  effective value depends on that broker's bounds. Is reporting the
  responding broker's view enough?
- Should streams.assignor.name, when the named assignor is not available,
  be covered the same way?
- Would you treat this as a change that needs a KIP?

Matthias, KAFKA-20848 proposes surfacing all effective group configs for
client-side logging. Given David's concerns, would improving
DescribeConfigs address the use case you had in mind, or is there a
specific client-side diagnostic need that would remain, such as a warning
when an override is not in effect, without the values?

I'll update the proposal based on the agreed scope.

Best,
Sunmin

Reply via email to