General mechanism for fetching metrics via RPC sounds like a good
idea. Especially since Kafka has clients in many languages, but
support for JMX is not wide-spread outside the Java ecosystem.
Command-line script seems like a bad fit for use-cases where you want
the clients themselves to use
Hi,
In general, metrics are a "broker status API" telling you important things
about the state of the broker, its performance, etc. etc., right? What
argument is there for creating a separate API for this particular metric? If
you argue that JMX is not convenient, it seems like that would
Hi Noa,
Re; uptime. Sure, it was just a suggestion. However, we should be clear
that there are actually two timestamps at play. Your initial proposal
suggested using the timestamp from the registration znode. However, this
changes each time the broker loses its session. It does not reflect the
Hi Noa,
I understand the desire for batching. However, once you do that, you either
need request forwarding or broker metadata propagation. It's worth
exploring, but I suspect you will end up with most of the changes needed by
the original proposal, in that case.
Ismael
On Mon, Nov 18, 2019 at
Hi Jason & Gwen,
Sure, this would solve our use case. I do have two questions, however:
Moving from start time to uptime in theory would neatly side step the clock
skew problem,
but in practice I’m not sure it helps that much as the straightforward way of
going about
implementing this by
Strong +1 to broker status API. It will be super helpful for other
operational use-cases.
If not too much extra effort, having "has leaders" or "number of
leaders" in there, will be useful too - it is only safe to modify the
network layer (change LB config, for instance) when a broker is no
Sorry, that should be "Hi Noa"
On Wed, Nov 13, 2019 at 8:43 AM Jason Gustafson wrote:
> Ni Noa,
>
> Thanks for the KIP. I agree with the concerns about Metadata bloat. In
> fact, I have wanted a BrokerStatus API for a while now. Perhaps this is a
> good excuse to introduce it. I was thinking
Ni Noa,
Thanks for the KIP. I agree with the concerns about Metadata bloat. In
fact, I have wanted a BrokerStatus API for a while now. Perhaps this is a
good excuse to introduce it. I was thinking something like this:
BrokerStatusRequest => BrokerId
BrokerStatusResponse =>
Listeners =>
I agree with that. And looking at the MetadataResponse fields it seems there
has been some feature creep already here. Does the client use rack information,
for example?
I guess one could do this either by introducing a new leaner message pair, used
specifically enable client operation, and
It isn't just about saving space. It increases complexity to default to
always sharing a bit of information that is really only needed in a single
use-case.
We avoid doing this as a matter of good protocol design.
Arguably, this should not really piggyback on cluster metadata at all,
since the
It would certainly be possible to have the field be optional and only include
it if some flag is set in the DescribeClusterOptions instance passed to
Admin.describeCluster() that in turn would translate to a boolean in
MetadataRequest indicating that we are asking for this piece of information.
Cluster metadata is sent to clients on a very regular basis. Adding
start-time there seems quite repetitive. Especially considering that
this information is only useful in very specific cases.
Can we add this capability to the Admin API in a way that won't impact
normal client workflow?
On Mon,
Thank you for the feedback, Stanislav!
I agree that it would be good to harmonise the naming, and start-time-ms
definitely more descriptive.
I have updated the proposal to reflect this, and also added the updated json
RPC changes. Please have a look.
/noa
> On 1 Nov 2019, at 09:13, Stanislav
Hey Noa,
KIP-436 added a JMX metric in Kafka for this exact use-case, called
`start-time-ms`. Perhaps it would be useful to name this public interface
in the same way for consistency.
Could you update the KIP to include the specific RPC changes regarding the
metadata request/responses? Here is a
We are in the process of migrating the pieces of automation that currently
reads and modifies zookeeper state to use the Admin API.
One of the things that we miss doing this is access to the start time of
brokers in a cluster which is used by our automation doing rolling restarts. We
15 matches
Mail list logo
