Re: Feature Discussion: Enhancing Kafka Connect's transformation interface to 1 -> Many / Many -> Many

Supporting 1 Kafka record -> many rows in sink connectors would be really useful to properly support some data formats without incurring additional overhead. As an example, when consuming OpenTelmetry OTLP protobuf data, each record represents multiple data points, which significantly reduces

Re: [DISCUSS] KIP-714: Client metrics and observability

> > Are there cases where the metrics plugin developers would want to forward > the compressed payload without decompressing? The only interoperable use-case I can think of would be to forward the payloads directly to an OpenTelemetry collector backend. Today OTLP only mandates gzip/none

Re: [DISCUSS] KIP-714: Client metrics and observability

> > 28. On the broker, we typically use Yammer metrics. Only for metrics that > depend on Kafka metric features (e.g., quota), we use the Kafka metric. > Yammer metrics have 4 types: gauge, meter, histogram and timer. meter > calculates a rate, but also exposes an accumulated value. > I don't see

Re: [DISCUSS] KIP-714: Client metrics and observability

24.2 Does delta only apply to Counter type? > 24.3 In the delta representation, the first request needs to send the full > value, how does the broker plugin know whether a value is full or delta? > The temporarily semantics are defined by the OpenTelemetry data model. Deferring to OpenTelemetry

Re: [DISCUSS] KIP-714: Client metrics and observability

> > 1. Did you consider using a `default ClientTelemetryReceiver > clientReceiver() { return null; }` method on the existing MetricsReporter > interface, avoiding the need for the ClientTelemetry trait? I did. Part of the motivation was to separate more clearly the MetricsReporter methods which

Re: [VOTE] KIP-748: Add Broker Count Metrics

I had one quick question in the discussion thread. Any chance we can provide some thought there? On Thu, Jun 10, 2021 at 9:46 AM Ryan Dielhenn wrote: > Hello, > > I would like to start a vote on KIP-748: Add Broker Count Metrics. > > Here is the KIP: > >

Re: [DISCUSS] KIP-748: Add Broker Count Metrics

Any reason we need two different metrics for ZK an Quorum based controllers? Wouldn't it make sense to have one metric that abstracts the implementation detail? On Mon, Jun 7, 2021 at 2:29 PM Ryan Dielhenn wrote: > Hey Colin and David, > > I added another table for the ZK version of

Re: [VOTE] KIP-629: Use racially neutral terms in our codebase

gt; >>> +1 (non-binding) > > > > >>> > > > > >>> On Fri, Jun 26, 2020 at 3:41 AM Jay Kreps > > > wrote: > > > > >>>> > > > > >>>> +1 > > > > >>&

[VOTE] KIP-629: Use racially neutral terms in our codebase

Hi Everyone, I would like to initiate the voting process for KIP-629. https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase Thank you, Xavier

Re: [DISCUSS] KIP-629: Use racially neutral terms in our codebase

> Probably obvious but is documentation and website considered as well as > part of the KIP? > Documentation and website changes don't require a KIP to my knowledge, however we should also update them as needed (beyond the obvious documentation updates for the configuration names).

Re: [DISCUSS] KIP-629: Use racially neutral terms in our codebase

Please check the list for updated config / argument names. I also added a proposal to replace the term "blackout" with "backoff", which is used internally in the replication protocol. On Mon, Jun 22, 2020 at 3:10 PM Xavier Léauté wrote: > I agree we could improve on s

Re: [DISCUSS] KIP-629: Use racially neutral terms in our codebase

20, 2020, at 09:31, Ron Dagostino wrote: > > > Yes. Thank you. > > > > > > > On Jun 20, 2020, at 12:20 AM, Gwen Shapira > wrote: > > > > > > > > Thank you so much for this initiative. Small change, but it makes our > > > > com

[DISCUSS] KIP-629: Use racially neutral terms in our codebase

Hi Everyone, There are a number of places in our codebase that use racially charged terms. I am proposing we update them to use more neutral terms. The KIP lists the ones I have found and proposes alternatives. If you see any I missed or did not consider, please reply and I'll add them.

Re: [DISCUSS] KIP-606: Add Metadata Context to MetricsReporter

On Sat, May 9, 2020 at 7:21 AM Randall Hauch wrote: > Thanks, Xavier. Looks great. > > On Fri, May 8, 2020 at 7:31 PM Xavier Léauté wrote: > > > > This does seem very useful. A minor request would be to mention the new > > > configs for Connect, Streams and clie

Re: [VOTE] KIP-606: Add Metadata Context to MetricsReporter

gt; On Wed, May 13, 2020 at 4:10 PM Gwen Shapira > wrote: > > > > > > > +1 (binding) > > > > Thanks for the proposal, Xavier. > > > > > > > > On Wed, May 13, 2020 at 11:54 AM Xavier Léauté > > wrote: > > > > > >

[VOTE] KIP-606: Add Metadata Context to MetricsReporter

Hi everyone, Folks seem happy with the state of the KIP, so I'd like to start the vote for KIP-606 https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter - Xavier

Re: [DISCUSS] KIP-606: Add Metadata Context to MetricsReporter

> This does seem very useful. A minor request would be to mention the new > configs for Connect, Streams and clients, specifically that because they > are optional they will not hinder upgrades, and because they are namespaced > are unlikely to clash or conflict with other configs from extensions.

[DISCUSS] KIP-606: Add Metadata Context to MetricsReporter

Hi Everyone, I've published a KIP to address some shortcoming of our current metrics reporter interface. Would appreciate feedback. https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter Thank you, Xavier

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

://github.com/prometheus/jmx_exporter > (*) On a 4-cores Xeon 8175M broker, hosting 1,000 replicas, the time to > fetch all beans dropped from 13 seconds to ~400 ms. > > Le ven. 8 nov. 2019 à 17:29, Guozhang Wang a écrit : > > > Sounds good, thanks. > > > > Guozhan

Re: [VOTE] KIP-544: Make metrics exposed via JMX configurable

Thanks everyone, KIP-544 has been adopted with 4 binding, and 2 non-binding votes. A PR for the corresponding changes has been submitted https://github.com/apache/kafka/pull/7674 Xavier On Mon, Nov 11, 2019 at 7:56 AM Stanislav Kozlovski wrote: > +1 (non-binding). Thanks Xavier > > > On Sat,

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

> > 1. I do feel there're similar needs for clients make JMX configurable. Some > context: in modules like Connect and Streams we have added / refactored a > large number of metrics so far [0, 1], and although we've added a reporting > level config [2] to clients, this is statically defined at

[VOTE] KIP-544: Make metrics exposed via JMX configurable

Hi everyone, I'd like to open up the vote for KIP-544 on making exposed JMX metrics more configurable. https://cwiki.apache.org/confluence/display/KAFKA/KIP-544%3A+Make+metrics+exposed+via+JMX+configurable Thank you! Xavier

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

> > Since these configs will work with Kafka's own metrics library, will the > configs be part of the clients' configurations? It would be good to point > that out explicitly in the KIP. > Those configs are currently only at the broker level. If we feel this is useful on the client as well, we

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

> > A follow-up question, maybe to list in the future work section as it's > somewhat parallel to this KIP: have you thought about implementing a REST > reporter for metrics? That's certainly an option, however it does not solve the problem for our users that still rely on JMX integration to

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

> > How would the practical application look like if this was implemented? > One useful application is to hide partition-level metrics, some of which may only be needed for debugging purposes. > Would monitoring agents switch between the whitelist and blacklist > periodically if they wanted to

[DISCUSS] KIP-544: Make metrics exposed via JMX configurable

Hi All, I wrote a short KIP to make the set of metrics exposed via JMX configurable. https://cwiki.apache.org/confluence/display/KAFKA/KIP-544%3A+Make+metrics+exposed+via+JMX+configurable Let me know what you think. Thanks, Xavier

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

> > This kind of change will be problematic to us as the total RequestsPerSec > will be double counted in our metric system as we do automatic aggregation. > It has been a headache for us as we always have to do some special handling > when querying such metrics. > I agree with Allen, most of the

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hi Steven, do you think you'll get a chance to address the points Ismael made? It'd be great to get this change into 1.1. Thanks! Xavier On Tue, Dec 19, 2017 at 12:20 PM Ismael Juma wrote: > Hi Steven, > > As a general rule, we don't freeze KIPs after the vote passes. It's >

Re: 1.1 KIPs

Hi Damian, I believe the list should also include KAFKA-5886 (KIP-91) which was voted for 1.0 but wasn't ready to be merged in time. On Tue, Jan 16, 2018 at 5:13 AM Damian Guy wrote: > Hi, > > This is a reminder that we have one week left until the KIP deadline of Jan >

Re: [VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

; > Colin > > > > > > On Mon, Dec 11, 2017, at 13:14, Ewen Cheslack-Postava wrote: > > > +1 (binding) > > > > > > -Ewen > > > > > > On Mon, Dec 11, 2017 at 12:40 PM, Gwen Shapira <g...@confluent.io> > > wrote: >

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

ended as a public API, and > > > it's a little weird. whenComplete is nice because it supports > chaining, > > > and should be more familiar to users of other async APIs. > > > > > > best, > > > Colin > > > > > > > > > On Fri, Dec 8, 201

Re: [VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

remarks in your PR. > > Thanks > > Op za 9 dec. 2017 om 01:29 schreef Xavier Léauté <xav...@confluent.io>: > > > Hi Steve, I just posted in the discussion thread, there's just one tiny > fix > > I think would be useful to add while we're making changes to this API. >

Re: [VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hi Steve, I just posted in the discussion thread, there's just one tiny fix I think would be useful to add while we're making changes to this API. Do you mind having a look? On Fri, Dec 8, 2017 at 11:37 AM Mickael Maison wrote: > +1 (non binding) > Thanks for the KIP >

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hi Steven, I noticed you are making KafkaFuture.addWaiter(...) public as part of your PR. This is a very useful method to add – and you should mention it in the KIP – however addWaiter currently doesn't guard against exceptions thrown inside of the BiConsumer function, which is something we

Re: [DISCUSS] KIP-228 Negative record timestamp support

I agree with Matthias that keeping -1 special will be prone to errors. We should accept this is mistake resulting from lack of foresight on our part when adding timestamps in the first place and correct it. Using deltas will probably cause lots of headaches. It means we have to figure out the

Re: Adding log4j-extras to Apache Kafka?

Is there anything that would prevents us from moving directly to log4j2 as the default log backend – independently of our efforts to move remaining pieces to sl4fj? Unless we have some very custom code, it should be possible to rely on log4j-1.2-api.jar to avoid having to migrate everything at

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

4, 2017 at 4:26 PM, Richard Yu <yohan.richard...@gmail.com> > wrote: > > > I think we can come up with this compromise: range(long timeFrom, long > > timeTo) will be changed to getKeys(long timeFrom, long timeTo). Sounds > fair? > > > > > > On Tu

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

> > Generally I think having `all / range` is better in terms of consistency > with key-value windows. I.e. queries with key are named as `get / fetch` > for kv / window stores, and queries without key are named as `range / all`. > For kv stores, range takes a range of keys, and with this

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

wrote: > Thanks for the clarifications, Xavier. > I have removed most of the methods except for keys() and all() which has > been renamed to Guozhang Wang's suggestions. > > Hope this helps. > > On Fri, Oct 13, 2017 at 3:28 PM, Xavier Léauté <xav...@confluent.io>

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

Thanks for the KIP Richard, this is a very useful addition! As far as the API changes, I just have a few comments on the methods that don't seem directly related to the KIP title, and naming of course :). On the implementation, see my notes further down that will hopefully clarify a few things.

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

A few comments on the KIP: - I'm a bit confused about the BytesStoreSupplier interface. Nothing in its definition is really specific to Bytes, and when I see return types like BytesStoreSupplier>, it seems redundant to have "Bytes" in the supplier name. Why can't we

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

gt; > >>> If they are really co-aggregated, why don't we turn this around: > >> > >>> KGroupedStream<K, V1> grouped1 = builder.stream("topic1"). > >> > groupByKey(); > >> > >>> KGroupedStream<K, V2> grouped2 = builder.st

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

ers, >> >> Michał >> >> On 08/06/17 22:44, Guozhang Wang wrote: >> > Note that although the internal `AbstractStoreSupplier` does maintain the >> key-value serdes, we do not enforce the interface of `StateStoreSupplier` >> to always retain that informa

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

e: > > > This suggestion lgtm. I would vote for the first alternative than adding > > it to the `KStreamBuilder` though. > > > > On Thu, Jun 8, 2017 at 10:58 AM, Xavier Léauté <xav...@confluent.io> > > wrote: > > > >> I have a minor suggestion to make the API a

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

rent) > > >>> window > > >>>>>> spec in each co-grouped input stream. So if these window specs are > > >>>>>> different how should we "align" them with different input > streams? I > > >>>>> think > &g

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

wishList:{} > ] > > ... > > > I'm wondering if it makes more sense to only start sending the update if > the corresponding agg-key has seen at least one input from each of the > input stream? Maybe it is out of the scope of this KIP and we can make it a > more general d

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

Hi Kyle, I left a few more comments in the discussion thread, if you wouldn't mind taking a look On Fri, May 19, 2017 at 5:31 AM Kyle Winkelman wrote: > Hello all, > > I would like to start the vote on KIP-150. > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Sorry to jump on this thread so late. I agree this is a very useful addition and wanted to provide an additional use-case and some more comments. This is actually a very common analytics use-case in the ad-tech industry. The typical setup will have an auction stream, an impression stream, and a

Re: [VOTE] KIP-155 Add range scan for windowed state stores

when the windowed key's key field would always be the same, as it > is more consistent with the added APIs. But we need to do it in a backward > compatible way, which we can discuss in a follow-up KIP. > > > Guozhang > > On Thu, May 11, 2017 at 10:38 AM, Xavier Léauté <xav

Re: [VOTE] KIP-155 Add range scan for windowed state stores

en having consistency in return types versus > > consistency in having the ability to know the next key without moving > > the iterator. To me the latter just feels more important. > > > > Thanks, > > Michal > > On 11 May 2017 12:46 a.m., Xavier Léauté <xav...

Re: [VOTE] KIP-155 Add range scan for windowed state stores

> >> +1 > >> > >> Thanks, > >> Bill > >> > >> On Wed, May 10, 2017 at 2:38 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > >> > >>> +1. Thank you! > >>> > >>> On Wed, May 10, 2017 at 11:30 A

[VOTE] KIP-155 Add range scan for windowed state stores

Hi everyone, Since there aren't any objections to this addition, I would like to start the voting on KIP-155 so we can hopefully get this into 0.11. https://cwiki.apache.org/confluence/display/KAFKA/KIP+155+-+Add+range+scan+for+windowed+state+stores Voting will stay active for at least 72

Re: [DISCUSS] KIP-155 Add range scan for windowed state stores

I'll be driving the implementation as well. Unless there are specific concerns about the implementation, I mainly wrote the KIP to agree on the additional public interfaces. Current window store fetch operations already rely on range scanning & filtering the relevant keys in the underlying state

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

+1 (non-binding) On Tue, May 9, 2017 at 11:00 AM Dong Lin wrote: > +1 > > On Sun, May 7, 2017 at 7:40 PM, Jun Rao wrote: > > > Hi, Everyone, > > > > Since this is a relatively simple change, I would like to start the > voting > > process for KIP-153 :

Re: [DISCUSS] KIP-155 Add range scan for windowed state stores

I updated the KIP to reflect Michal's comment. Unless there are any further comments, I will initiate a vote this evening. On Mon, May 8, 2017 at 11:01 AM Xavier Léauté <xav...@confluent.io> wrote: > That sounds reasonable, Michal. Given the underlying implementation uses > the sa

Re: [DISCUSS] KIP-155 Add range scan for windowed state stores

P. Do you think the same should apply to session stores? > > IMHO, all three should be on par wrt the ability to query key ranges > (although I understand the implementation concerns for windowed stores are > more involved than for normal key value stores). > > Thanks, > > Michal >

[DISCUSS] KIP-155 Add range scan for windowed state stores

Hi everyone, I am proposing to add the ability to range scan windowed state stores in Kafka Streams. The required interface changes are relatively minimal and follow our existing conventions for state stores. Let me know if that sounds reasonable.

Re: [DISCUSS] KIP-153 : Include only client traffic in BytesOutPerSec metric

Any reason we are not keeping the per-topic breakdown for inter-broker traffic? On Fri, May 5, 2017 at 4:52 PM Onur Karaman wrote: > Looks good. Thanks! > > On Fri, May 5, 2017 at 4:44 PM, Roger Hoover > wrote: > > > Very helpful. Thank

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

One current benefit of having those classes extensible is the ability to write simple wrappers around KStreamBuilder to add functionality that currently doesn't exist. In my case, I extend KStreamBuilder mainly to provide syntactic sugar and make it easier to work with. For instance, I overload

Re: [DISCUSS] KIP-100 - Relax Type constraints in Kafka Streams API

the decision was made to go with invariant result types in those cases (for now). Let me know if you think everything makes sense. Xavier On Thu, Dec 8, 2016 at 10:35 AM Damian Guy <damian@gmail.com> wrote: Hi Xavier, The KIP looks good - thanks! Damian On Thu, 8 Dec 2016 at 18:12

Re: [VOTE] KIP-100 - Relax Type constraints in Kafka Streams API

te as per your proposal. > However, it seems that only the first case is covered by your proposal. > This is an improvement, but is there any reason not to do the latter as > well? It would be good to get it completely right this time. > > Ismael > > [1] http://openjdk.java.ne

[VOTE] KIP-100 - Relax Type constraints in Kafka Streams API

Hi everyone, I would like to start the vote for KIP-100 unless there are any more comments. https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+Relax+Type+constraints+in+Kafka+Streams+API corresponding PR here https://github.com/apache/kafka/pull/2205 Thanks, Xavier

[DISCUSS] KIP-100 - Relax Type constraints in Kafka Streams API

I recently filed https://issues.apache.org/jira/browse/KAFKA-4481, which Guozhang agreed was a bug. However fixing this will require some minor changes to the existing Kafka Streams APIs. I've outlined the required changes and binary / source compatibility implications in the KIP

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

On Wed, Nov 30, 2016 at 2:00 PM, Jason Gustafson <ja...@confluent.io> > > > wrote: > > > > > > > +1. Thanks for the KIP! > > > > > > > > On Wed, Nov 30, 2016 at 1:47 PM, Gwen Shapira <g...@confluent.io> > > wrote: > > >

[VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

Based on the feedback KIP-96 seems pretty uncontroversial, so I'd like to initiate a vote on it. https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+count Xavier

Re: [DISCUSS] KIP-96 - Add per partition metrics for in-sync and assigned replica count

rote: > > > Hi Xavier, > > > > Thanks for the KIP. Sounds good to me. > > > > Ismael > > > > On Tue, Nov 29, 2016 at 12:40 AM, Xavier Léauté <xav...@confluent.io> > > wrote: > > > > > Hi, > > > > > > I created KIP-

[DISCUSS] KIP-96 - Add per partition metrics for in-sync and assigned replica count

Hi, I created KIP-96 to propose per partition in-sync / assigned replica metrics. Should be straightforward, but submitting it for proposal since we require it for metrics changes. Here's the link to the KIP:

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

> > I kind of agree with James that it is a bit questionable how valuable any > data in a delete marker can be since it will be deleted somewhat > nondeterministically. > One could argue that even in normal topics, assuming a time-based log retention policy is in place, any message will be

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Does this mean that starting with V4 requests we would allow storing null messages in compacted topics? The KIP should probably clarify the behavior for null messages where the tombstone flag is not net. On Wed, Oct 26, 2016 at 1:32 AM Magnus Edenhill wrote: > 2016-10-25