Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1543

[Apache Jenkins Server]

See 

Re: [DISCUSS] Tiered-Storage: Implement RocksdbBasedMetadataCache for TopicBased RLMM

[Alexandre Dupriez]

Hi, hzh0425,

Thank you for raising the question. There have been discussions about
using RocksDB as a RLMM in the community, one of which can be found on
the dev list [1].
What is the size of the metadata set in your use case? Have you
considered using an external data store to abstract metadata
persistence away from the local storage?

[1] https://lists.apache.org/thread/8lcslnwrnj1s7mk2c3g3fw0zqjwrogds

Thanks,
Alexandre

Le sam. 21 janv. 2023 à 05:35, hzh0425  a écrit :
>
> Background:
> In KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software Foundation, 
>  kafka introduced the feature of hierarchical storage, and RLMM is 
> responsible for storing remote segment's metadata.
>
>
> BTW, [KAFKA-9555] Topic-based implementation for the RemoteLogMetadataManager 
> - ASF JIRA (apache.org) implements the default RLMM - 'TopicBased-RLMM'.
>
>
>
>
> Problem:
> TopicBased RLMM stores all metadata of subscriptions in memory.
>
> In our practice, we found that when the metadata gradually increases, there 
> will be a huge burden on the broker's memory (GB level), and at the same 
> time, it will be very time-consuming to save the snapshot of the full amount 
> of metadata to the disk.
>
>
>
>
> Solution
>
> We hope to introduce rocksdb to solve this problem:
> - Implement a RocksdbBasedMetadataCache
> - All metadata is stored on disk, only a small amount of rocksdb memory cache 
> is required.
> - There is no need to bear the time consumption caused by saving the full 
> amount of snapshot metadata to disk, rocksdb can guarantee incremental 
> storage.
>
>
> You are welcome to discuss this Improvement by replying email !
>
>
> Thanks,
> Hzh0425
>
>
> | |
> hzhkafka
> |
> |
> hzhka...@163.com
> |

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

[Guozhang Wang]

Thanks for all the very helpful discussions, I'm closing the vote with
a tally here:

+1: 7 (Nick, John, Walker, Bruno, Lucas, Matthias, Guozhang), with 5
binding votes and 2 non-binding votes.
-1: 0


Guozhang

On Wed, Jan 25, 2023 at 5:48 PM Matthias J. Sax  wrote:
>
> Thanks!
>
> +1 (binding)
>
> -Matthias
>
> On 1/24/23 1:17 PM, Guozhang Wang wrote:
> > Hi Matthias:
> >
> > re "paused" -> "suspended": I got your point now, thanks. Just to
> > clarify the two functions are a bit different: "paused" tasks are
> > because of the topology being paused, i.e. from KIP-834; whereas
> > "suspended" tasks are when a restoring tasks are being removed before
> > it completes due to a follow-up rebalance, and this is to distinguish
> > with "onRestoreEnd", as described in KAFKA-10575. A suspended task is
> > no longer owned by the thread and hence there's no need to measure the
> > number of such tasks.
> >
> > re: "restore-ratio": that's a good point. I like it to function in the
> > same way as the "records-rate" metrics. Will update the wiki.
> >
> > re: making "restore-remaining-records-total" at INFO level: sounds
> > good to me too. I will also update the metric name a bit to be more
> > specific.
> >
> >
> >
> > On Thu, Jan 19, 2023 at 2:35 PM Guozhang Wang
> >  wrote:
> >>
> >> Hello Matthias,
> >>
> >> Thanks for the feedback. I was on vacation for a while. Pardon for the
> >> late replies. Please see them inline below
> >>
> >> On Thu, Dec 1, 2022 at 11:23 PM Matthias J. Sax  wrote:
> >>>
> >>> Seems I am late to the party... Great KIP. Couple of questions from my 
> >>> side:
> >>>
> >>> (1) What is the purpose of `standby-updating-tasks`? It seems to be the
> >>> same as the number of assigned standby task? Not sure how useful it
> >>> would be?
> >>>
> >> In general, yes, it is the number of assigned standby tasks --- there
> >> will be transit times when the assigned standby tasks are not yet
> >> being updated but it would not last long --- but we do not yet have a
> >> direct gauge to expose this before, and users have to infer this from
> >> other indirect metrics.
> >>
> >>>
> >>>
> >>> (2) `active-paused-tasks` / `standby-paused-tasks` -- what does "paused"
> >>> exactly mean? There was a discussion about renaming the callback method
> >>> from pause to suspended. So should this be called `suspended`, too? And
> >>> if yes, how is it useful for users?
> >>>
> >> Pausing here refers to "KIP-834: Pause / Resume KafkaStreams
> >> Topologies" 
> >> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832).
> >> When a topology is paused, all its tasks including standbys will be
> >> paused too.
> >>
> >> I'm not aware of a discussion to rename the call name to "suspend" for
> >> KIP-834. Could you point me to the reference?
> >>
> >>>
> >>>
> >>> (3) `restore-ratio`: the description says
> >>>
>  The fraction of time the thread spent on restoring active or standby 
>  tasks
> >>>
> >>> I find the term "restoring" does only apply to active tasks, but not to
> >>> standbys. Can we reword this?
> >>>
> >> Yeah I have been discussing this with others in the community a bit as
> >> well, but so far I have not been convinced of a better name than it.
> >> Some other alternatives being discussed but not win everyone's love is
> >> "restore-or-update-ratio", "process-ratio" (for the restore thread
> >> that means restoring or updating), and "io-ratio".
> >>
> >> The only one so far that I feel is probably better, is
> >> "state-update-ratio". If folks feel this one is better than
> >> "restore-ratio" I'm happy to update.
> >>
> >>>
> >>> (4) `restore-call-rate`: not sure what you exactly mean by "restore 
> >>> calls"?
> >>>
> >> This is similar to the "io-calls-rate" in the selector classes, i.e.
> >> the number of "restore" function calls made. It's argurably a very
> >> low-level metrics but I included it since it could be useful in some
> >> debugging scenarios.
> >>
> >>>
> >>> (5) `restore-remaining-records-total` -- why is this a task metric?
> >>> Seems we could roll it up into a thread metric that we report at INFO
> >>> level (we could still have per-task DEBUG level metric for it in 
> >>> addition).
> >>>
> >> The rationale behind it is the general principle in metrics design
> >> that "Kafka would provide the lowest necessary metrics levels, and
> >> users can do the roll-ups however they want".
> >>
> >>>
> >>> (6) What about "warmup tasks"? Internally, we treat them as standbys,
> >>> but it seems it's hard for users to reason about it in the scale-out
> >>> warm-up case. Would it be helpful (and possible) to report "warmup
> >>> progress" explicitly?
> >>>
> >> At the restore thread level, we cannot differentiate standby tasks
> >> from warmup tasks since the latter is created exactly just like the
> >> former. But I do agree this is an issue for visibility that worth
> >> addressing, I think another KIP would be needed to first consider
> >> distinguishing these 

[GitHub] [kafka-site] mimaison commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

mimaison commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1405361230

   @mjsax The reason I did not merge is because I don't understand why when 
testing locally we need changes to the `.htaccess` file that don't seem 
necessary on the website. See the few comments starting from 
https://github.com/apache/kafka-site/pull/410#issuecomment-1126118797. Do you 
have an idea why this may be the case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-821: Connect Transforms support for nested structures

[Mickael Maison]

+1 (binding)
Thanks for the KIP

On Tue, Jun 28, 2022 at 10:42 PM Jorge Esteban Quilcate Otoya
 wrote:
>
> Hi everyone,
>
> I'd like to bump this vote thread. Currently it's missing 1 +1 binding vote
> to pass (2 +1 binding, 1 +1 non-binding).
>
> There has been additional discussions to consider array access and
> deep-scan (similar to JsonPath) but hasn't been included as part of this
> KIP.
> The only minor change since the previous votes has been the change of
> configuration name: from `field.style` to `field.syntax.version`.
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
>
> Cheers,
> Jorge.
>
> On Fri, 22 Apr 2022 at 00:01, Bill Bejeck  wrote:
>
> > Thanks for the KIP, Jorge.
> >
> > This looks like a great addition to Kafka Connect.
> >
> > +1(binding)
> >
> > -Bill
> >
> > On Thu, Apr 21, 2022 at 6:41 PM John Roesler  wrote:
> >
> > > Thanks for the KIP, Jorge!
> > >
> > > I’ve just looked over the KIP, and it looks good to me.
> > >
> > > I’m +1 (binding)
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Apr 21, 2022, at 09:10, Chris Egerton wrote:
> > > > This is a worthwhile addition to the SMTs that ship out of the box with
> > > > Kafka Connect. +1 non-binding
> > > >
> > > > On Thu, Apr 21, 2022, 09:51 Jorge Esteban Quilcate Otoya <
> > > > quilcate.jo...@gmail.com> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'd like to start a vote on KIP-821:
> > > >>
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
> > > >>
> > > >> Thanks,
> > > >> Jorge
> > > >>
> > >
> >

[jira] [Created] (KAFKA-14656) Brokers rejecting LISR during ZK migration

[David Arthur (Jira)]

David Arthur created KAFKA-14656:


 Summary: Brokers rejecting LISR during ZK migration
 Key: KAFKA-14656
 URL: https://issues.apache.org/jira/browse/KAFKA-14656
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.4.0
Reporter: David Arthur
Assignee: David Arthur
 Fix For: 3.4.0


During the ZK migration, the KRaft controller sends controller RPCs to the ZK 
brokers (LISR, UMR, etc). Since the migration can begin immediately after a ZK 
broker starts up with migration enabled, it is possible that this broker is not 
seen as alive by the rest of the brokers. This is due to the KRaft controller 
taking over before the ZK controller can send out UMR with the restarted broker.

 

The result is that the parts of the LISR sent by KRaft immediately after the 
metadata migration is rejected by brokers due the leader being offline. 

 

The fix for this is to send an UMR to all brokers after the migration with the 
set of alive brokers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-864: Add End-To-End Latency Metrics to Connectors

[Jorge Esteban Quilcate Otoya]

No worries, thanks Chris!

I think most feedback has been covered and the KIP is ready for vote. Will
be starting the vote thread soon.

Cheers,
Jorge.

On Mon, 5 Dec 2022 at 15:10, Chris Egerton  wrote:

> Hi Jorge,
>
> Thanks for indulging my paranoia. LGTM!
>
> Cheers,
>
> Chris
>
> On Mon, Dec 5, 2022 at 10:06 AM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Sure! I have a added the following to the proposed changes section:
> >
> > ```
> > The per-record metrics will definitely be added to Kafka Connect as part
> of
> > this KIP, but their metric level will be changed pending the performance
> > testing described in KAFKA-14441, and will otherwise only be exposed at
> > lower level (DEBUG instead of INFO, and TRACE instead of DEBUG)
> > ```
> >
> > Let me know if how does it look.
> >
> > Many thanks!
> > Jorge.
> >
> > On Mon, 5 Dec 2022 at 14:11, Chris Egerton 
> > wrote:
> >
> > > Hi Jorge,
> > >
> > > Thanks for filing KAFKA-14441! In the ticket description we mention
> that
> > > "there will be more confidence whether to design metrics to be exposed
> > at a
> > > DEBUG or INFO level depending on their impact" but it doesn't seem like
> > > this is called out in the KIP and, just based on what's in the KIP, the
> > > proposal is still to have several per-record metrics exposed at INFO
> > level.
> > >
> > > Could we explicitly call out that the per-record metrics will
> definitely
> > be
> > > added to Kafka Connect as part of this KIP, but they will only be
> exposed
> > > at INFO level pending pending the performance testing described in
> > > KAFKA-14441, and will otherwise only be exposed at DEBUG level?
> > Otherwise,
> > > it's possible that a vote for the KIP as it's written today would be a
> > vote
> > > in favor of unconditionally exposing these metrics at INFO level, even
> if
> > > the performance testing reveals issues.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Sun, Dec 4, 2022 at 7:08 PM Jorge Esteban Quilcate Otoya <
> > > quilcate.jo...@gmail.com> wrote:
> > >
> > > > Thanks for the reminder Chris!
> > > >
> > > > I have added a note on the KIP to include this as part of the KIP as
> > most
> > > > of the metrics proposed are per-record and having all on DEBUG would
> > > limit
> > > > the benefits, and created
> > > > https://issues.apache.org/jira/browse/KAFKA-14441
> > > > to keep track of this task.
> > > >
> > > > Cheers,
> > > > Jorge.
> > > >
> > > > On Tue, 29 Nov 2022 at 19:40, Chris Egerton  >
> > > > wrote:
> > > >
> > > > > Hi Jorge,
> > > > >
> > > > > Thanks! What were your thoughts on the possible benchmarking and/or
> > > > > downgrading of per-record metrics to DEBUG?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Nov 24, 2022 at 8:20 AM Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jo...@gmail.com> wrote:
> > > > >
> > > > > > Thanks Chris! I have updated the KIP with "transform" instead of
> > > > "alias".
> > > > > > Agree it's clearer.
> > > > > >
> > > > > > Cheers,
> > > > > > Jorge.
> > > > > >
> > > > > > On Mon, 21 Nov 2022 at 21:36, Chris Egerton
> >  > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jorge,
> > > > > > >
> > > > > > > Thanks for the updates, and apologies for the delay. The new
> > > diagram
> > > > > > > directly under the "Proposed Changes" section is absolutely
> > > gorgeous!
> > > > > > >
> > > > > > >
> > > > > > > Follow-ups:
> > > > > > >
> > > > > > > RE 2: Good point. We can use the same level for these metrics,
> > it's
> > > > > not a
> > > > > > > big deal.
> > > > > > >
> > > > > > > RE 3: As long as all the per-record metrics are kept at DEBUG
> > > level,
> > > > it
> > > > > > > should be fine to leave JMH benchmarking for a follow-up. If we
> > > want
> > > > to
> > > > > > add
> > > > > > > new per-record, INFO-level metrics, I would be more comfortable
> > > with
> > > > > > > including benchmarking as part of the testing plan for the KIP.
> > One
> > > > > > > possible compromise could be to propose that these features be
> > > merged
> > > > > at
> > > > > > > DEBUG level, and then possibly upgraded to INFO level in the
> > future
> > > > > > pending
> > > > > > > benchmarks to guard against performance degradation.
> > > > > > >
> > > > > > > RE 4: I think for a true "end-to-end" metric, it'd be useful to
> > > > include
> > > > > > the
> > > > > > > time taken by the task to actually deliver the record. However,
> > > with
> > > > > the
> > > > > > > new metric names and descriptions provided in the KIP, I have
> no
> > > > > > objections
> > > > > > > with what's currently proposed, and a new "end-to-end" metric
> can
> > > be
> > > > > > taken
> > > > > > > on later in a follow-up KIP.
> > > > > > >
> > > > > > > RE 6: You're right, existing producer metrics should be enough
> > for
> > > > now.
> > > > > > We
> > > > > > > can revisit this later if/when we add delivery-centric metrics
> > for
> > > > sink
> > > > > > > tasks as 

[VOTE] KIP-864: Add End-To-End Latency Metrics to Connectors

[Jorge Esteban Quilcate Otoya]

Hi all,

I'd like to call for a vote on KIP-864, which proposes to add metrics to
measure end-to-end latency in source and sink connectors.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-864%3A+Add+End-To-End+Latency+Metrics+to+Connectors

Discussion thread:
https://lists.apache.org/thread/k6rh2mr7pg94935fgpqw8b5fj308f2n7

Many thanks,
Jorge.

Re: [VOTE] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Federico Valeri]

+1 (non binding)

Thanks.

On Thu, Jan 26, 2023 at 2:30 PM Gantigmaa Selenge  wrote:
>
> Hi
>
> I'd like to call for a vote on KIP-894, which updates MirrorMaker to use
> IncrementalAlterConfigs API to sync topic configurations between clusters.
>
> The KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations
>
> The discussion thread:
> https://lists.apache.org/thread/4chr4s5vbd9rqhml2tk60fsghojwo6bb
>
> Thank you
> Tina

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Yash,

1) To avoid conflicts with other sensors, the PluginMetrics
implementation will append a suffix to sensor names to unique identify
the plugin (based on the class name and tags). Also I changed the
semantics of the sensor() method to only create sensors (originally it
was get or create). If a sensor with the same name already exists, the
method will throw.
2) Tags will be automatically added to metrics and sensors to unique
identify the plugin. For Connect plugins, the connector name, task id
and alias can be added if available. The class implementing
PluginMetrics will be similar to ConnectMetrics, as in it will provide
a simplified API wrapping Metrics. I'm planning to use PluginMetrics
for Connect plugin too and should not need to interact with
ConnectMetrics.
3) Right, I fixed the last rejected alternative.

Thanks,
Mickael

On Thu, Jan 26, 2023 at 4:04 PM Mickael Maison  wrote:
>
> Hi Federico,
>
> - The metricName() method does not register anything, it just builds a
> MetricName instance which is just a container holding a name, group,
> description and tags for a metric. Each time it is called, it returns
> a new instance. If called with the same arguments, the returned value
> will be equal.
> - Initially I just copied the API of Metrics. I made some small
> changes so the metric and sensor methods are a bit more similar
> - Good catch! I fixed the example.
>
> Thanks,
> Mickael
>
>
> On Thu, Jan 26, 2023 at 3:54 PM Mickael Maison  
> wrote:
> >
> > Hi Chris,
> >
> > 1) I updated the KIP to only mention the interface.
> > 2) This was a mistake. I've added ReplicationPolicy to the list of plugins.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Jan 24, 2023 at 11:16 AM Yash Mayya  wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the updated KIP, this is looking really good! I had a couple
> > > more questions -
> > >
> > > 1) Sensor names need to be unique across all groups for a `Metrics`
> > > instance. How are we planning to avoid naming clashes (both between
> > > different plugins as well as with pre-defined sensors)?
> > >
> > > 2) Connect has a `ConnectMetrics` wrapper around `Metrics` via which
> > > rebalance / worker / connector / task metrics are recorded. Could you
> > > please elaborate in the KIP how the plugin metrics for connectors / tasks
> > > will inter-operate with this?
> > >
> > > Another minor point is that the third rejected alternative appears to be 
> > > an
> > > incomplete sentence?
> > >
> > > Thanks,
> > > Yash
> > >
> > > On Fri, Jan 13, 2023 at 10:56 PM Mickael Maison 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I've updated the KIP based on the feedback.
> > > >
> > > > Now instead of receiving a Metrics instance, plugins get access to
> > > > PluginMetrics that exposes a much smaller API. I've removed the
> > > > special handling for connectors and tasks and they must now implement
> > > > the Monitorable interface as well to use this feature. Finally the
> > > > goal is to allow all plugins (apart from metrics reporters) to use
> > > > this feature. I've listed them all (there are over 30 pluggable APIs)
> > > > but I've not added the list in the KIP. The reason is that new plugins
> > > > could be added in the future and instead I'll focus on adding support
> > > > in all the place that instantiate classes.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Jan 10, 2023 at 7:00 PM Mickael Maison 
> > > > 
> > > > wrote:
> > > > >
> > > > > Hi Chris/Yash,
> > > > >
> > > > > Thanks for taking a look and providing feedback.
> > > > >
> > > > > 1) Yes you're right, when using incompatible version, metrics() would
> > > > > trigger NoSuchMethodError. I thought using the context to pass the
> > > > > Metrics object would be more idiomatic for Connect but maybe
> > > > > implementing Monitorable would be simpler. It would also allow other
> > > > > Connect plugins (transformations, converters, etc) to register
> > > > > metrics. So I'll make that change.
> > > > >
> > > > > 2) As mentioned in the rejected alternatives, I considered having a
> > > > > PluginMetrics class/interface with a limited API. But since Metrics is
> > > > > part of the public API, I thought it would be simpler to reuse it.
> > > > > That said you bring interesting points so I took another look today.
> > > > > It's true that the Metrics API is pretty complex and most methods are
> > > > > useless for plugin authors. I'd expect most use cases only need one
> > > > > addMetric and one sensor methods. Rather than subclassing Metrics, I
> > > > > think a delegate/forwarding pattern might work well here. A
> > > > > PluginMetric class would forward its method to the Metrics instance
> > > > > and could perform some basic validations such as only letting plugins
> > > > > delete metrics they created, or automatically injecting tags with the
> > > > > class name for example.
> > > > >
> > > > > 3) Between the clients, brokers, streams and connect, Kafka has quite
> > > > > a 

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Federico,

- The metricName() method does not register anything, it just builds a
MetricName instance which is just a container holding a name, group,
description and tags for a metric. Each time it is called, it returns
a new instance. If called with the same arguments, the returned value
will be equal.
- Initially I just copied the API of Metrics. I made some small
changes so the metric and sensor methods are a bit more similar
- Good catch! I fixed the example.

Thanks,
Mickael


On Thu, Jan 26, 2023 at 3:54 PM Mickael Maison  wrote:
>
> Hi Chris,
>
> 1) I updated the KIP to only mention the interface.
> 2) This was a mistake. I've added ReplicationPolicy to the list of plugins.
>
> Thanks,
> Mickael
>
> On Tue, Jan 24, 2023 at 11:16 AM Yash Mayya  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the updated KIP, this is looking really good! I had a couple
> > more questions -
> >
> > 1) Sensor names need to be unique across all groups for a `Metrics`
> > instance. How are we planning to avoid naming clashes (both between
> > different plugins as well as with pre-defined sensors)?
> >
> > 2) Connect has a `ConnectMetrics` wrapper around `Metrics` via which
> > rebalance / worker / connector / task metrics are recorded. Could you
> > please elaborate in the KIP how the plugin metrics for connectors / tasks
> > will inter-operate with this?
> >
> > Another minor point is that the third rejected alternative appears to be an
> > incomplete sentence?
> >
> > Thanks,
> > Yash
> >
> > On Fri, Jan 13, 2023 at 10:56 PM Mickael Maison 
> > wrote:
> >
> > > Hi,
> > >
> > > I've updated the KIP based on the feedback.
> > >
> > > Now instead of receiving a Metrics instance, plugins get access to
> > > PluginMetrics that exposes a much smaller API. I've removed the
> > > special handling for connectors and tasks and they must now implement
> > > the Monitorable interface as well to use this feature. Finally the
> > > goal is to allow all plugins (apart from metrics reporters) to use
> > > this feature. I've listed them all (there are over 30 pluggable APIs)
> > > but I've not added the list in the KIP. The reason is that new plugins
> > > could be added in the future and instead I'll focus on adding support
> > > in all the place that instantiate classes.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Jan 10, 2023 at 7:00 PM Mickael Maison 
> > > wrote:
> > > >
> > > > Hi Chris/Yash,
> > > >
> > > > Thanks for taking a look and providing feedback.
> > > >
> > > > 1) Yes you're right, when using incompatible version, metrics() would
> > > > trigger NoSuchMethodError. I thought using the context to pass the
> > > > Metrics object would be more idiomatic for Connect but maybe
> > > > implementing Monitorable would be simpler. It would also allow other
> > > > Connect plugins (transformations, converters, etc) to register
> > > > metrics. So I'll make that change.
> > > >
> > > > 2) As mentioned in the rejected alternatives, I considered having a
> > > > PluginMetrics class/interface with a limited API. But since Metrics is
> > > > part of the public API, I thought it would be simpler to reuse it.
> > > > That said you bring interesting points so I took another look today.
> > > > It's true that the Metrics API is pretty complex and most methods are
> > > > useless for plugin authors. I'd expect most use cases only need one
> > > > addMetric and one sensor methods. Rather than subclassing Metrics, I
> > > > think a delegate/forwarding pattern might work well here. A
> > > > PluginMetric class would forward its method to the Metrics instance
> > > > and could perform some basic validations such as only letting plugins
> > > > delete metrics they created, or automatically injecting tags with the
> > > > class name for example.
> > > >
> > > > 3) Between the clients, brokers, streams and connect, Kafka has quite
> > > > a lot! In practice I think registering metrics should be beneficial
> > > > for all plugins, I think the only exception would be metrics reporters
> > > > (which are instantiated before the Metrics object). I'll try to build
> > > > a list of all plugin types and add that to the KIP.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > > On Tue, Dec 27, 2022 at 4:54 PM Chris Egerton 
> > > wrote:
> > > > >
> > > > > Hi Yash,
> > > > >
> > > > > Yes, a default no-op is exactly what I had in mind should the
> > > Connector and
> > > > > Task classes implement the Monitorable interface.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Dec 20, 2022 at 2:46 AM Yash Mayya 
> > > wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Thanks for creating this KIP, this will be a super useful feature to
> > > > > > enhance existing connectors in the Kafka Connect ecosystem.
> > > > > >
> > > > > > I have some similar concerns to the ones that Chris has outlined
> > > above,
> > > > > > especially with regard to directly exposing Connect's Metrics 

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Mickael Maison]

Hi Chris,

1) I updated the KIP to only mention the interface.
2) This was a mistake. I've added ReplicationPolicy to the list of plugins.

Thanks,
Mickael

On Tue, Jan 24, 2023 at 11:16 AM Yash Mayya  wrote:
>
> Hi Mickael,
>
> Thanks for the updated KIP, this is looking really good! I had a couple
> more questions -
>
> 1) Sensor names need to be unique across all groups for a `Metrics`
> instance. How are we planning to avoid naming clashes (both between
> different plugins as well as with pre-defined sensors)?
>
> 2) Connect has a `ConnectMetrics` wrapper around `Metrics` via which
> rebalance / worker / connector / task metrics are recorded. Could you
> please elaborate in the KIP how the plugin metrics for connectors / tasks
> will inter-operate with this?
>
> Another minor point is that the third rejected alternative appears to be an
> incomplete sentence?
>
> Thanks,
> Yash
>
> On Fri, Jan 13, 2023 at 10:56 PM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I've updated the KIP based on the feedback.
> >
> > Now instead of receiving a Metrics instance, plugins get access to
> > PluginMetrics that exposes a much smaller API. I've removed the
> > special handling for connectors and tasks and they must now implement
> > the Monitorable interface as well to use this feature. Finally the
> > goal is to allow all plugins (apart from metrics reporters) to use
> > this feature. I've listed them all (there are over 30 pluggable APIs)
> > but I've not added the list in the KIP. The reason is that new plugins
> > could be added in the future and instead I'll focus on adding support
> > in all the place that instantiate classes.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Jan 10, 2023 at 7:00 PM Mickael Maison 
> > wrote:
> > >
> > > Hi Chris/Yash,
> > >
> > > Thanks for taking a look and providing feedback.
> > >
> > > 1) Yes you're right, when using incompatible version, metrics() would
> > > trigger NoSuchMethodError. I thought using the context to pass the
> > > Metrics object would be more idiomatic for Connect but maybe
> > > implementing Monitorable would be simpler. It would also allow other
> > > Connect plugins (transformations, converters, etc) to register
> > > metrics. So I'll make that change.
> > >
> > > 2) As mentioned in the rejected alternatives, I considered having a
> > > PluginMetrics class/interface with a limited API. But since Metrics is
> > > part of the public API, I thought it would be simpler to reuse it.
> > > That said you bring interesting points so I took another look today.
> > > It's true that the Metrics API is pretty complex and most methods are
> > > useless for plugin authors. I'd expect most use cases only need one
> > > addMetric and one sensor methods. Rather than subclassing Metrics, I
> > > think a delegate/forwarding pattern might work well here. A
> > > PluginMetric class would forward its method to the Metrics instance
> > > and could perform some basic validations such as only letting plugins
> > > delete metrics they created, or automatically injecting tags with the
> > > class name for example.
> > >
> > > 3) Between the clients, brokers, streams and connect, Kafka has quite
> > > a lot! In practice I think registering metrics should be beneficial
> > > for all plugins, I think the only exception would be metrics reporters
> > > (which are instantiated before the Metrics object). I'll try to build
> > > a list of all plugin types and add that to the KIP.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > > On Tue, Dec 27, 2022 at 4:54 PM Chris Egerton 
> > wrote:
> > > >
> > > > Hi Yash,
> > > >
> > > > Yes, a default no-op is exactly what I had in mind should the
> > Connector and
> > > > Task classes implement the Monitorable interface.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Dec 20, 2022 at 2:46 AM Yash Mayya 
> > wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for creating this KIP, this will be a super useful feature to
> > > > > enhance existing connectors in the Kafka Connect ecosystem.
> > > > >
> > > > > I have some similar concerns to the ones that Chris has outlined
> > above,
> > > > > especially with regard to directly exposing Connect's Metrics object
> > to
> > > > > plugins. I believe it would be a lot friendlier to developers if we
> > instead
> > > > > exposed wrapper methods in the context classes - such as one for
> > > > > registering a new metric, one for recording metric values and so on.
> > This
> > > > > would also have the added benefit of minimizing the surface area for
> > > > > potential misuse by custom plugins.
> > > > >
> > > > > > for connectors and tasks they should handle the
> > > > > > metrics() method returning null when deployed on
> > > > > > an older runtime.
> > > > >
> > > > > I believe this won't be the case, and instead they'll need to handle
> > a
> > > > > `NoSuchMethodError` right? This is similar to previous KIPs that
> > added
> > > > > methods to connector context 

[VOTE] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Gantigmaa Selenge]

Hi

I'd like to call for a vote on KIP-894, which updates MirrorMaker to use
IncrementalAlterConfigs API to sync topic configurations between clusters.

The KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations

The discussion thread:
https://lists.apache.org/thread/4chr4s5vbd9rqhml2tk60fsghojwo6bb

Thank you
Tina

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #70

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Philip and all,

> if you don't use the client for a long time, why can't you just close the
client and re-instantiate a new one when needed? I'm not familiar with the
stream thread, so I don't know if that's possible.

Yes, it's always possible to recreate the client (I think, it's the main
workaround for this issue).
Streams don't do this currently. If we do the fix on the Streams level
(catch and recreate the client), it solves the problem there, but it
remains unsolved fundamentally and it will be still necessary to do the
same fix in other applications.

> Another idea here is, would it make sense to expose a
maybeUpdateMetadata() API to serve such a purpose?

Could you please elaborate, what exactly this function should do?

Best,
Ivan

On Tue, 24 Jan 2023 at 00:33, Philip Nee  wrote:

> Hey Ivan,
>
> Thanks for the KIP. Some questions for clarification: It seems like the
> main problem is that if we don't poll frequently enough, the cluster
> topology can change entirely before the metadata is refreshed and thus
> causing staled clients. My question is: if you don't use the client for a
> long time, why can't you just close the client and re-instantiate a new one
> when needed? I'm not familiar with the stream thread, so I don't know if
> that's possible.  Another idea here is, would it make sense to expose a
> maybeUpdateMetadata() API to serve such a purpose?
>
> Thanks,
> P
>
>
> On Wed, Jan 18, 2023 at 4:07 AM Ivan Yurchenko 
> wrote:
>
> > Hello!
> > I would like to start the discussion thread on KIP-899: Allow clients to
> > rebootstrap.
> > This KIP proposes to allow Kafka clients to repeat the bootstrap process
> > when fetching metadata if none of the known nodes are available.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap
> >
> > A question right away: should we eventually change the default behavior
> or
> > it can remain configurable "forever"? The latter is proposed in the KIP.
> >
> > Thank you!
> >
> > Ivan
> >
>

[jira] [Created] (KAFKA-14655) Case-insensitive header key operations in Java client

[Romain Quinio (Jira)]

Romain Quinio created KAFKA-14655:
-

 Summary: Case-insensitive header key operations in Java client
 Key: KAFKA-14655
 URL: https://issues.apache.org/jira/browse/KAFKA-14655
 Project: Kafka
  Issue Type: New Feature
Reporter: Romain Quinio


In the Java client, Kafka header keys are manipulated as String. The current 
implementation uses case-sensitive comparisons via #equals (rather than 
#equalsIgnoreCase): 
[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/header/internals/RecordHeaders.java#L89]

It would be useful to be able to ignore the case during header operations 
(get/delete/...) similarly to how HTTP header keys are manipulated by clients.

In particular in case of bridging HTTP -> Kafka, Kafka headers may be conveyed 
first as HTTP headers where the case of the key doesn't matter, the difference 
of behavior can be surprising.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)