Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2709

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #105

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2708

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-16325) Add missing producer metrics to documentation

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16325.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Add missing producer metrics to documentation
> -
>
> Key: KAFKA-16325
> URL: https://issues.apache.org/jira/browse/KAFKA-16325
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation, website
>Reporter: Divij Vaidya
>Assignee: Owen C.H. Leung
>Priority: Minor
>  Labels: newbie
> Fix For: 3.8.0
>
>
> Some producer metrics such as buffer-exhausted-rate [1]are missing from the 
> documentation at 
> [https://kafka.apache.org/documentation.html#producer_monitoring] 
> Hence, users of Kafka sometimes don't know about these metrics at all.
> This task will add these (and possibly any other missing) metrics to the 
> documentation. An example of a similar PR where metrics were added to the 
> documentation is at [https://github.com/apache/kafka/pull/12934] 
> [1] 
> [https://github.com/apache/kafka/blob/c254b22a4877e70617b2710b95ef44b8cc55ce97/clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java#L91]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15576) Add 3.6.0 to broker/client and streams upgrade/compatibility tests

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15576.
-
Resolution: Fixed

> Add 3.6.0 to broker/client and streams upgrade/compatibility tests
> --
>
> Key: KAFKA-15576
> URL: https://issues.apache.org/jira/browse/KAFKA-15576
> Project: Kafka
>  Issue Type: Task
>Reporter: Satish Duggana
>Priority: Major
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16354) FinalizedFeatureChangeListenerTest should use mocked latches

[Greg Harris (Jira)]

Greg Harris created KAFKA-16354:
---

 Summary: FinalizedFeatureChangeListenerTest should use mocked 
latches
 Key: KAFKA-16354
 URL: https://issues.apache.org/jira/browse/KAFKA-16354
 Project: Kafka
  Issue Type: Test
  Components: core
Affects Versions: 3.7.0
Reporter: Greg Harris


testCacheUpdateWaitFailsForUnreachableVersion takes 30 seconds, and 
testInitFailureDueToFeatureIncompatibility takes 5 seconds. This appears to be 
caused by FeatureCacheUpdater#awaitUpdateOrThrow waiting for a real 
CountDownLatch with a real-time timeout.

Instead, a mocked latch should be used to exit the await immediately.

This should be done both for CPU-independence, and for test execution speed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16202) Extra dot in error message in producer

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16202.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Extra dot in error message in producer
> --
>
> Key: KAFKA-16202
> URL: https://issues.apache.org/jira/browse/KAFKA-16202
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Reporter: Mickael Maison
>Priority: Trivial
> Fix For: 3.8.0
>
>
> If the broker hits a StorageException while handling a record from the 
> producer, the producer prints the following warning:
> [2024-01-29 15:33:30,722] WARN [Producer clientId=console-producer] Received 
> invalid metadata error in produce request on partition topic1-0 due to 
> org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying 
> to access log file on the disk.. Going to request metadata update now 
> (org.apache.kafka.clients.producer.internals.Sender)
> There's an extra dot between disk and Going.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2707

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Hi Colin,

Thanks for bringing up these points. I believe Jose suggested failing the
command if some features upgrade on the downgrade call and vice versa. This
could address some of the concerns you bring up here. I agree we should not
have an rpc to do both.

As for setting metadata only, we do have the option of using the --feature
flag to set only metadata version, but that requires writing it by short
rather than by metadata string. Not sure if that's what folks had in mind.

Including a command to list the features for a given version is pretty
useful. I can add that to the kip unless folks have concerns.

Justine

On Thu, Mar 7, 2024 at 1:28 PM Colin McCabe  wrote:

> Hi Jun and Justine,
>
> My understanding is that in the KIP, --metadata does not do the same thing
> as --release-version. The --metadata flag sets the metadata version only,
> and does not change any other feature. So neither flag replaces the other.
>
> In general, I'm not sure that this functionality belongs in the features
> command. It will have a lot of confusing interactions with the other flags.
> If we do want it in the features command it should be its own subcommand,
> not upgrade or downgrade. Because setting all the features could involve
> upgrading some, but downgrading others. So really what we're doing is
> neither an upgrade nor a downgrade, but a sync to a release version.
>
> This also raises some questions about the RPC. If we want to use the
> existing features RPC, we cannot perform a sync that does both upgrades and
> downgrades in a single RPC. So we'd either need to be non-atomic, or just
> refuse to do it in that case. Or add a new server-side RPC for this.
>
> Perhaps it would be more useful to just have a command that tells people
> what the levels of each feature would be for a given release version? Then
> the user could choose which ones to upgrade / downgrade (if any) and do
> them carefully one at a time. The shotgun approach, where you upgrade
> everything at once, actually isn't what I think most users want to do in
> production.
>
> best,
> Colin
>
>
> On Fri, Mar 1, 2024, at 13:25, Jun Rao wrote:
> > Hi, Justine,
> >
> > Thanks for the reply.
> >
> > 11. Yes, we can still have the option to set individual features. I was
> > suggesting that if one uses the tool without either --release-version
> > or --features, it will just use the latest version of every feature that
> it
> > knows. This is what's proposed in the original KIP-584.
> >
> > 12. If we can't deprecate --metadata METADATA, it would be useful to
> > describe the reason.
> >
> > Jun
> >
> > On Thu, Feb 29, 2024 at 3:39 PM Justine Olshan
> 
> > wrote:
> >
> >> Hey folks,
> >>
> >> Thanks for the discussion. Let me try to cover everyone's comments.
> >>
> >> Artem --
> >> I can add the examples you mentioned. As for naming, right now the
> feature
> >> is called "transaction version" or "TV". Are you suggesting it should be
> >> called "transaction protocol version" or "TPV"? I don't mind that, but
> just
> >> wanted to clarify if we want to include protocol or if simply
> "transaction
> >> version" is enough.
> >>
> >> Jun --
> >>
> >> 10.  *With **more features, would each of those be controlled by a
> separate
> >> feature or*
> >>
> >> *multiple features. For example, is the new transaction record format*
> >>
> >> *controlled only by MV with TV having a dependency on MV or is it
> >> controlled*
> >>
> >> *by both MV and TV.*
> >>
> >>
> >> I think this will need to be decided on a case by case basis. There
> should
> >> be a mechanism to set dependencies among features.
> >> For transaction version specifically, I have no metadata version
> >> dependencies besides requiring 3.3 to write the feature records and use
> the
> >> feature tools. I would suspect all new features would have this
> >> requirement.
> >>
> >>
> >> 11. *Basically, if **--release-version is not used, the command will
> just
> >> use the latest*
> >>
> >> *production version of every feature. Should we apply that logic to
> both*
> >>
> >> *tools?*
> >>
> >>
> >> How would this work with the upgrade tool? I think we want a way to set
> a
> >> new feature version for one feature and not touch any of the others.
> >>
> >>
> >> *12. Should we remove --metadata METADATA from kafka-features? It does
> the*
> >>
> >> *same thing as --release-version.*
> >>
> >>
> >> When I previously discussed with Colin McCabe offline about this tool,
> he
> >> was strongly against deprecation or changing flags. I personally think
> it
> >> could good
> >>
> >> to unify and not support a ton of flags, but I would want to make sure
> he
> >> is aligned.
> >>
> >>
> >> *13. KIP-853 also extends the tools to support a new feature
> >> kraft.version.*
> >>
> >> *It would be useful to have alignment between that KIP and this one.*
> >>
> >>
> >> Sounds good. Looks like Jose is in on the discussion so we can continue
> >> here. :)
> >>
> >>
> >>
> >> Jose --
> >>
> >>
> >> *1. KIP-853 uses 

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Colin McCabe]

Hi Jun and Justine,

My understanding is that in the KIP, --metadata does not do the same thing as 
--release-version. The --metadata flag sets the metadata version only, and does 
not change any other feature. So neither flag replaces the other.

In general, I'm not sure that this functionality belongs in the features 
command. It will have a lot of confusing interactions with the other flags. If 
we do want it in the features command it should be its own subcommand, not 
upgrade or downgrade. Because setting all the features could involve upgrading 
some, but downgrading others. So really what we're doing is neither an upgrade 
nor a downgrade, but a sync to a release version.

This also raises some questions about the RPC. If we want to use the existing 
features RPC, we cannot perform a sync that does both upgrades and downgrades 
in a single RPC. So we'd either need to be non-atomic, or just refuse to do it 
in that case. Or add a new server-side RPC for this.

Perhaps it would be more useful to just have a command that tells people what 
the levels of each feature would be for a given release version? Then the user 
could choose which ones to upgrade / downgrade (if any) and do them carefully 
one at a time. The shotgun approach, where you upgrade everything at once, 
actually isn't what I think most users want to do in production.

best,
Colin


On Fri, Mar 1, 2024, at 13:25, Jun Rao wrote:
> Hi, Justine,
>
> Thanks for the reply.
>
> 11. Yes, we can still have the option to set individual features. I was
> suggesting that if one uses the tool without either --release-version
> or --features, it will just use the latest version of every feature that it
> knows. This is what's proposed in the original KIP-584.
>
> 12. If we can't deprecate --metadata METADATA, it would be useful to
> describe the reason.
>
> Jun
>
> On Thu, Feb 29, 2024 at 3:39 PM Justine Olshan 
> wrote:
>
>> Hey folks,
>>
>> Thanks for the discussion. Let me try to cover everyone's comments.
>>
>> Artem --
>> I can add the examples you mentioned. As for naming, right now the feature
>> is called "transaction version" or "TV". Are you suggesting it should be
>> called "transaction protocol version" or "TPV"? I don't mind that, but just
>> wanted to clarify if we want to include protocol or if simply "transaction
>> version" is enough.
>>
>> Jun --
>>
>> 10.  *With **more features, would each of those be controlled by a separate
>> feature or*
>>
>> *multiple features. For example, is the new transaction record format*
>>
>> *controlled only by MV with TV having a dependency on MV or is it
>> controlled*
>>
>> *by both MV and TV.*
>>
>>
>> I think this will need to be decided on a case by case basis. There should
>> be a mechanism to set dependencies among features.
>> For transaction version specifically, I have no metadata version
>> dependencies besides requiring 3.3 to write the feature records and use the
>> feature tools. I would suspect all new features would have this
>> requirement.
>>
>>
>> 11. *Basically, if **--release-version is not used, the command will just
>> use the latest*
>>
>> *production version of every feature. Should we apply that logic to both*
>>
>> *tools?*
>>
>>
>> How would this work with the upgrade tool? I think we want a way to set a
>> new feature version for one feature and not touch any of the others.
>>
>>
>> *12. Should we remove --metadata METADATA from kafka-features? It does the*
>>
>> *same thing as --release-version.*
>>
>>
>> When I previously discussed with Colin McCabe offline about this tool, he
>> was strongly against deprecation or changing flags. I personally think it
>> could good
>>
>> to unify and not support a ton of flags, but I would want to make sure he
>> is aligned.
>>
>>
>> *13. KIP-853 also extends the tools to support a new feature
>> kraft.version.*
>>
>> *It would be useful to have alignment between that KIP and this one.*
>>
>>
>> Sounds good. Looks like Jose is in on the discussion so we can continue
>> here. :)
>>
>>
>>
>> Jose --
>>
>>
>> *1. KIP-853 uses --feature for kafka-storage instead of --features.*
>>
>> *This is consistent with the use of --feature in the "kafka-feature.sh*
>>
>> *upgrade" command.*
>>
>>
>> I wanted to include multiple features in one command, so it seems like
>> features is a better name. I discuss more below about why I think we should
>> allow setting multiple features at once.
>>
>>
>> *2. I find it a bit inconsistent that --feature and --release-version*
>>
>> *are mutually exclusive in the kafka-feature CLI but not in the*
>>
>> *kafka-storage CLI. What is the reason for this decision?*
>>
>>
>> For the storage tool, we are setting all the features for the cluster. By
>> default, all are set. For the upgrade tool, the default is to set one
>> feature. In the storage tool, it is natural for the --release-version to
>> set the remaining features that --features didn't cover since otherwise we
>> would need to set them all
>>
>> If we 

[jira] [Created] (KAFKA-16353) Offline protocol migration integration tests

[Dongnuo Lyu (Jira)]

Dongnuo Lyu created KAFKA-16353:
---

 Summary: Offline protocol migration integration tests
 Key: KAFKA-16353
 URL: https://issues.apache.org/jira/browse/KAFKA-16353
 Project: Kafka
  Issue Type: Sub-task
Reporter: Dongnuo Lyu
Assignee: Dongnuo Lyu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16299) Classic group error responses should contain a no generation id and an empty members list

[Jeff Kim (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Kim resolved KAFKA-16299.
--
Resolution: Invalid

Not needed. Refer to 
https://github.com/apache/kafka/pull/15446#discussion_r1509452557

> Classic group error responses should contain a no generation id and an empty 
> members list
> -
>
> Key: KAFKA-16299
> URL: https://issues.apache.org/jira/browse/KAFKA-16299
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
>
> In the new coordinator, the classic group response handling is not consistent 
> compared to the old coordinator.
>  
> The old coordinator responds with a NoGeneration id (-1) and an empty members 
> list whereas the new coordinator responds with an empty generation id (0) and 
> a null members list.
>  
> We should have the new coordinator respond with the same values. Similarly, 
> SyncGroupResponseData#assignment is also populated incorrectly in the case of 
> errors



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16265) Add durationFilter to ListTransactionsRequest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16265.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Add durationFilter to ListTransactionsRequest
> -
>
> Key: KAFKA-16265
> URL: https://issues.apache.org/jira/browse/KAFKA-16265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Yang Yu
>Assignee: Yang Yu
>Priority: Major
> Fix For: 3.8.0
>
>
> * Add durationFilter field to ListTransactionsRequest, make corresponding 
> server side changes
>  * Make appropriate version bumps for ListTransactionsRequest and 
> ListTransationsResponse
>  * Add durationFilter option to kafka-transactions.sh --list



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16352) Transaction may get get stuck in PrepareCommit or PrepareAbort state

[Artem Livshits (Jira)]

Artem Livshits created KAFKA-16352:
--

 Summary: Transaction may get get stuck in PrepareCommit or 
PrepareAbort state
 Key: KAFKA-16352
 URL: https://issues.apache.org/jira/browse/KAFKA-16352
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Artem Livshits
Assignee: Artem Livshits


A transaction took a long time to complete, trying to restart a producer would 
lead to CONCURRENT_TRANSACTION errors.  Investigation has shown that the 
transaction was stuck in PrepareCommit for a few days:

(current time when the investigation happened: Feb 27 2024), transaction state:

{{Type   |Name                  |Value}}
{{-}}
{{ref    |transactionalId       |xxx-yyy}}
{{long   |producerId            |299364}}
{{ref    |state                 |kafka.coordinator.transaction.PrepareCommit$ @ 
0x44fe22760}}
{{long   |txnStartTimestamp     |1708619624810  Thu Feb 22 2024 16:33:44.810 
GMT+}}
{{long   |txnLastUpdateTimestamp|1708619625335  Thu Feb 22 2024 16:33:45.335 
GMT+}}
{{-}}

The partition list was empty and transactionsWithPendingMarkers didn't contain 
the reference to the transactional state.  In the log there were the following 
relevant messages:

{{22 Feb 2024 @ 16:33:45.623 UTC [Transaction State Manager 1]: Completed 
loading transaction metadata from __transaction_state-3 for coordinator epoch 
611}}

(this is the partition that contains the transactional id).  After the data is 
loaded, it sends out markers and etc.

Then there is this message:

{{22 Feb 2024 @ 16:33:45.696 UTC [Transaction Marker Request Completion Handler 
4]: Transaction coordinator epoch for xxx-yyy has changed from 610 to 611; 
cancel sending transaction markers TxnMarkerEntry\{producerId=299364, 
producerEpoch=1005, coordinatorEpoch=610, result=COMMIT, partitions=[foo-bar]} 
to the brokers}}

this message is logged just before the state is removed 
transactionsWithPendingMarkers, but the state apparently contained the entry 
that was created by the load operation.  So the sequence of events probably 
looked like the following:
 # partition load completed
 # commit markers were sent for transactional id xxx-yyy; entry in 
transactionsWithPendingMarkers was created
 # zombie reply from the previous epoch completed, removed entry from 
transactionsWithPendingMarkers
 # commit markers properly completed, but couldn't transition to CommitComplete 
state because transactionsWithPendingMarkers didn't have the proper entry, so 
it got stuck there until the broker was restarted

Looking at the code there are a few cases that could lead to similar race 
conditions.  The fix it to keep track of the PendingCompleteTxn value that was 
used when sending the marker, so that we can only remove the state that was 
created when the marker was sent and not accidentally remove the state someone 
else created.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

[Matthias J. Sax]

Seems I am late to this party. Can we pick this up again aiming for 3.8 
release? I think it would be a great addition. Few comments:



- I think it does make sense to report `iterator-duration-avg` and 
`iterator-duration-max` for all *closed* iterators -- it just seems to 
be a useful metric (wondering if this would be _overall_ or bounded to 
some time window?)


- About the duration iterators are currently open, I believe the only 
useful way is to report the "oldest iterator", ie, the smallest iterator 
open-time, of all currently open-iterator? We all agree that in general, 
leaking iterator would bump the count metric, and if there is a few ones 
which are not closed and open for a long time, it seem sufficient to 
detect the single oldest one for alerting purpose?


- What I don't like about the KIP is it focus on RocksDB. I don't think 
we should build on the internal RocksDB counters as proposed (I guess, 
we could still expose them, similar to other RocksDB metrics which we 
expose already). However, for this new metric, we should track it 
ourselves and thus make it independent of RocksDB -- in the end, an 
in-memory store could also leak memory (and kill a JVM with an 
out-of-memory error) and we should be able to track it.


- Not sure if we would like to add support for custom stores, to allow 
them to register their iterators with this metric? Or would this not be 
necessary, because custom stores could just register a custom metric 
about it to begin with?




-Matthias

On 10/25/23 4:41 PM, Sophie Blee-Goldman wrote:


  If we used "iterator-duration-max", for
example, would it not be confusing that it includes Iterators that are
still open, and therefore the duration is not yet known?



1. Ah, I think I understand your concern better now -- I totally agree that
a
  "iterator-duration-max" metric would be confusing/misleading. I was
thinking about it a bit differently, something more akin to the
"last-rebalance-seconds-ago" consumer metric. As the name suggests,
that basically just tracks how long the consumer has gone without
rebalancing -- it doesn't purport to represent the actual duration between
rebalances, just the current time since the last one.  The hard part is
really
in choosing a name that reflects this -- maybe you have some better ideas
but off the top of my head, perhaps something like "iterator-lifetime-max"?

2. I'm not quite sure how to interpret the "iterator-duration-total" metric
-- what exactly does it mean to add up all the iterator durations? For
some context, while this is not a hard-and-fast rule, in general you'll
find that Kafka/Streams metrics tend to come in pairs of avg/max or
rate/total. Something that you might measure the avg for usually is
also useful to measure the max, whereas a total metric is probably
also useful as a rate but not so much as an avg. I actually think this
is part of why it feels like it makes so much sense to include a "max"
version of this metric, as Lucas suggested, even if the name of
"iterator-duration-max" feels misleading. Ultimately the metric names
are up to you, but for this reason, I would personally advocate for
just going with an "iterator-duration-avg" and "iterator-duration-max"

I did see your example in which you mention one could monitor the
rate of change of the "-total" metric. While this does make sense to
me, if the only way to interpret a metric is by computing another
function over it, then why not just make that computation the metric
and cut out the middle man? And in this case, to me at least, it feels
much easier to understand a metric like "iterator-duration-max" vs
something like "iterator-duration-total-rate"

3. By the way, can you add another column to the table with the new metrics
that lists the recording level? My suggestion would be to put the
"number-open-iterators" at INFO and the other two at DEBUG. See
the following for my reasoning behind this recommendation

4. I would change the "Type" entry for the "number-open-iterators" from
"Value" to "Gauge". This helps justify the "INFO" level for this metric,
since unlike the other metrics which are "Measurables", the current
timestamp won't need to be retrieved on each recording

5. Can you list the tags that would be associated with each of these
metrics (either in the table, or separately above/below if they will be
the same for all)

6. Do you have a strong preference for the name "number-open-iterators"
or would you be alright in shortening this to "num-open-iterators"? The
latter is more in line with the naming scheme used elsewhere in Kafka
for similar kinds of metrics, and a shorter name is always nice.

7. With respect to the rocksdb cache metrics, those sound useful but
if it was me, I would probably save them for a separate KIP mainly just
because the KIP freeze deadline is in a few weeks, and I wouldn't want
to end up blocking all the new metrics just because there was ongoing
debate about a subset of them. That said, you do have 3 full 

Re: [DISCUSS] KIP-1024: Make the restore behavior of GlobalKTables with custom processors configureable

[Matthias J. Sax]

@Bruno:

(1), I think you are spot for the ts-extractor: on the restore code 
path, we only support record-ts, but there is no need for a custom-ts 
because for regular changelog topics KS sets the ts, and thus, the 
optimization this KIP proposes required that the global topic follow the 
changelog format, ie, the ts must be in the record-ts.


However, for the regular processing path, I am not sure if we can omit 
deserializers. The way the PAPI is wired up, seems to require that we 
give proper types to _other_ Processor that read from the global state 
store. For this reason, the store (which takes `Serdes` with proper 
types) is wrapped with a `MeteredStore` (like all others) to do the 
Serde work, and this MeteredStore is also exposed to the 
global-Processor? Might be good for Walker to dig into this to find out 
the details?


If would of course be nice if we could avoid the unnecessary 
deserialization on topic read, and re-serialization on global-store put 
for this case, but it seems not to be straightforward to do...



(2). Is this about the PAPI/Topology? For this case, we don't have any 
config object across the board. We only do this in the DSL. Hence, I 
would propose to just follow the existing pattern in this KIP to keep 
the API consistent. For the DSL, it could make sense of course. -- Of 
course, if we think the PAPI could be improved with config objects, we 
could do this in a dedicate KIP.



@Lucas:

The PAPI is unfortunately (by design) much more open and less 
restrictive. If a users has a custom state store, we need some 
`Processor` code from them, because we cannot provide a built-in 
processor for an unknown store. The overload which won't take a 
processor would only work for the built-in key-value store, what I 
assume would cover most use-cases, however, we should keep the door open 
for other use cases. Otherwise, we disallow this optimization for custom 
stores. PAPI is really about flexibility, and yes, with great power 
comes great responsibility for the users :)


But this actually highlights a different aspect: the overload not 
accepting a custom `Processor` but using a built-in processor, should 
not accept a generic `StoreBuilder` but should restrict the type to 
`StoreBuilder`?



-Matthias



On 3/6/24 1:14 PM, Lucas Brutschy wrote:

Hey Walker

Thanks for the KIP, and congrats on the KiBiKIP ;)

My main point is that I'd vote against introducing
`reprocessOnRestore`. The behavior for `reprocessOnRestore = false` is
just incorrect and should be removed or deprecated. If we think we
need to keep the old behavior around, renaming the methods, e.g., to
`addGlobalReadOnlyStore`, is a great opportunity to deprecate the old
behavior. But at a first glance, the old behavior just looks like a
bug to me and should just be removed.

So for this KIP, I'd keep two variants as you propose and drop the
boolean parameter, but the two variants will be
  1) a copy-restore variant without custom processing, as you propose.
  2) a process-restore variant with custom processing (parameters the
same as before). This should be combined with a clear warning in the
Javadoc of the performance downside of this approach.

Presentation:
1) I wonder if you could make another pass on the motivation section.
I was lacking some context on this problem, and I think the nature of
the restore issue only became clear to me when I read through the
comments in the JIRA ticket you linked.
2) If we decide to keep the parameter `reprocessOnRestore`, the
Javadoc on it should be extended. This is a somewhat subtle issue, and
I don't think `restore by reprocessing` is enough of an explanation.

Nits:

`{@link ValueTransformer ValueTransformer}` -> `{@link
ValueTransformer ValueTransformers}`
`user defined` -> `user-defined`

Cheers,
Lucas

On Wed, Mar 6, 2024 at 9:55 AM Bruno Cadonna  wrote:


Hi Walker,

Thanks for the KIP!

Great that you are going to fix this long-standing issue!

1.
I was wondering if we need the timestamp extractor as well as the key
and value deserializer in Topology#addGlobalStore() that do not take a
ProcessorSupplier? What about Consumed in StreamsBuilder#addGlobalStore()?
Since those methods setup a global state store that does not process any
records, do they still need to deserialize records and extract
timestamps? Name might still be needed, right?

2.
  From an API point of view, it might make sense to put all
processor-related arguments into a parameter object. Something like:
GlobalStoreParameters.globalStore().withKeySerde(keySerde).disableReprocessOnRestore()
Just an idea, open for discussion.

3.
Could you please go over the KIP and correct typos and other mistakes in
the KIP?


Best,
Bruno



On 3/2/24 1:43 AM, Matthias J. Sax wrote:

Thanks for the KIP Walker.

Fixing this issue, and providing users some flexibility to opt-in/out on
"restore reprocessing" is certainly a good improvement.

  From an API design POV, I like the idea to not require passing in a

Re: KIP process

[Arpit Goyal]

Thanks Divij.
I will start creating KIP offline for now and share it with you when ready.

On Thu, Mar 7, 2024, 22:23 Divij Vaidya  wrote:

> cc: PMC
>
> Hey Arpit
>
> We are currently facing a problem with adding new users to the confluence
> wiki which Apache Kafka uses to maintain KIPs. We are working with Apache
> Infrastructure on a resolution -
> https://issues.apache.org/jira/browse/INFRA-25451
>
> If this is blocking you, we have two options. You can either create a KIP
> offline and share it in the discussion email or I can create the KIP page
> on your behalf and copy/paste the content that you send to me.
>
> --
> Divij Vaidya
>
>
>
> On Thu, Mar 7, 2024 at 5:15 PM Arpit Goyal 
> wrote:
>
>> + @Kamal Chandraprakash   @Vaidya, Divij
>>   Can you help me setting up the wiki id for the
>> confluence.
>> Thanks and Regards
>> Arpit Goyal
>> 8861094754
>>
>>
>> On Thu, Mar 7, 2024 at 9:27 PM Arpit Goyal 
>> wrote:
>>
>>> Hi Team,
>>> I want to start contributing to KIP but I am unable to login with the
>>> jira credentials.
>>> Can somebody help with the process?
>>> Jira userid is : goyarpit.
>>>
>>> [image: Screenshot 2024-03-07 at 9.27.03 PM.png]
>>>
>>>
>>> Thanks and Regards
>>> Arpit Goyal
>>> 8861094754
>>>
>>

Re: KIP process

[Divij Vaidya]

cc: PMC

Hey Arpit

We are currently facing a problem with adding new users to the confluence
wiki which Apache Kafka uses to maintain KIPs. We are working with Apache
Infrastructure on a resolution -
https://issues.apache.org/jira/browse/INFRA-25451

If this is blocking you, we have two options. You can either create a KIP
offline and share it in the discussion email or I can create the KIP page
on your behalf and copy/paste the content that you send to me.

--
Divij Vaidya



On Thu, Mar 7, 2024 at 5:15 PM Arpit Goyal  wrote:

> + @Kamal Chandraprakash   @Vaidya, Divij
>   Can you help me setting up the wiki id for the
> confluence.
> Thanks and Regards
> Arpit Goyal
> 8861094754
>
>
> On Thu, Mar 7, 2024 at 9:27 PM Arpit Goyal 
> wrote:
>
>> Hi Team,
>> I want to start contributing to KIP but I am unable to login with the
>> jira credentials.
>> Can somebody help with the process?
>> Jira userid is : goyarpit.
>>
>> [image: Screenshot 2024-03-07 at 9.27.03 PM.png]
>>
>>
>> Thanks and Regards
>> Arpit Goyal
>> 8861094754
>>
>

Re: KIP process

[Arpit Goyal]

+ @Kamal Chandraprakash   @Vaidya, Divij
  Can you help me setting up the wiki id for the
confluence.
Thanks and Regards
Arpit Goyal
8861094754


On Thu, Mar 7, 2024 at 9:27 PM Arpit Goyal  wrote:

> Hi Team,
> I want to start contributing to KIP but I am unable to login with the jira
> credentials.
> Can somebody help with the process?
> Jira userid is : goyarpit.
>
> [image: Screenshot 2024-03-07 at 9.27.03 PM.png]
>
>
> Thanks and Regards
> Arpit Goyal
> 8861094754
>

KIP process

[Arpit Goyal]

Hi Team,
I want to start contributing to KIP but I am unable to login with the jira
credentials.
Can somebody help with the process?
Jira userid is : goyarpit.

[image: Screenshot 2024-03-07 at 9.27.03 PM.png]


Thanks and Regards
Arpit Goyal
8861094754

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2706

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 453303 lines...]
[2024-03-07T15:22:23.259Z] > Task :server:compileTestJava
[2024-03-07T15:22:23.259Z] > Task :server:testClasses
[2024-03-07T15:22:24.360Z] > Task :server-common:compileTestJava
[2024-03-07T15:22:24.360Z] > Task :server-common:testClasses
[2024-03-07T15:22:26.547Z] > Task :raft:compileTestJava
[2024-03-07T15:22:26.547Z] > Task :raft:testClasses
[2024-03-07T15:22:28.629Z] 
[2024-03-07T15:22:28.629Z] > Task :clients:javadoc
[2024-03-07T15:22:28.629Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API
[2024-03-07T15:22:28.629Z] 
[2024-03-07T15:22:28.629Z]  This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
[2024-03-07T15:22:28.629Z]  The type field in both files must match and must 
not change. The type field
[2024-03-07T15:22:28.629Z]  is used both for passing ScramCredentialUpsertion 
and for the internal
[2024-03-07T15:22:28.629Z]  UserScramCredentialRecord. Do not change the type 
field."
[2024-03-07T15:22:29.729Z] 
[2024-03-07T15:22:29.729Z] > Task :core:compileScala
[2024-03-07T15:22:30.830Z] > Task :group-coordinator:compileTestJava
[2024-03-07T15:22:30.830Z] > Task :group-coordinator:testClasses
[2024-03-07T15:22:30.830Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2024-03-07T15:22:32.420Z] 
[2024-03-07T15:22:32.420Z] > Task :clients:javadoc
[2024-03-07T15:22:32.420Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2024-03-07T15:22:32.420Z] 2 warnings
[2024-03-07T15:22:32.420Z] 
[2024-03-07T15:22:32.420Z] > Task :clients:javadocJar
[2024-03-07T15:22:32.420Z] > Task :clients:srcJar
[2024-03-07T15:22:33.553Z] > Task :clients:testJar
[2024-03-07T15:22:33.553Z] > Task :clients:testSrcJar
[2024-03-07T15:22:33.553Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2024-03-07T15:22:33.553Z] > Task :clients:publishToMavenLocal
[2024-03-07T15:22:33.553Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2024-03-07T15:22:33.553Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2024-03-07T15:22:33.553Z] > Task :connect:api:testClasses UP-TO-DATE
[2024-03-07T15:22:33.553Z] > Task :connect:api:testJar
[2024-03-07T15:22:33.553Z] > Task :connect:api:testSrcJar
[2024-03-07T15:22:33.553Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2024-03-07T15:22:33.553Z] > Task :connect:api:publishToMavenLocal
[2024-03-07T15:22:34.768Z] > Task :metadata:compileTestJava
[2024-03-07T15:22:34.768Z] > Task :metadata:testClasses
[2024-03-07T15:22:39.414Z] > Task :streams:javadoc
[2024-03-07T15:22:39.414Z] > Task :streams:javadocJar
[2024-03-07T15:22:40.580Z] > Task :streams:srcJar
[2024-03-07T15:22:40.580Z] > Task :streams:processTestResources UP-TO-DATE
[2024-03-07T15:23:29.364Z] > Task :core:classes
[2024-03-07T15:23:29.364Z] > Task :core:compileTestJava NO-SOURCE
[2024-03-07T15:23:58.148Z] > Task :core:compileTestScala
[2024-03-07T15:25:05.862Z] > Task :core:testClasses
[2024-03-07T15:25:30.688Z] > Task :streams:compileTestJava
[2024-03-07T15:27:05.313Z] > Task :streams:testClasses
[2024-03-07T15:27:05.313Z] > Task :streams:testJar
[2024-03-07T15:27:05.313Z] > Task :streams:testSrcJar
[2024-03-07T15:27:05.313Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2024-03-07T15:27:05.313Z] > Task :streams:publishToMavenLocal
[2024-03-07T15:27:05.313Z] 
[2024-03-07T15:27:05.313Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2024-03-07T15:27:05.313Z] 
[2024-03-07T15:27:05.313Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2024-03-07T15:27:05.313Z] 
[2024-03-07T15:27:05.313Z] For more on this, please refer to 
https://docs.gradle.org/8.6/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
[2024-03-07T15:27:05.313Z] 
[2024-03-07T15:27:05.313Z] BUILD SUCCESSFUL in 5m 24s
[2024-03-07T15:27:05.313Z] 96 actionable tasks: 41 executed, 55 up-to-date
[2024-03-07T15:27:05.313Z] 
[2024-03-07T15:27:05.313Z] Publishing build scan...
[2024-03-07T15:27:05.313Z] https://ge.apache.org/s/n2jijyo6pijxi
[2024-03-07T15:27:05.313Z] 
[Pipeline] sh
[2024-03-07T15:27:08.876Z] + grep ^version= gradle.properties
[2024-03-07T15:27:08.876Z] + cut -d= -f 2
[Pipeline] dir
[2024-03-07T15:27:09.760Z] Running in 

[jira] [Created] (KAFKA-16351) add groupid in fetch request

[zou shengfu (Jira)]

zou shengfu created KAFKA-16351:
---

 Summary: add groupid in fetch request
 Key: KAFKA-16351
 URL: https://issues.apache.org/jira/browse/KAFKA-16351
 Project: Kafka
  Issue Type: Improvement
  Components: clients, core
Reporter: zou shengfu


Why we do not add groupid  field in fetch request? If  fetch request contains 
the group id, firstly, kafka broker can measure the traffic for every consumer 
that consume the same topic. And than kafka broker can limit the traffic for 
every consumer. Will we add the group id filed in fetch request in the future?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2705

[Apache Jenkins Server]

See