Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #110

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15490) Invalid path provided to the log failure channel upon I/O error when writing broker metadata checkpoint

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-15490.
---
Fix Version/s: 3.6.2
   Resolution: Fixed

> Invalid path provided to the log failure channel upon I/O error when writing 
> broker metadata checkpoint
> ---
>
> Key: KAFKA-15490
> URL: https://issues.apache.org/jira/browse/KAFKA-15490
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 3.4.0, 3.4.1, 3.5.1, 3.6.1
>Reporter: Alexandre Dupriez
>Assignee: Divij Vaidya
>Priority: Minor
> Fix For: 3.6.2
>
>
> There is a small bug/typo in the handling of I/O error when writing broker 
> metadata checkpoint in {{{}KafkaServer{}}}. The path provided to the log dir 
> failure channel is the full path of the checkpoint file whereas only the log 
> directory is expected 
> ([source|https://github.com/apache/kafka/blob/3.4/core/src/main/scala/kafka/server/KafkaServer.scala#L958C8-L961C8]).
> {code:java}
> case e: IOException =>
>val dirPath = checkpoint.file.getAbsolutePath
>logDirFailureChannel.maybeAddOfflineLogDir(dirPath, s"Error while writing 
> meta.properties to $dirPath", e){code}
> As a result, after an {{IOException}} is captured and enqueued in the log dir 
> failure channel ({{{}{}}} is to be replaced with the actual path of 
> the log directory):
> {code:java}
> [2023-09-22 17:07:32,052] ERROR Error while writing meta.properties to 
> /meta.properties (kafka.server.LogDirFailureChannel) 
> java.io.IOException{code}
> The log dir failure handler cannot lookup the log directory:
> {code:java}
> [2023-09-22 17:07:32,053] ERROR [LogDirFailureHandler]: Error due to 
> (kafka.server.ReplicaManager$LogDirFailureHandler) 
> org.apache.kafka.common.errors.LogDirNotFoundException: Log dir 
> /meta.properties is not found in the config.{code}
> An immediate fix for this is to use the {{logDir}} provided from to the 
> checkpointing method instead of the path of the metadata file.
> For brokers with only one log directory, this bug will result in preventing 
> the broker from shutting down as expected.
> The L{{{}ogDirNotFoundException{}}} then kills the log dir failure handler 
> thread, and subsequent {{IOException}} are not handled, and the broker never 
> stops.
> {code:java}
> [2024-02-27 02:13:13,564] INFO [LogDirFailureHandler]: Stopped 
> (kafka.server.ReplicaManager$LogDirFailureHandler){code}
> Another consideration here is whether the {{LogDirNotFoundException}} should 
> terminate the log dir failure handler thread.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2722

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Ismael Juma]

Hi team,

We should focus on regressions and cve fixes for releases like this one
(second bug fix and it's not the most recently released version). The idea
is to reduce the risk of regressions since it may well be the last official
release on this branch.

Ismael

On Thu, Mar 14, 2024, 1:39 AM Divij Vaidya  wrote:

> Hi Manikumar,
>
> 1. Can you please take a look at
> https://github.com/apache/kafka/pull/15490
> which is a bug fix specific to the 3.6.x branch?
> 2. Should we do a one-time update of all dependencies in 3.6.x branch
> before releasing 3.6.2?
> 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> backporting is needed to make the release qualification easier.
> 4. There are a large number of bugs reported as impacting 3.6.1 [1] Some of
> them have attached PRs and pending review. Maybe we can request all
> committers to take a look at the ones which have a PR attached and see if
> we can close them in the next few days before 3.6.2. Note that this will be
> on a best-effort basis and won't block release of 3.6.2.
> 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged whether
> something needs to be backported? Usually it is the responsibility of the
> reviewer but I have observed that sometimes we forget to backport important
> onces as well. I can help with this one early next week.
>
> [1]
>
> https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>
>
> --
> Divij Vaidya
>
>
>
> On Thu, Mar 14, 2024 at 7:55 AM Manikumar 
> wrote:
>
> > Hi all,
> >
> > Here is the release plan for 3.6.2:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> >
> > Currently there is one open non-blocker issue. I plan to generate the
> first
> > release candidate
> > once the issue is resolved and no other issues are raised in the
> meantime.
> >
> > Thanks,
> > Manikumar
> >
> > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana  >
> > wrote:
> >
> > > +1, Thanks Mani for volunteering.
> > >
> > > On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
> > > >
> > > > +1, Thanks Manikumar!
> > > >
> > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna 
> > > wrote:
> > > >
> > > > > Thanks Manikumar!
> > > > >
> > > > > +1
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > +1 thanks for volunteering!
> > > > > >
> > > > > > Best
> > > > > > ---
> > > > > >
> > > > > > Josep Prat
> > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > > +491715557497 | aiven.io
> > > > > > Aiven Deutschland GmbH
> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> +1
> > > > > >>
> > > > > >> Thank you for volunteering.
> > > > > >>
> > > > > >> --
> > > > > >> Divij Vaidya
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > > >> 
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Thanks Manikumar!
> > > > > >>> +1 from me
> > > > > >>>
> > > > > >>> Justine
> > > > > >>>
> > > > > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar <
> > > manikumar.re...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > >  Hi,
> > > > > 
> > > > >  I'd like to volunteer to be the release manager for a bug fix
> > > release
> > > > > >> of
> > > > >  the 3.6 line.
> > > > >  If there are no objections, I'll send out the release plan
> soon.
> > > > > 
> > > > >  Thanks,
> > > > >  Manikumar
> > > > > 
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>

[jira] [Created] (KAFKA-16378) Under tiered storage, deleting local logs does not free disk space

[Jianbin Chen (Jira)]

Jianbin Chen created KAFKA-16378:


 Summary: Under tiered storage, deleting local logs does not free 
disk space
 Key: KAFKA-16378
 URL: https://issues.apache.org/jira/browse/KAFKA-16378
 Project: Kafka
  Issue Type: Wish
Affects Versions: 3.7.0
Reporter: Jianbin Chen
 Attachments: image-2024-03-15-09-33-13-903.png

Of course, this is an occasional phenomenon, as long as the tiered storage 
topic triggered the deletion of the local log action, there is always the 
possibility of residual file references, but these files on the local disk is 
already impossible to find!

I use the implementation as: [Aiven-Open/tiered-storage-for-apache-kafka: 
RemoteStorageManager for Apache Kafka® Tiered Storage 
(github.com)|https://github.com/Aiven-Open/tiered-storage-for-apache-kafka]

I also filed an issue in their community, which also contains a full 
description of the problem

[Disk space not released · Issue #513 · 
Aiven-Open/tiered-storage-for-apache-kafka 
(github.com)|https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/issues/513]

!image-2024-03-15-09-33-13-903.png!

You can clearly see in this figure that the kafka log has already output the 
log of the operation that deleted the log, but the log is still referenced and 
the disk space has not been released



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

[Sophie Blee-Goldman]

>
> Should we change it do `.serializer` and `.deserialize`?

That's a good point -- if we're going to split this up by defining the
config
in both the TimeWindowedSerializer and TimeWindowedDeserializer,
then it makes perfect sense to go a step further and actually define
only the relevant de/serializer class instead of the full serde

Just to put this all together, it sounds like the proposal is to:

1) Deprecate both these configs where they appear in StreamsConfig
(as per the original plan in the KIP, just reiterating it here)

2) Don't "define" either config in any specific client's Config class,
but just define a String variable with the config name in the relevant
de/serializer class, and maybe point people to it in the docs somewhere

3) We would add three new public String variables for three different
configs across two classes, specifically:

In TimeWindowedSerializer:
  - define a constant for "windowed.inner.serializer.class"
In TimeWindowedDeserializer:
  - define a constant for "windowed.inner.deserializer.class"
  - define a constant for "window.size.ms"

4) Lastly, we would update the windowed de/serializer implementations
to check for the new configs (ie "windowed.inner.de/serializer.class")
and use the provided de/serializer class, if one was given. If the new
configs are not present, they would fall back to the original/current
logic (ie that based on the old "windowed.inner.serde.class" config)

I think that's everything. Does this sound about right for where we want
to go with these configs?

On Thu, Mar 14, 2024 at 4:58 PM Matthias J. Sax  wrote:

> >> By "don't add them" do you just mean we would not have any actual
> >> variables defined anywhere for these configs (eg WINDOW_SIZE_MS)
> >> and simply document -- somewhere -- that one can use the string
> >> "window.size.ms" when configuring a command-line client with a
> >> windowed serde?
>
> Yes. That's the idea.
>
>
>
> > I assume you aren't proposing
> >> to remove the ability to use and understand this config from the
> >> implementations themselves, but correct me if that's wrong.
>
> No, that would effectively break what we fixed with the original KIP :)
>
>
>
> >> Are there any other configs in similar situations that we could look
> >> to for precedent?
>
> Not aware of any others, either.
>
>
>
> >> If these are truly the first/only of their kind, I would vote to just
> stick
> >> them in the appropriate class. As for which class to put them in, I
> >> think I'm convinced that "window.size.ms" should only go in the
> >> TimeWindowedDeserializer rather than sticking them both in the
> >> TimeWindowedSerde as I originally suggested. However, I would
> >> even go a step further and not place the "inner.window.class.serde"
> >> in the TimeWindowedSerde class either. To me, it actually makes
> >> the most sense to define it in both the TimeWindowedSerializer
> >> and the TimeWindowedDeserializer.
>
> Not sure either. What you propose is fine with me. However, I am
> wondering about the config names... Why is it `serde` for this case?
> Should we change it do `.serializer` and `.deserialize`?
>
>
>
> -Matthias
>
>
> On 3/13/24 8:19 PM, Sophie Blee-Goldman wrote:
> > By "don't add them" do you just mean we would not have any actual
> > variables defined anywhere for these configs (eg WINDOW_SIZE_MS)
> > and simply document -- somewhere -- that one can use the string
> > "window.size.ms" when configuring a command-line client with a
> > windowed serde? Or something else? I assume you aren't proposing
> > to remove the ability to use and understand this config from the
> > implementations themselves, but correct me if that's wrong.
> >
> > Are there any other configs in similar situations that we could look
> > to for precedent? I personally am not aware of any but by definition
> > I suppose these would be hard to discover unless you were actively
> > looking for them, so I'm wondering if there might be other "shadow
> > configs" elsewhere in the code base.
> >
> > If these are truly the first/only of their kind, I would vote to just
> stick
> > them in the appropriate class. As for which class to put them in, I
> > think I'm convinced that "window.size.ms" should only go in the
> > TimeWindowedDeserializer rather than sticking them both in the
> > TimeWindowedSerde as I originally suggested. However, I would
> > even go a step further and not place the "inner.window.class.serde"
> > in the TimeWindowedSerde class either. To me, it actually makes
> > the most sense to define it in both the TimeWindowedSerializer
> > and the TimeWindowedDeserializer.
> >
> > The reason being that, as discussed above, the only use case for
> > these configs would be in the console consumer/producer which
> > only uses the Serializer or Deserializer, and would never actually
> > be used by/in Streams where we use the Serde version. And while
> > defining the  "inner.window.class.serde" in two places might seem
> > redundant, this would mean that all 

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

[Shashwat Pandey]

Thanks for the feedback Matthias!

The reason I proposed the extension of MockProcessorContext was more to do
with the internals of the class (MockRecordMetadata, CapturedPunctuator and
CapturedForward).

However, I do see your point, I would then think to split
MockProcessorContext and MockFixedKeyProcessorContext, some of the internal
classes should also be extracted i.e. MockRecordMetadata,
CapturedPunctuator and probably a new CapturedFixedKeyForward.

Let me know what you think!


Regards,
Shashwat Pandey


On Mon, Mar 11, 2024 at 10:09 PM Matthias J. Sax  wrote:

> Thanks for the KIP Shashwat. Closing this testing gap is great! It did
> come up a few time already...
>
> One question: why do you propose to `extend MockProcessorContext`?
>
> Given how the actual runtime context classes are setup, it seems that
> the regular context and fixed-key-context are distinct, and thus I
> believe both mock-context classes should be distinct, too?
>
> What I mean is that FixedKeyProcessorContext does not extend
> ProcessorContext. Both classes have a common parent ProcessINGContext
> (note the very similar but different names), but they are "siblings"
> only, so why make the mock processor a parent-child relationship?
>
> It seems better to do
>
> public class MockFixedKeyProcessorContext
>implements FixedKeyProcessorContext,
>   RecordCollector.Supplier
>
>
> Of course, if there is code we can share between both mock-context we
> should so this, but it should not leak into the public API?
>
>
> -Matthias
>
>
>
> On 3/11/24 5:21 PM, Shashwat Pandey wrote:
> > Hi everyone,
> >
> > I would like to start the discussion on
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1027%3A+Add+MockFixedKeyProcessorContext
> >
> > This adds MockFixedKeyProcessorContext to the Kafka Streams Test Utils
> > library.
> >
> > Regards,
> > Shashwat Pandey
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2721

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

[Matthias J. Sax]

By "don't add them" do you just mean we would not have any actual
variables defined anywhere for these configs (eg WINDOW_SIZE_MS)
and simply document -- somewhere -- that one can use the string
"window.size.ms" when configuring a command-line client with a
windowed serde? 


Yes. That's the idea.




I assume you aren't proposing

to remove the ability to use and understand this config from the
implementations themselves, but correct me if that's wrong.


No, that would effectively break what we fixed with the original KIP :)




Are there any other configs in similar situations that we could look
to for precedent?


Not aware of any others, either.




If these are truly the first/only of their kind, I would vote to just stick
them in the appropriate class. As for which class to put them in, I
think I'm convinced that "window.size.ms" should only go in the
TimeWindowedDeserializer rather than sticking them both in the
TimeWindowedSerde as I originally suggested. However, I would
even go a step further and not place the "inner.window.class.serde"
in the TimeWindowedSerde class either. To me, it actually makes
the most sense to define it in both the TimeWindowedSerializer
and the TimeWindowedDeserializer.


Not sure either. What you propose is fine with me. However, I am 
wondering about the config names... Why is it `serde` for this case? 
Should we change it do `.serializer` and `.deserialize`?




-Matthias


On 3/13/24 8:19 PM, Sophie Blee-Goldman wrote:

By "don't add them" do you just mean we would not have any actual
variables defined anywhere for these configs (eg WINDOW_SIZE_MS)
and simply document -- somewhere -- that one can use the string
"window.size.ms" when configuring a command-line client with a
windowed serde? Or something else? I assume you aren't proposing
to remove the ability to use and understand this config from the
implementations themselves, but correct me if that's wrong.

Are there any other configs in similar situations that we could look
to for precedent? I personally am not aware of any but by definition
I suppose these would be hard to discover unless you were actively
looking for them, so I'm wondering if there might be other "shadow
configs" elsewhere in the code base.

If these are truly the first/only of their kind, I would vote to just stick
them in the appropriate class. As for which class to put them in, I
think I'm convinced that "window.size.ms" should only go in the
TimeWindowedDeserializer rather than sticking them both in the
TimeWindowedSerde as I originally suggested. However, I would
even go a step further and not place the "inner.window.class.serde"
in the TimeWindowedSerde class either. To me, it actually makes
the most sense to define it in both the TimeWindowedSerializer
and the TimeWindowedDeserializer.

The reason being that, as discussed above, the only use case for
these configs would be in the console consumer/producer which
only uses the Serializer or Deserializer, and would never actually
be used by/in Streams where we use the Serde version. And while
defining the  "inner.window.class.serde" in two places might seem
redundant, this would mean that all the configs needed to properly
configure the specific class being used by the particular kind of
consumer client -- that is, Deserializer for a console consumer and
Serializer for a console producer -- would be located in that exact
class. I assume this would make them much easier to discover
and be used than having to search for configs defined in classes
you don't even need for the console client, like the Serde form

Just my two cents -- happy to hear other opinions on this

On Mon, Mar 11, 2024 at 6:58 PM Matthias J. Sax  wrote:


Yes, it's used inside `TimeWindowedSerializer` and actually also inside
`TimeWindowDeserializer`.

However, it does IMHO not change that we should remove it from
`StreamsConfig` because both configs are not intended to be used in Java
code... If one writes Java code, they should use

new TimeWindowedSerializer(Serializer)
new TimeWindowDeserializer(Deserializer, Long)
new TimeWindowSerdes(Serde, Long)

and thus they don't need either config.

The configs are only needed for command line tool, that create the
(de)serializer via reflection using the default constructor.

Does this make sense?



The only open question is really, if and where to add them... Strictly
speaking, we don't need either config as public variable as nobody
should use them in Java code. To me, it just feels right/better do make
them public for documentation purpose that these configs exists?

`inner.window.class.serde` has "serde" in it's name, so we could add it
to `TimeWindowSerdes`? For `window.size.ms`, it's only used by the
deserialize to maybe add it there? Just some ideas. -- Or we sidestep
this question and just don't add them; also fine with me.


-Matthias

On 3/11/24 10:53 AM, Lucia Cerchie wrote:

PS-- I was re-reading the PR that originated this discussion and realized
that 

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[Jun Rao]

Hi, Jose,

Thanks for the reply. A few more comments.

37. Have you updated the wiki? It seems that LeaderIdAndEpoch and
NodeEpoint are still two separate structs.

45. kafka-storage format --cluster-id  --release-version 3.8
--standalone --config controller.properties
It seems that --release-version is optional and the default value of this
initial metadata.version is statically defined by the binary used to run
the tool. So, in the common case, it seems that we could just omit
--release-version.

46. The KIP says "The leader will learn the range of supported version from
the UpdateVoter RPC.
46.1 Then "KRaft will use the information in the controller registration,
broker registration and add voter records to determine if the new version
is compatible." is no longer accurate.
46.2 Do we have a bit of a chicken-and-egg problem? A voter can't send
UpdateVoter RPC until kraft.version is upgraded to 1. But to upgrade
kraft.version, the leader needs to receive UpdateVoter RPCs.
46.3 If the leader always learns the range of supported kraft.version from
UpdateVoter RPC, does the VotersRecord need to include KRaftVersionFeature?

47. When a voter is started, in what order does it send the UpdateVoter and
ControllerRegistration RPC?

48. Voters set: "If the partition doesn't contain a VotersRecord control
record then the value stored in the controller.quorum.voters property will
be used."
  Hmm, I thought controller.quorum.voters is only the fallback
for controller.quorum.bootstrap.servers?

49. "If the local replica is getting added to the voters set, it will allow
the transition to prospective candidate when the fetch timer expires."
  Could you explain why this is needed?

50. Quorum state: AppliedOffset will get removed in version 1.
  This is the original description of AppliedOffset: Reflects the maximum
offset that has been applied to this quorum state. This is used for log
recovery. The broker must scan from this point on initialization to detect
updates to this file. If we remove this field, how do we reason about the
consistency between the quorum state and the metadata log?

51. AddVoter has the following steps.
1. Wait for the fetch offset of the replica (ID, UUID) to catch up to the
log end offset of the leader.
2. Wait until there are no uncommitted VotersRecord. Note that the
implementation may just return a REQUEST_TIMED_OUT error if there are
pending operations.
3. Wait for the LeaderChangeMessage control record from the current epoch
to get committed. Note that the implementation may just return a
REQUEST_TIMED_OUT error if there are pending operations.
4. Send an ApiVersions RPC to the first listener to discover the supported
kraft.version of the new voter.
It seems that doing the check on supported kraft.version of the new voter
in step 4 is too late. If the new voter doesn't support kraft.version of 1,
it can't process the metadata log records and step 1 could fail.

52. Admin client: Could you provide a bit more details on the changes?

53. A few more typos.
53.1 be bootstrap => be bootstrapped
53.2 If the new leader supports the current kraft.version,
  new leader => new voter
53.3 Voter are removed
  Voter => Voters
53.4 that are part that are voters
  This part doesn't read well.
53.5 the the current
  extra the
53.6 it will be discover
  discover => discovered
53.7 it would beneficial
  beneficial => be beneficial

Jun

On Mon, Mar 11, 2024 at 10:39 AM José Armando García Sancio
 wrote:

> Hi Jun
>
> Thanks for the feedback. See my comments below.
>
> On Wed, Mar 6, 2024 at 4:47 PM Jun Rao  wrote:
> > 20.1. It seems that the PreferredSuccessors field didn't get fixed. It's
> > still there together with PreferredCandidates.
> > +{ "name": "PreferredSuccessors", "type": "[]int32", "versions":
> > "0",
> > +  "about": "A sorted list of preferred successors to start the
> > election" },
> > +{ "name": "PreferredCandidates", "type": "[]ReplicaInfo",
> > "versions": "1+",
> > +  "about": "A sorted list of preferred candidates to start the
> > election", "fields": [
>
> Notice that the PreferredSuccessors field is only for version 0 while
> the PreferredCandidate field is for version 1 or greater. I had to
> create a new field because arrays of int32 ([]int32) are not
> compatible with arrays of structs because of tagged fields in
> sub-structs.
>
> > 37. If we don't support batching in AddVoterResponse, RemoveVoterResponse
> > and UpdateVoterResponse, should we combine CurrentLeader and NodeEndpoint
> > into a single field?
>
> Yes. I replaced the LeaderIdAndEpoch and NodeEpoint structs into a
> single struct that contains the leader id, epoch, host and port.
>
> > 42. We include VoterId and VoterUuid for the receiver in Vote and
> > BeginQuorumEpoch requests, but not in EndQuorumEpoch, Fetch and
> > FetchSnapshot. Could you explain how they are used?
>
> For the Vote request and BeginQuorumEpoch request the replica
> (candidate for Vote and leader for BeginQuoru

[jira] [Resolved] (KAFKA-14971) Flaky Test org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest#testSyncTopicConfigs

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-14971.

Resolution: Duplicate

KAFKA-15945 has filed PR so close this ticket

> Flaky Test 
> org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest#testSyncTopicConfigs
> --
>
> Key: KAFKA-14971
> URL: https://issues.apache.org/jira/browse/KAFKA-14971
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sagar Rao
>Priority: Major
>  Labels: flaky-test, mirror-maker
>
> The test testSyncTopicConfigs in 
> `org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest#testSyncTopicConfigs`
>  seems to be flaky. Found here : 
> [https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13594/6/tests]
> Ran on local against the [same PR  
> |https://github.com/apache/kafka/pull/13594]and  it passed.
>  
>  
> {code:java}
> org.opentest4j.AssertionFailedError: `delete.retention.ms` should be 2000, 
> because it's explicitly defined on the target topic! ==> expected: <2000> but 
> was: <8640>
> Stacktrace
> org.opentest4j.AssertionFailedError: `delete.retention.ms` should be 2000, 
> because it's explicitly defined on the target topic! ==> expected: <2000> but 
> was: <8640>
> at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
> at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
> at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1153)
> at 
> app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.lambda$testSyncTopicConfigs$8(MirrorConnectorsIntegrationBaseTest.java:758)
> at 
> app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:325)
> at 
> app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:373)
> at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:322)
> at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:306)
> at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:296)
> at 
> app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testSyncTopicConfigs(MirrorConnectorsIntegrationBaseTest.java:752)
> at 
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
> at 
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at 
> java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
> at 
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
> at 
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> at 
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:2

[jira] [Resolved] (KAFKA-15523) Flaky test org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testSyncTopicConfigs

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-15523.

Resolution: Duplicate

KAFKA-15945 has filed PR so close this ticket

> Flaky test  
> org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testSyncTopicConfigs
> ---
>
> Key: KAFKA-15523
> URL: https://issues.apache.org/jira/browse/KAFKA-15523
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 3.6.0, 3.5.1
>Reporter: Josep Prat
>Priority: Major
>  Labels: flaky, flaky-test
>
> Last seen: 
> [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14458/3/testReport/junit/org.apache.kafka.connect.mirror.integration/MirrorConnectorsIntegrationSSLTest/Build___JDK_17_and_Scala_2_13___testSyncTopicConfigs__/]
>  
> h3. Error Message
> {code:java}
> org.opentest4j.AssertionFailedError: Condition not met within timeout 3. 
> Topic: mm2-status.backup.internal didn't get created in the cluster ==> 
> expected:  but was: {code}
> h3. Stacktrace
> {code:java}
> org.opentest4j.AssertionFailedError: Condition not met within timeout 3. 
> Topic: mm2-status.backup.internal didn't get created in the cluster ==> 
> expected:  but was:  at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>  at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>  at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) at 
> app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36) at 
> app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210) at 
> app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)
>  at 
> app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
>  at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328) 
> at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312) 
> at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302) 
> at 
> app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.waitForTopicCreated(MirrorConnectorsIntegrationBaseTest.java:1041)
>  at 
> app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.startClusters(MirrorConnectorsIntegrationBaseTest.java:224)
>  at 
> app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.startClusters(MirrorConnectorsIntegrationBaseTest.java:149)
>  at 
> app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.startClusters(MirrorConnectorsIntegrationSSLTest.java:63)
>  at 
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method) at 
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>  at 
> java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568) at 
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
>  at 
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>  at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>  at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
>  at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:128)
>  at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeEachMethod(TimeoutExtension.java:78)
>  at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
>  at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
>  at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
>  at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>  at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
>  at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
>  at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoke

[jira] [Created] (KAFKA-16377) Fix flaky HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16377:
--

 Summary: Fix flaky 
HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores
 Key: KAFKA-16377
 URL: https://issues.apache.org/jira/browse/KAFKA-16377
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai


{quote}
[2024-03-13T16:07:11.125Z] Gradle Test Run :streams:test > Gradle Test Executor 
95 > HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(String, TestInfo) > 
"shouldScaleOutWithWarmupTasksAndPersistentStores(String, 
TestInfo).balance_subtopology" FAILED
[2024-03-13T16:07:11.125Z] java.lang.AssertionError: the first assignment 
after adding a node should be unstable while we warm up the state.
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.assertFalseNoRetry(HighAvailabilityTaskAssignorIntegrationTest.java:310)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.lambda$shouldScaleOutWithWarmupTasks$7(HighAvailabilityTaskAssignorIntegrationTest.java:237)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:395)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:443)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:392)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:366)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasks(HighAvailabilityTaskAssignorIntegrationTest.java:232)
[2024-03-13T16:07:11.125Z] at 
org.apache.kafka.streams.integration.HighAvailabilityTaskAssignorIntegrationTest.shouldScaleOutWithWarmupTasksAndPersistentStores(HighAvailabilityTaskAssignorIntegrationTest.java:130)
{quote}
https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-15474/runs/12/nodes/9/steps/88/log/?start=0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16376) Fix flaky ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16376:
--

 Summary: Fix flaky 
ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric
 Key: KAFKA-16376
 URL: https://issues.apache.org/jira/browse/KAFKA-16376
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai


{quote}
[2024-03-13T17:22:47.835Z] > Task :core:test
[2024-03-13T17:22:47.835Z] 
kafka.server.ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric() failed, 
log available in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-15474/core/build/reports/testOutput/kafka.server.ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric().test.stdout
[2024-03-13T17:22:47.835Z] 
[2024-03-13T17:22:49.409Z] Gradle Test Run :core:test > Gradle Test Executor 97 
> ReplicaManagerTest > testRemoteFetchExpiresPerSecMetric() FAILED
[2024-03-13T17:22:49.409Z] org.opentest4j.AssertionFailedError: The 
ExpiresPerSec value is not incremented. Current value is: 0
[2024-03-13T17:22:49.409Z] at 
org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
[2024-03-13T17:22:49.409Z] at 
org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
[2024-03-13T17:22:49.409Z] at 
kafka.server.ReplicaManagerTest.testRemoteFetchExpiresPerSecMetric(ReplicaManagerTest.scala:4174)
{quote}

https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-15474/runs/12/nodes/9/steps/88/log/?start=0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15691) Add new system tests to use new consumer

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True reopened KAFKA-15691:
---

> Add new system tests to use new consumer
> 
>
> Key: KAFKA-15691
> URL: https://issues.apache.org/jira/browse/KAFKA-15691
> Project: Kafka
>  Issue Type: Test
>  Components: clients, consumer, system tests
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
>  Labels: kip-848-client-support, system-tests
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

[Greg Harris]

Hi Arpit,

Thanks for the clarification. Replying here without updating the KIP
is fine for now.

I disagree with your evaluation of the backwards compatibility. If you
change the return type of a method, that breaks both source and binary
compatibility.
After upgrading, plugin implementations using this method would face
compilation errors. Implementations that were compiled against the old
interface will not be able to be loaded when the new interface is
present.
I see that the interface is marked Evolving which permits breaking
compatibility at minor releases, but that doesn't change the
compatibility of the change itself.

Thanks,
Greg

On Thu, Mar 14, 2024 at 8:55 AM Arpit Goyal  wrote:
>
> Hi Greg,
> I do not have access to update the KIP , Divij is helping me to do it.
> Meanwhile let me update your queries here.
>
> Backward compatibility:
> These changes will not impact the existing functionality as the existing
> behaviour always expects producer snapshot files to be present for a given
> segment. Making Producer Snapshot file optional helps to cover both the
> scenario i.e. both existing  and non existing of the producer snapshot file.
>
> The getter of producer snapshot file  would also be changed as described
> below:
>
> Current
>
> /**
>  * @return Producer snapshot file until this segment.
>  */
> public Path producerSnapshotIndex() {
> return producerSnapshotIndex;
> }
>
>
> Proposed
>
> /**
>  * @return Producer snapshot file until this segment.
>  */
> public Optional producerSnapshotIndex() {
> return producerSnapshotIndex;
> }
>
>
> Thanks and Regards
> Arpit Goyal
> 8861094754
>
>
> On Wed, Mar 13, 2024 at 9:25 PM Greg Harris 
> wrote:
>
> > Hi Arpit,
> >
> > Thanks for the KIP!
> >
> > I am not familiar with the necessity of producer snapshots, but your
> > explanation sounds like this should be made optional.
> >
> > Can you expand the KIP to include the changes that need to be made to
> > the constructor and getter, and explain more about backwards
> > compatibility? From the description I can't tell if this change is
> > backwards-compatible or not.
> >
> > Thanks,
> > Greg
> >
> > On Wed, Mar 13, 2024 at 6:48 AM Arpit Goyal 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I just wanted to bump up this thread.
> > >
> > > The KIP introduces a really small change  and it would not take much of
> > the
> > > time reviewing it.  This change would enable kafka users to use  tiered
> > > storage features seamlessly  for the topics migrated  from < 2.8 version
> > > which currently failed with NullPointerException.
> > >
> > > I am waiting for this KIP to get approved and then start working on it.
> > >
> > > On Mon, Mar 11, 2024, 14:26 Arpit Goyal 
> > wrote:
> > >
> > > > Hi All,
> > > > Just a Reminder, KIP-1026  is open for discussion.
> > > > Thanks and Regards
> > > > Arpit Goyal
> > > > 8861094754
> > > >
> > > >
> > > > On Sat, Mar 9, 2024 at 9:27 AM Arpit Goyal 
> > > > wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> I have created KIP-1026 for handling producerSnapshot empty scenarios
> > > >> when the topic is upgraded from the kafka  < 2.8 version.
> > > >>
> > > >>
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1026%3A+Handling+producer+snapshot+when+upgrading+from+%3C+v2.8.0+for+Tiered+Storage
> > > >>
> > > >> Feedback and suggestions are welcome.
> > > >>
> > > >> Thanks and Regards
> > > >> Arpit Goyal
> > > >> 8861094754
> > > >>
> > > >
> >

Re: [VOTE] KIP-981: Manage Connect topics with custom implementation of Admin

[Greg Harris]

Hi Omnia,

Thanks for the KIP.

I don't think we can adapt the ForwardingAdmin as-is for use as a
first-class Connect plugin.
1. It doesn't have a default constructor, and so can't be included in
the existing plugin discovery mechanisms.
2. It doesn't implement Versioned, and so won't have version
information exposed in the REST API

I also don't think that we should make the ForwardingAdmin a
second-class Connect plugin.
1. Having some plugins but not others benefit from classloader
isolation would be a "gotcha" for anyone familiar with existing
Connect plugins
2. Some future implementations may have a use-case for classloader
isolation (such as depending on their own HTTP/json library) and
retrofitting isolation would be more complicated than including it
initially.

I also have concerns about the complexity of the implementation as a
superclass instead of an interface, especially when considering the
evolution of the Admin interface.

I don't think the original proposal included the rejected alternative
of having the existing AdminClient talk to the federation layer, which
could implement a Kafka-compatible endpoint.
If a federation layer needs to intercept the Admin client behaviors,
It sounds more reasonable for that to be addressed for all Admin
clients at the network boundary rather than one-by-one updating the
Java APIs to use this new plugin.
This KIP appearing as a follow-up to KIP-787 is evidence that the
problem is more general than the proposed solution.

At this time I'm -1 for this proposal. I'm happy to discuss this more
in the DISCUSS thread.

Thanks,
Greg

On Thu, Mar 14, 2024 at 11:07 AM Mickael Maison
 wrote:
>
> Hi Omnia,
>
> +1 (binding), thanks for the KIP
>
> Mickael
>
> On Tue, Mar 5, 2024 at 10:46 AM Omnia Ibrahim  wrote:
> >
> > Hi everyone, I would like to start the vote on KIP-981: Manage Connect 
> > topics with custom implementation of Admin 
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-981%3A+Manage+Connect+topics+with+custom+implementation+of+Admin
> >
> > Thanks
> > Omnia

[jira] [Created] (KAFKA-16375) Fix logic for discarding reconciliation if member rejoined

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16375:
--

 Summary: Fix logic for discarding reconciliation if member rejoined
 Key: KAFKA-16375
 URL: https://issues.apache.org/jira/browse/KAFKA-16375
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Lianet Magrans
Assignee: Lianet Magrans


The current implementation of the new consumer discards the result of a 
reconciliation if the member rejoined, based on a comparison of the member 
epoch at the start and end of the reconciliation. If the epochs changed the 
reconciliation is discarded. This is not right because the member epoch could 
be incremented without an assignment change. This should be fixed to ensure 
that the reconciliation is discarded if the member rejoined, probably based on 
a flag that truly reflects that it went through a transition to joining. 
As a potential improvement, consider if the member could keep the 
reconciliation if it rejoined but got the same assignment.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Personal branches under apache/kafka

[Chia-Ping Tsai]

+1 to delete personal branches

Also +1 to Matthias to disable push if the branch is nonexistent

> committers for various
> reasons, bugfix, tests.

That is interesting. Can anyone share the details here? I’d like to know the 
bug/test which need to have a branch.

Chia-Ping


> Mickael Maison  於 2024年3月14日 凌晨2:02 寫道：
> 
> Hi,
> 
> We have accumulated a number of personal branches in the github
> repository: https://github.com/apache/kafka/branches/all
> 
> All these branches have been created by committers for various
> reasons, bugfix, tests.
> 
> I wonder if we should avoid creating branches in the apache repository
> (always use your own fork like regular contributors) and in the rare
> cases this is necessary ensure we delete them once done? This way we
> would only have branches for the various releases (3.7, 3.6, etc).
> 
> What do you think?
> 
> Thanks,
> Mickael

Re: [DISCUSS] Personal branches under apache/kafka

[Ismael Juma]

An alternative would be to have a convention for personal branches. For
example "tmp/ijuma/blah". I don't have a strong opinion, but it seems
useful if people want to collaborate on a branch.

Ismael

On Wed, Mar 13, 2024 at 11:02 AM Mickael Maison 
wrote:

> Hi,
>
> We have accumulated a number of personal branches in the github
> repository: https://github.com/apache/kafka/branches/all
>
> All these branches have been created by committers for various
> reasons, bugfix, tests.
>
> I wonder if we should avoid creating branches in the apache repository
> (always use your own fork like regular contributors) and in the rare
> cases this is necessary ensure we delete them once done? This way we
> would only have branches for the various releases (3.7, 3.6, etc).
>
> What do you think?
>
> Thanks,
> Mickael
>

Re: [VOTE] KIP-981: Manage Connect topics with custom implementation of Admin

[Mickael Maison]

Hi Omnia,

+1 (binding), thanks for the KIP

Mickael

On Tue, Mar 5, 2024 at 10:46 AM Omnia Ibrahim  wrote:
>
> Hi everyone, I would like to start the vote on KIP-981: Manage Connect topics 
> with custom implementation of Admin 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-981%3A+Manage+Connect+topics+with+custom+implementation+of+Admin
>
> Thanks
> Omnia

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #109

[Apache Jenkins Server]

See 

Re: [DISCUSS] Personal branches under apache/kafka

[Bill Bejeck]

+1

Sounds like a good idea to me.

-Bill

On Wed, Mar 13, 2024 at 2:55 PM Matthias J. Sax  wrote:

> +1
>
> Should be fine to just delete all of them (as long as nobody raised
> objections).
>
> Not sure if we could enable some protection GitHub side that disallow to
> push into non-existing branches and thus avoids accidental branch creation?
>
>
> -Matthias
>
> On 3/13/24 11:39 AM, Josep Prat wrote:
> > Hi Michael,
> >
> > I think it's a good idea. Only "official" branches should exist in the
> > upstream repo.
> > I guess the only exception would be if a massive feature would be done by
> > different individuals collaborating and they would need a "neutral" place
> > for the branch to be. But This didn't happen yet and I doubt it will in
> the
> > near future.
> >
> > Best,
> >
> > ---
> > Josep Prat
> > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Wed, Mar 13, 2024, 19:27 José Armando García Sancio
> >  wrote:
> >
> >> On Wed, Mar 13, 2024 at 11:02 AM Mickael Maison
> >>  wrote:
> >>> What do you think?
> >>
> >> I agree. I wouldn't be surprised if these branches (not trunk or
> >> release branches) were created by mistake by the committer.
> >>
> >> Thanks,
> >> --
> >> -José
> >>
> >
>

[PR] Adding OREXES to powered-by section [kafka-site]

[via GitHub]

siifuu opened a new pull request, #591:
URL: https://github.com/apache/kafka-site/pull/591

   OREXES is an it-security consultancy providing customers with specialized 
solutions suited to their use cases.
   They service large customers in the public and private sector in Germany.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

[Arpit Goyal]

Hi Greg,
I do not have access to update the KIP , Divij is helping me to do it.
Meanwhile let me update your queries here.

Backward compatibility:
These changes will not impact the existing functionality as the existing
behaviour always expects producer snapshot files to be present for a given
segment. Making Producer Snapshot file optional helps to cover both the
scenario i.e. both existing  and non existing of the producer snapshot file.

The getter of producer snapshot file  would also be changed as described
below:

Current

/**
 * @return Producer snapshot file until this segment.
 */
public Path producerSnapshotIndex() {
return producerSnapshotIndex;
}


Proposed

/**
 * @return Producer snapshot file until this segment.
 */
public Optional producerSnapshotIndex() {
return producerSnapshotIndex;
}


Thanks and Regards
Arpit Goyal
8861094754


On Wed, Mar 13, 2024 at 9:25 PM Greg Harris 
wrote:

> Hi Arpit,
>
> Thanks for the KIP!
>
> I am not familiar with the necessity of producer snapshots, but your
> explanation sounds like this should be made optional.
>
> Can you expand the KIP to include the changes that need to be made to
> the constructor and getter, and explain more about backwards
> compatibility? From the description I can't tell if this change is
> backwards-compatible or not.
>
> Thanks,
> Greg
>
> On Wed, Mar 13, 2024 at 6:48 AM Arpit Goyal 
> wrote:
> >
> > Hi all,
> >
> > I just wanted to bump up this thread.
> >
> > The KIP introduces a really small change  and it would not take much of
> the
> > time reviewing it.  This change would enable kafka users to use  tiered
> > storage features seamlessly  for the topics migrated  from < 2.8 version
> > which currently failed with NullPointerException.
> >
> > I am waiting for this KIP to get approved and then start working on it.
> >
> > On Mon, Mar 11, 2024, 14:26 Arpit Goyal 
> wrote:
> >
> > > Hi All,
> > > Just a Reminder, KIP-1026  is open for discussion.
> > > Thanks and Regards
> > > Arpit Goyal
> > > 8861094754
> > >
> > >
> > > On Sat, Mar 9, 2024 at 9:27 AM Arpit Goyal 
> > > wrote:
> > >
> > >> Hi All,
> > >>
> > >> I have created KIP-1026 for handling producerSnapshot empty scenarios
> > >> when the topic is upgraded from the kafka  < 2.8 version.
> > >>
> > >>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1026%3A+Handling+producer+snapshot+when+upgrading+from+%3C+v2.8.0+for+Tiered+Storage
> > >>
> > >> Feedback and suggestions are welcome.
> > >>
> > >> Thanks and Regards
> > >> Arpit Goyal
> > >> 8861094754
> > >>
> > >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2720

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-16180) Full metadata request sometimes fails during zk migration

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-16180.
--
Resolution: Fixed

> Full metadata request sometimes fails during zk migration
> -
>
> Key: KAFKA-16180
> URL: https://issues.apache.org/jira/browse/KAFKA-16180
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Colin McCabe
>Priority: Blocker
> Fix For: 3.6.2, 3.7.0
>
>
> Example:
> {code:java}
> java.util.NoSuchElementException: topic_name
> at 
> scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:508)
> at 
> scala.collection.mutable.AnyRefMap$ExceptionDefault.apply(AnyRefMap.scala:507)
> at scala.collection.mutable.AnyRefMap.apply(AnyRefMap.scala:207)
> at 
> kafka.server.metadata.ZkMetadataCache$.$anonfun$maybeInjectDeletedPartitionsFromFullMetadataRequest$2(ZkMetadataCache.scala:112)
> at 
> kafka.server.metadata.ZkMetadataCache$.$anonfun$maybeInjectDeletedPartitionsFromFullMetadataRequest$2$adapted(ZkMetadataCache.scala:105)
> at scala.collection.immutable.HashSet.foreach(HashSet.scala:958)
> at 
> kafka.server.metadata.ZkMetadataCache$.maybeInjectDeletedPartitionsFromFullMetadataRequest(ZkMetadataCache.scala:105)
> at 
> kafka.server.metadata.ZkMetadataCache.$anonfun$updateMetadata$1(ZkMetadataCache.scala:506)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:183)
> at 
> kafka.server.metadata.ZkMetadataCache.updateMetadata(ZkMetadataCache.scala:496)
> at 
> kafka.server.ReplicaManager.maybeUpdateMetadataCache(ReplicaManager.scala:2482)
> at 
> kafka.server.KafkaApis.handleUpdateMetadataRequest(KafkaApis.scala:733)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:349)
> at 
> kafka.server.KafkaRequestHandler.$anonfun$poll$8(KafkaRequestHandler.scala:210)
> at 
> kafka.server.KafkaRequestHandler.$anonfun$poll$8$adapted(KafkaRequestHandler.scala:210)
> at 
> io.confluent.kafka.availability.ThreadCountersManager.wrapEngine(ThreadCountersManager.java:146)
> at 
> kafka.server.KafkaRequestHandler.poll(KafkaRequestHandler.scala:210)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:151)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> at org.apache.kafka.common.utils.KafkaThread.run(KafkaThread.java:66)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16226) Java client: Performance regression in Trogdor benchmark with high partition counts

[Ismael Juma (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-16226.
-
Resolution: Fixed

> Java client: Performance regression in Trogdor benchmark with high partition 
> counts
> ---
>
> Key: KAFKA-16226
> URL: https://issues.apache.org/jira/browse/KAFKA-16226
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 3.7.0, 3.6.1
>Reporter: Mayank Shekhar Narula
>Assignee: Mayank Shekhar Narula
>Priority: Major
>  Labels: kip-951
> Fix For: 3.6.2, 3.8.0, 3.7.1
>
> Attachments: baseline_lock_profile.png, kafka_15415_lock_profile.png
>
>
> h1. Background
> https://issues.apache.org/jira/browse/KAFKA-15415 implemented optimisation in 
> java-client to skip backoff period if client knows of a newer leader, for 
> produce-batch being retried.
> h1. What changed
> The implementation introduced a regression noticed on a trogdor-benchmark 
> running with high partition counts(36000!).
> With regression, following metrics changed on the produce side.
>  # record-queue-time-avg: increased from 20ms to 30ms.
>  # request-latency-avg: increased from 50ms to 100ms.
> h1. Why it happened
> As can be seen from the original 
> [PR|https://github.com/apache/kafka/pull/14384] 
> RecordAccmulator.partitionReady() & drainBatchesForOneNode() started using 
> synchronised method Metadata.currentLeader(). This has led to increased 
> synchronization between KafkaProducer's application-thread that call send(), 
> and background-thread that actively send producer-batches to leaders.
> Lock profiles clearly show increased synchronisation in KAFKA-15415 
> PR(highlighted in {color:#de350b}Red{color}) Vs baseline ( see below ). Note 
> the synchronisation is much worse for paritionReady() in this benchmark as 
> its called for each partition, and it has 36k partitions!
> h3. Lock Profile: Kafka-15415
> !kafka_15415_lock_profile.png!
> h3. Lock Profile: Baseline
> !baseline_lock_profile.png!
> h1. Fix
> Synchronization has to be reduced between 2 threads in order to address this. 
> [https://github.com/apache/kafka/pull/15323] is a fix for it, as it avoids 
> using Metadata.currentLeader() instead rely on Cluster.leaderFor().
> With the fix, lock-profile & metrics are similar to baseline.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Manikumar]

Hi Edoardo,

Sure, we can include KAFKA-16369
 based PR reviews.

Thanks,

On Thu, Mar 14, 2024 at 3:16 PM Edoardo Comar  wrote:

> Hi Manikumar,
> can we please include
> https://issues.apache.org/jira/browse/KAFKA-16369
> in 3.6.2 ?
> While it's an issue still in trunk we actually discovered it our 3.6.1
> systems
>
> thanks,
> Edo
>
> On Thu, 14 Mar 2024 at 08:39, Divij Vaidya 
> wrote:
> >
> > Hi Manikumar,
> >
> > 1. Can you please take a look at
> https://github.com/apache/kafka/pull/15490
> > which is a bug fix specific to the 3.6.x branch?
> > 2. Should we do a one-time update of all dependencies in 3.6.x branch
> > before releasing 3.6.2?
> > 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> > backporting is needed to make the release qualification easier.
> > 4. There are a large number of bugs reported as impacting 3.6.1 [1] Some
> of
> > them have attached PRs and pending review. Maybe we can request all
> > committers to take a look at the ones which have a PR attached and see if
> > we can close them in the next few days before 3.6.2. Note that this will
> be
> > on a best-effort basis and won't block release of 3.6.2.
> > 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged
> whether
> > something needs to be backported? Usually it is the responsibility of the
> > reviewer but I have observed that sometimes we forget to backport
> important
> > onces as well. I can help with this one early next week.
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> >
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Thu, Mar 14, 2024 at 7:55 AM Manikumar 
> wrote:
> >
> > > Hi all,
> > >
> > > Here is the release plan for 3.6.2:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> > >
> > > Currently there is one open non-blocker issue. I plan to generate the
> first
> > > release candidate
> > > once the issue is resolved and no other issues are raised in the
> meantime.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > > wrote:
> > >
> > > > +1, Thanks Mani for volunteering.
> > > >
> > > > On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
> > > > >
> > > > > +1, Thanks Manikumar!
> > > > >
> > > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna 
> > > > wrote:
> > > > >
> > > > > > Thanks Manikumar!
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > Best,
> > > > > > Bruno
> > > > > >
> > > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > > +1 thanks for volunteering!
> > > > > > >
> > > > > > > Best
> > > > > > > ---
> > > > > > >
> > > > > > > Josep Prat
> > > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > > > +491715557497 | aiven.io
> > > > > > > Aiven Deutschland GmbH
> > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >
> > > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> +1
> > > > > > >>
> > > > > > >> Thank you for volunteering.
> > > > > > >>
> > > > > > >> --
> > > > > > >> Divij Vaidya
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > > > >> 
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Thanks Manikumar!
> > > > > > >>> +1 from me
> > > > > > >>>
> > > > > > >>> Justine
> > > > > > >>>
> > > > > > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar <
> > > > manikumar.re...@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > >  Hi,
> > > > > > 
> > > > > >  I'd like to volunteer to be the release manager for a bug
> fix
> > > > release
> > > > > > >> of
> > > > > >  the 3.6 line.
> > > > > >  If there are no objections, I'll send out the release plan
> soon.
> > > > > > 
> > > > > >  Thanks,
> > > > > >  Manikumar
> > > > > > 
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > >
>

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Manikumar]

Hi Divij,

1. I will take a look
2. I am not sure what we did for earlier releases. But we can update the
dependencies if any CVE exists
3. Thanks for looking into backports.
4. Sure, we can ping the committers  where PR is available
5. I have not looked at 3.7 bugs. I will also take a look.

Thanks,
Manikumar

On Thu, Mar 14, 2024 at 2:09 PM Divij Vaidya 
wrote:

> Hi Manikumar,
>
> 1. Can you please take a look at
> https://github.com/apache/kafka/pull/15490
> which is a bug fix specific to the 3.6.x branch?
> 2. Should we do a one-time update of all dependencies in 3.6.x branch
> before releasing 3.6.2?
> 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> backporting is needed to make the release qualification easier.
> 4. There are a large number of bugs reported as impacting 3.6.1 [1] Some of
> them have attached PRs and pending review. Maybe we can request all
> committers to take a look at the ones which have a PR attached and see if
> we can close them in the next few days before 3.6.2. Note that this will be
> on a best-effort basis and won't block release of 3.6.2.
> 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged whether
> something needs to be backported? Usually it is the responsibility of the
> reviewer but I have observed that sometimes we forget to backport important
> onces as well. I can help with this one early next week.
>
> [1]
>
> https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>
>
> --
> Divij Vaidya
>
>
>
> On Thu, Mar 14, 2024 at 7:55 AM Manikumar 
> wrote:
>
> > Hi all,
> >
> > Here is the release plan for 3.6.2:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> >
> > Currently there is one open non-blocker issue. I plan to generate the
> first
> > release candidate
> > once the issue is resolved and no other issues are raised in the
> meantime.
> >
> > Thanks,
> > Manikumar
> >
> > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana  >
> > wrote:
> >
> > > +1, Thanks Mani for volunteering.
> > >
> > > On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
> > > >
> > > > +1, Thanks Manikumar!
> > > >
> > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna 
> > > wrote:
> > > >
> > > > > Thanks Manikumar!
> > > > >
> > > > > +1
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > +1 thanks for volunteering!
> > > > > >
> > > > > > Best
> > > > > > ---
> > > > > >
> > > > > > Josep Prat
> > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > > +491715557497 | aiven.io
> > > > > > Aiven Deutschland GmbH
> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> +1
> > > > > >>
> > > > > >> Thank you for volunteering.
> > > > > >>
> > > > > >> --
> > > > > >> Divij Vaidya
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > > >> 
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Thanks Manikumar!
> > > > > >>> +1 from me
> > > > > >>>
> > > > > >>> Justine
> > > > > >>>
> > > > > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar <
> > > manikumar.re...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > >  Hi,
> > > > > 
> > > > >  I'd like to volunteer to be the release manager for a bug fix
> > > release
> > > > > >> of
> > > > >  the 3.6 line.
> > > > >  If there are no objections, I'll send out the release plan
> soon.
> > > > > 
> > > > >  Thanks,
> > > > >  Manikumar
> > > > > 
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>

[jira] [Created] (KAFKA-16374) High watermark updates should have a higher priority

[David Jacot (Jira)]

David Jacot created KAFKA-16374:
---

 Summary: High watermark updates should have a higher priority
 Key: KAFKA-16374
 URL: https://issues.apache.org/jira/browse/KAFKA-16374
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16373) Docker Official Image for Apache Kafka

[Krish Vora (Jira)]

Krish Vora created KAFKA-16373:
--

 Summary: Docker Official Image for Apache Kafka
 Key: KAFKA-16373
 URL: https://issues.apache.org/jira/browse/KAFKA-16373
 Project: Kafka
  Issue Type: New Feature
Affects Versions: 3.8.0
Reporter: Krish Vora
Assignee: Krish Vora


KIP-1028: Docker Official Image for Apache Kafka: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Official+Docker+Image+for+Kafka]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Minimum constraint for segment.ms

[Haruki Okada]

Hi, Divij.

This isn't about config default value/constraint change though, I found
there's a behavior discrepancy in max.block.ms config, which may cause
breaking change if we change the behavior.
The detail is described in the ticket:
https://issues.apache.org/jira/browse/KAFKA-16372

What do you think?

2024年3月14日(木) 13:09 Kamal Chandraprakash :

> One use case I see for setting the `segment.bytes` to 1 is to delete all
> the records from the topic.
> We can mention about it in the doc to use the `kafka-delete-records` API
> instead.
>
>
>
>
> On Wed, Mar 13, 2024 at 6:59 PM Divij Vaidya 
> wrote:
>
> > + users@kafka
> >
> > Hi users of Apache Kafka
> >
> > With the upcoming 4.0 release, we have an opportunity to improve the
> > constraints and default values for various Kafka configurations.
> >
> > We are soliciting your feedback and suggestions on configurations where
> the
> > default values and/or constraints should be adjusted. Please reply in
> this
> > thread directly.
> >
> > --
> > Divij Vaidya
> > Apache Kafka PMC
> >
> >
> >
> > On Wed, Mar 13, 2024 at 12:56 PM Divij Vaidya 
> > wrote:
> >
> > > Thanks for the discussion folks. I have started a KIP
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > to keep track of the changes that we are discussion. Please consider
> this
> > > as a collaborative work-in-progress KIP and once it is ready to be
> > > published, we can start a discussion thread on it.
> > >
> > > I am also going to start a thread to solicit feedback from users@
> > mailing
> > > list as well.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Wed, Mar 13, 2024 at 12:55 PM Christopher Shannon <
> > > christopher.l.shan...@gmail.com> wrote:
> > >
> > >> I think it's a great idea to raise a KIP to look at adjusting defaults
> > and
> > >> minimum/maximum config values for version 4.0.
> > >>
> > >> As pointed out, the minimum values for segment.ms and segment.bytes
> > don't
> > >> make sense and would probably bring down a cluster pretty quickly if
> set
> > >> that low, so version 4.0 is a good time to fix it and to also look at
> > the
> > >> other configs as well for adjustments.
> > >>
> > >> On Wed, Mar 13, 2024 at 4:39 AM Sergio Daniel Troiano
> > >>  wrote:
> > >>
> > >> > hey guys,
> > >> >
> > >> > Regarding to num.recovery.threads.per.data.dir: I agree, in our
> > company
> > >> we
> > >> > use the number of vCPUs to do so as this is not competing with ready
> > >> > cluster traffic.
> > >> >
> > >> >
> > >> > On Wed, 13 Mar 2024 at 09:29, Luke Chen  wrote:
> > >> >
> > >> > > Hi Divij,
> > >> > >
> > >> > > Thanks for raising this.
> > >> > > The valid minimum value 1 for `segment.ms` is completely
> > >> unreasonable.
> > >> > > Similarly for `segment.bytes`, `metadata.log.segment.ms`,
> > >> > > `metadata.log.segment.bytes`.
> > >> > >
> > >> > > In addition to that, there are also some config default values
> we'd
> > >> like
> > >> > to
> > >> > > propose to change in v4.0.
> > >> > > We can collect more comments from the community, and come out
> with a
> > >> KIP
> > >> > > for them.
> > >> > >
> > >> > > 1. num.recovery.threads.per.data.dir:
> > >> > > The current default value is 1. But the log recovery is happening
> > >> before
> > >> > > brokers are in ready state, which means, we should use all the
> > >> available
> > >> > > resource to speed up the log recovery to bring the broker to ready
> > >> state
> > >> > > soon. Default value should be... maybe 4 (to be decided)?
> > >> > >
> > >> > > 2. Other configs might be able to consider to change the default,
> > but
> > >> > open
> > >> > > for comments:
> > >> > >2.1. num.replica.fetchers: default is 1, but that's not enough
> > when
> > >> > > there are multiple partitions in the cluster
> > >> > >2.2. `socket.send.buffer.bytes`/`socket.receive.buffer.bytes`:
> > >> > > Currently, we set 100kb as default value, but that's not enough
> for
> > >> > > high-speed network.
> > >> > >
> > >> > > Thank you.
> > >> > > Luke
> > >> > >
> > >> > >
> > >> > > On Tue, Mar 12, 2024 at 1:32 AM Divij Vaidya <
> > divijvaidy...@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Hey folks
> > >> > > >
> > >> > > > Before I file a KIP to change this in 4.0, I wanted to
> understand
> > >> the
> > >> > > > historical context for the value of the following setting.
> > >> > > >
> > >> > > > Currently, segment.ms minimum threshold is set to 1ms [1].
> > >> > > >
> > >> > > > Segments are expensive. Every segment uses multiple file
> > descriptors
> > >> > and
> > >> > > > it's easy to run out of OS limits when creating a large number
> of
> > >> > > segments.
> > >> > > > Large number of segments also delays log loading on startup
> > because
> > >> of
> > >> > > > expensive operations such as iterating through all directories &
> > >> > > > conditionally loading all producer state.
> > >> > > >
> > >> 

[jira] [Created] (KAFKA-16372) max.block.ms behavior inconsistency with javadoc and the config description

[Haruki Okada (Jira)]

Haruki Okada created KAFKA-16372:


 Summary: max.block.ms behavior inconsistency with javadoc and the 
config description
 Key: KAFKA-16372
 URL: https://issues.apache.org/jira/browse/KAFKA-16372
 Project: Kafka
  Issue Type: Bug
  Components: clients
Reporter: Haruki Okada


As of Kafka 3.7.0, the javadoc of 
[KafkaProducer.send|https://github.com/apache/kafka/blob/3.7.0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L956]
 states that it throws TimeoutException when max.block.ms is exceeded on buffer 
allocation or initial metadata fetch.

Also it's stated in [max.block.ms config 
description|https://kafka.apache.org/37/documentation.html#producerconfigs_buffer.memory].

However, I found that this is not true because TimeoutException extends 
ApiException, and KafkaProducer.doSend catches ApiException and [wraps it as 
FutureFailure|https://github.com/apache/kafka/blob/3.7.0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L1075-L1086]
 instead of throwing it.

I wonder if this is a bug or the documentation error.

Seems this discrepancy exists since 0.9.0.0, which max.block.ms is introduced.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

[Omnia Ibrahim]

Hi everyone, I would like to start a discussion thread for KIP-1031:
Control offset translation in MirrorSourceConnector
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector

Thanks
Omnia

[jira] [Created] (KAFKA-16371) Unstable committed offsets after triggering commits where some metadata for some partitions are over the limit

[mlowicki (Jira)]

mlowicki created KAFKA-16371:


 Summary: Unstable committed offsets after triggering commits where 
some metadata for some partitions are over the limit
 Key: KAFKA-16371
 URL: https://issues.apache.org/jira/browse/KAFKA-16371
 Project: Kafka
  Issue Type: Bug
  Components: offset manager
Affects Versions: 3.7.0
Reporter: mlowicki


Issue is reproducible with simple CLI tool - 
[https://gist.github.com/mlowicki/c3b942f5545faced93dc414e01a2da70]

 

 
{code:java}
#!/usr/bin/env bash

for i in {1..100}
do
kafka-committer --bootstrap "ADDR:9092" --topic "TOPIC" --group bar 
--metadata-min 6000 --metadata-max 1 --partitions 72 --fetch
done{code}
 

 

What it does it that initially it fetches committed offsets and then tries to 
commit for multiple partitions. If some of commits have metadata over the 
allowed limit then:
1. I see errors about too large commits (expected)

2. Another run the tool fails at the stage of fetching commits with (this is 
the problem):

 

 
{code:java}
config: ClientConfig { conf_map: { "group.id": "bar", "bootstrap.servers": 
"ADDR:9092", }, log_level: Error, }
fetching committed offsets..
Error: Meta data fetch error: OperationTimedOut (Local: Timed out) Caused by: 
OperationTimedOut (Local: Timed out){code}
 


On the Kafka side I see _unstable_offset_commits_ errors reported by metric 
{code:java}
kafka.net.error_rate{code}

Increasing the timeout doesn't help and the only solution I've found is to 
trigger commits for all partitions with metadata below the limit or to use: 
{code:java}
isolation.level=read_uncommitted{code}
 

 

I don't know that code very but 
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L495-L500]
 seems fishy as it's using _offsetMetadata_ and not _filteredOffsetMetadata_ 
and I see that while removing those pending commits we use filtered offset 
metadata around 
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L400-L425
 
|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L400-L425]so
 the problem might be related to not cleaning up the data structure with 
pending commits properly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Edoardo Comar]

Hi Manikumar,
can we please include
https://issues.apache.org/jira/browse/KAFKA-16369
in 3.6.2 ?
While it's an issue still in trunk we actually discovered it our 3.6.1 systems

thanks,
Edo

On Thu, 14 Mar 2024 at 08:39, Divij Vaidya  wrote:
>
> Hi Manikumar,
>
> 1. Can you please take a look at https://github.com/apache/kafka/pull/15490
> which is a bug fix specific to the 3.6.x branch?
> 2. Should we do a one-time update of all dependencies in 3.6.x branch
> before releasing 3.6.2?
> 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> backporting is needed to make the release qualification easier.
> 4. There are a large number of bugs reported as impacting 3.6.1 [1] Some of
> them have attached PRs and pending review. Maybe we can request all
> committers to take a look at the ones which have a PR attached and see if
> we can close them in the next few days before 3.6.2. Note that this will be
> on a best-effort basis and won't block release of 3.6.2.
> 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged whether
> something needs to be backported? Usually it is the responsibility of the
> reviewer but I have observed that sometimes we forget to backport important
> onces as well. I can help with this one early next week.
>
> [1]
> https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>
>
> --
> Divij Vaidya
>
>
>
> On Thu, Mar 14, 2024 at 7:55 AM Manikumar  wrote:
>
> > Hi all,
> >
> > Here is the release plan for 3.6.2:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> >
> > Currently there is one open non-blocker issue. I plan to generate the first
> > release candidate
> > once the issue is resolved and no other issues are raised in the meantime.
> >
> > Thanks,
> > Manikumar
> >
> > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana 
> > wrote:
> >
> > > +1, Thanks Mani for volunteering.
> > >
> > > On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
> > > >
> > > > +1, Thanks Manikumar!
> > > >
> > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna 
> > > wrote:
> > > >
> > > > > Thanks Manikumar!
> > > > >
> > > > > +1
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > +1 thanks for volunteering!
> > > > > >
> > > > > > Best
> > > > > > ---
> > > > > >
> > > > > > Josep Prat
> > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > > +491715557497 | aiven.io
> > > > > > Aiven Deutschland GmbH
> > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya 
> > > > > wrote:
> > > > > >
> > > > > >> +1
> > > > > >>
> > > > > >> Thank you for volunteering.
> > > > > >>
> > > > > >> --
> > > > > >> Divij Vaidya
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > > >> 
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Thanks Manikumar!
> > > > > >>> +1 from me
> > > > > >>>
> > > > > >>> Justine
> > > > > >>>
> > > > > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar <
> > > manikumar.re...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > >  Hi,
> > > > > 
> > > > >  I'd like to volunteer to be the release manager for a bug fix
> > > release
> > > > > >> of
> > > > >  the 3.6 line.
> > > > >  If there are no objections, I'll send out the release plan soon.
> > > > > 
> > > > >  Thanks,
> > > > >  Manikumar
> > > > > 
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > >
> >

[jira] [Created] (KAFKA-16370) offline rollback procedure from kraft mode to zookeeper mode.

[kaushik srinivas (Jira)]

kaushik srinivas created KAFKA-16370:


 Summary: offline rollback procedure from kraft mode to zookeeper 
mode.
 Key: KAFKA-16370
 URL: https://issues.apache.org/jira/browse/KAFKA-16370
 Project: Kafka
  Issue Type: Improvement
Reporter: kaushik srinivas


>From the KIP, 
>[https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration,]

 
h2. Finalizing the Migration

Once the cluster has been fully upgraded to KRaft mode, the controller will 
still be running in migration mode and making dual writes to KRaft and ZK. 
Since the data in ZK is still consistent with that of the KRaft metadata log, 
it is still possible to revert back to ZK.

*_The time that the cluster is running all KRaft brokers/controllers, but still 
running in migration mode, is effectively unbounded._*

Once the operator has decided to commit to KRaft mode, the final step is to 
restart the controller quorum and take it out of migration mode by setting 
_zookeeper.metadata.migration.enable_ to "false" (or unsetting it). The active 
controller will only finalize the migration once it detects that all members of 
the quorum have signaled that they are finalizing the migration (again, using 
the tagged field in ApiVersionsResponse). Once the controller leaves migration 
mode, it will write a ZkMigrationStateRecord to the log and no longer perform 
writes to ZK. It will also disable its special handling of ZK RPCs.

*At this point, the cluster is fully migrated and is running in KRaft mode. A 
rollback to ZK is still possible after finalizing the migration, but it must be 
done offline and it will cause metadata loss (which can also cause partition 
data loss).*

 

Trying out the same in a kafka cluster which is migrated from zookeeper into 
kraft mode. We observe the rollback is possible by deleting the "/controller" 
node in the zookeeper before the rollback from kraft mode to zookeeper is done.

The above snippet indicates that the rollback from kraft to zk after migration 
is finalized is still possible in offline method. Is there any already known 
steps to be done as part of this offline method of rollback ?

>From our experience, we currently know of the step "deletion of /controller 
>node in zookeeper to force zookeper based brokers to be elected as new 
>controller after the rollback is done". Are there any additional steps/actions 
>apart from this ?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Divij Vaidya]

Hi Manikumar,

1. Can you please take a look at https://github.com/apache/kafka/pull/15490
which is a bug fix specific to the 3.6.x branch?
2. Should we do a one-time update of all dependencies in 3.6.x branch
before releasing 3.6.2?
3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
backporting is needed to make the release qualification easier.
4. There are a large number of bugs reported as impacting 3.6.1 [1] Some of
them have attached PRs and pending review. Maybe we can request all
committers to take a look at the ones which have a PR attached and see if
we can close them in the next few days before 3.6.2. Note that this will be
on a best-effort basis and won't block release of 3.6.2.
5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged whether
something needs to be backported? Usually it is the responsibility of the
reviewer but I have observed that sometimes we forget to backport important
onces as well. I can help with this one early next week.

[1]
https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC


--
Divij Vaidya



On Thu, Mar 14, 2024 at 7:55 AM Manikumar  wrote:

> Hi all,
>
> Here is the release plan for 3.6.2:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
>
> Currently there is one open non-blocker issue. I plan to generate the first
> release candidate
> once the issue is resolved and no other issues are raised in the meantime.
>
> Thanks,
> Manikumar
>
> On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana 
> wrote:
>
> > +1, Thanks Mani for volunteering.
> >
> > On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
> > >
> > > +1, Thanks Manikumar!
> > >
> > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna 
> > wrote:
> > >
> > > > Thanks Manikumar!
> > > >
> > > > +1
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > +1 thanks for volunteering!
> > > > >
> > > > > Best
> > > > > ---
> > > > >
> > > > > Josep Prat
> > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > +491715557497 | aiven.io
> > > > > Aiven Deutschland GmbH
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya 
> > > > wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >> Thank you for volunteering.
> > > > >>
> > > > >> --
> > > > >> Divij Vaidya
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > >> 
> > > > >> wrote:
> > > > >>
> > > > >>> Thanks Manikumar!
> > > > >>> +1 from me
> > > > >>>
> > > > >>> Justine
> > > > >>>
> > > > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar <
> > manikumar.re...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > >  Hi,
> > > > 
> > > >  I'd like to volunteer to be the release manager for a bug fix
> > release
> > > > >> of
> > > >  the 3.6 line.
> > > >  If there are no objections, I'll send out the release plan soon.
> > > > 
> > > >  Thanks,
> > > >  Manikumar
> > > > 
> > > > >>>
> > > > >>
> > > > >
> > > >
> >
>

[jira] [Resolved] (KAFKA-15997) Ensure fairness in the uniform assignor

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15997.
-
Resolution: Fixed

This issue got resolved by https://issues.apache.org/jira/browse/KAFKA-16249.

> Ensure fairness in the uniform assignor
> ---
>
> Key: KAFKA-15997
> URL: https://issues.apache.org/jira/browse/KAFKA-15997
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Emanuele Sabellico
>Assignee: David Jacot
>Priority: Minor
>
>  
>  
> Fairness has to be ensured in uniform assignor as it was in 
> cooperative-sticky one.
> There's this test 0113 subtest u_multiple_subscription_changes in librdkafka 
> where 8 consumers are subscribing to the same topic, and it's verifying that 
> all of them are getting 2 partitions assigned. But with new protocol it seems 
> two consumers get assigned 3 partitions and 1 has zero partitions. The test 
> doesn't configure any client.rack.
> {code:java}
> [0113_cooperative_rebalance  /478.183s] Consumer assignments 
> (subscription_variation 0) (stabilized) (no rebalance cb):
> [0113_cooperative_rebalance  /478.183s] Consumer C_0#consumer-3 assignment 
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [5] (2000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [8] (4000msgs)
> [0113_cooperative_rebalance  /478.183s] Consumer C_1#consumer-4 assignment 
> (3): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [0] (1000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [3] (2000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [13] (1000msgs)
> [0113_cooperative_rebalance  /478.184s] Consumer C_2#consumer-5 assignment 
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [6] (1000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [10] (2000msgs)
> [0113_cooperative_rebalance  /478.184s] Consumer C_3#consumer-6 assignment 
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [7] (1000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [9] (2000msgs)
> [0113_cooperative_rebalance  /478.184s] Consumer C_4#consumer-7 assignment 
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [11] (1000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [14] (3000msgs)
> [0113_cooperative_rebalance  /478.184s] Consumer C_5#consumer-8 assignment 
> (3): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [1] (2000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [2] (2000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [4] (1000msgs)
> [0113_cooperative_rebalance  /478.184s] Consumer C_6#consumer-9 assignment 
> (0): 
> [0113_cooperative_rebalance  /478.184s] Consumer C_7#consumer-10 assignment 
> (2): rdkafkatest_rnd24419cc75e59d8de_0113u_1 [12] (1000msgs), 
> rdkafkatest_rnd24419cc75e59d8de_0113u_1 [15] (1000msgs)
> [0113_cooperative_rebalance  /478.184s] 16/32 partitions assigned
> [0113_cooperative_rebalance  /478.184s] Consumer C_0#consumer-3 has 2 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance  /478.184s] Consumer C_1#consumer-4 has 3 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance  /478.184s] Consumer C_2#consumer-5 has 2 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance  /478.184s] Consumer C_3#consumer-6 has 2 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance  /478.184s] Consumer C_4#consumer-7 has 2 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance  /478.184s] Consumer C_5#consumer-8 has 3 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance  /478.184s] Consumer C_6#consumer-9 has 0 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [0113_cooperative_rebalance  /478.184s] Consumer C_7#consumer-10 has 2 
> assigned partitions (1 subscribed topic(s)), expecting 2 assigned partitions
> [                      /479.057s] 1 test(s) running: 
> 0113_cooperative_rebalance
> [                      /480.057s] 1 test(s) running: 
> 0113_cooperative_rebalance
> [                      /481.057s] 1 test(s) running: 
> 0113_cooperative_rebalance
> [0113_cooperative_rebalance  /482.498s] TEST FAILURE
> ### Test "0113_cooperative_rebalance (u_multiple_subscription_changes:2390: 
> use_rebalance_cb: 0, subscription_variation: 0)" failed at 
> test.c:1243:check_test_timeouts() at Thu Dec  7 15:52:15 2023: ###
> Test 0113_cooperative_rebalance (u_multiple_subscription_changes:2390: 
> use_rebalance_cb: 0, subscription_variation: 0) timed out (timeout set to 480 
> seconds)
> ./run-test.sh: line 62: 3512920 Killed                  $TEST $ARGS
> ###
> ### Test ./test-runner in bare mode FAILED! (return code 137) ###
> ###{code}
>  
>  



-

[jira] [Resolved] (KAFKA-16249) Improve reconciliation state machine

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16249.
-
Fix Version/s: 3.8.0
   Resolution: Fixed

> Improve reconciliation state machine
> 
>
> Key: KAFKA-16249
> URL: https://issues.apache.org/jira/browse/KAFKA-16249
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Major
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)