Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

[Jack Tomy]

Bumping up.

On Sun, Sep 17, 2023 at 9:34 AM Jack Tomy  wrote:

> Hey Ismael, Sagar and everyone,
>
> Sorry I seem to have interpreted the thread wrong. Before we go ahead with
> the DTO based approach I have few reasons not to go with it.
> a. It is not following the pattern we are following today. But here I
> agree that patterns are to be changed for good.
> b. The client is not supposed to modify the record object (This is my
> understanding, If this is not a necessary requirement, please call it
> out.), passing the entire object lets the client do that. To avoid that,
> there has to be a way to deep copy the record object each time, this adds
> unnecessary requirements on the record object to support the deepcopy
> implementation. I see a lot of complexity and coupling coming in here due
> to this N I believe it's a strong reason not to go ahead with the DTO
> approach.
>
> Please let me know what you think.
>
> Thanks.
>
>
>
>
>
> On Wed, Sep 6, 2023 at 7:06 AM Sagar  wrote:
>
>> Hey Jack,
>>
>> The way I interpreted this thread, it seems like there's more alignment on
>> the DTO based approach. I spent some time on the suggestion that Ismael
>> had
>> regarding the usage of ProducerRecord. Did you get a chance to look at the
>> reply I had posted and whether that makes sense? Also, checking out the
>> AdminClient APIs examples provided by Ismael will give you more context.
>> Let me know what you think.
>>
>> Thanks!
>> Sagar.
>>
>> On Thu, Aug 31, 2023 at 12:49 PM Jack Tomy  wrote:
>>
>> > Hey everyone,
>> >
>> > As I see devs favouring the current style of implementation, and that is
>> > inline with existing code. I would like to go ahead with the same
>> approach
>> > as mentioned in the KIP.
>> > Can I get a few more votes so that I can take the KIP forward.
>> >
>> > Thanks
>> >
>> >
>> >
>> > On Sun, Aug 27, 2023 at 1:38 PM Sagar 
>> wrote:
>> >
>> > > Hi Ismael,
>> > >
>> > > Thanks for pointing us towards the direction of a DTO based approach.
>> The
>> > > AdminClient examples seem very neat and extensible in that sense.
>> > > Personally, I was trying to think only along the lines of how the
>> current
>> > > Partitioner interface has been designed, i.e having all requisite
>> > > parameters as separate arguments (Topic, Key, Value etc).
>> > >
>> > > Regarding this question of yours:
>> > >
>> > > A more concrete question: did we consider having the method
>> `partition`
>> > > > take `ProduceRecord` as one of its parameters and `Cluster` as the
>> > other?
>> > >
>> > >
>> > > No, I don't think in the discussion thread it was brought up and as I
>> > said
>> > > it appears that could be due to an attempt to keep the new method's
>> > > signature similar to the existing one within Partitioner. If I
>> understood
>> > > the intent of the question correctly, are you trying to hint here that
>> > > `ProducerRecord` already contains all the arguments that the
>> `partition`
>> > > method accepts and also has a `headers` field within it. So, instead
>> of
>> > > adding another method for the `headers` field, why not create a new
>> > method
>> > > taking ProducerRecord directly?
>> > >
>> > > If my understanding is correct, then it seems like a very clean way of
>> > > adding support for `headers`. Anyways, the partition method within
>> > > KafkaProducer already takes a ProducerRecord as an argument so that
>> makes
>> > > it easier. Keeping that in mind, should this new method's (which
>> takes a
>> > > ProducerRecord as an input) default implementation invoke the existing
>> > > method ? One challenge I see there is that the existing partition
>> method
>> > > expects serialized keys and values while ProducerRecord doesn't have
>> > access
>> > > to those (It directly operates on K, V).
>> > >
>> > > Thanks!
>> > > Sagar.
>> > >
>> > >
>> > > On Sun, Aug 27, 2023 at 8:51 AM Ismael Juma 
>> wrote:
>> > >
>> > > > A more concrete question: did we consider having the method
>> `partition`
>> > > > take `ProduceRecord` as one of its parameters and `Cluster` as the
>> > other?
>> > > >
>> > > > Ismael
>> > > >
>> > > > On Sat, Aug 26, 2023 at 12:50 PM Greg Harris
>> > > > > > > >
>> > > > wrote:
>> > > >
>> > > > > Hey Ismael,
>> > > > >
>> > > > > > The mention of "runtime" is specific to Connect. When it comes
>> to
>> > > > > clients,
>> > > > > one typically compiles and runs with the same version or runs
>> with a
>> > > > newer
>> > > > > version than the one used for compilation. This is standard
>> practice
>> > in
>> > > > > Java and not something specific to Kafka.
>> > > > >
>> > > > > When I said "older runtimes" I was being lazy, and should have
>> said
>> > > > > "older versions of clients at runtime," thank you for figuring out
>> > > > > what I meant.
>> > > > >
>> > > > > I don't know how common it is to compile a partitioner against one
>> > > > > version of clients, and then distribute and run the partitioner
>> with
>> > > > > older versions of clients and expect 

[jira] [Created] (KAFKA-15516) Provide kafka-client library as a proper java Jigsaw module

[Chad Preisler (Jira)]

Chad Preisler created KAFKA-15516:
-

 Summary: Provide kafka-client library as a proper java Jigsaw 
module
 Key: KAFKA-15516
 URL: https://issues.apache.org/jira/browse/KAFKA-15516
 Project: Kafka
  Issue Type: Wish
  Components: clients
Affects Versions: 3.5.1
Reporter: Chad Preisler


When a client application depends on the kafka-client library the jlink tool 
can not be used to generate a custom JDK because the client jar is treated as 
an automatic module. 

The module-info class can be generated for the JAR files using the jdeps tool 
or a module-info can be created by hand for each jar. I believe the current 
structure of the Kafka code can be used as is.

It would be ideal if the Kafka build could properly bundle module-info in the 
client jar. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi everyone,

After a conversation with Colin McCabe and Proven Provenzano yesterday,
we decided that the benefits outweigh the concerns with the overhead
of associating a directory UUID to every replica in the metadata
partition records.
i.e. We prefer to always associate the log dir UUID even when only
one log dir is configured in the broker.

This reduces complexity in several ways:

* We no longer need to distinguish between JBOD and non JBOD
modes when changing or when interpreting partition metadata.
* Determining whether a replica is online through the metadata cache
no longer depends on the number of registered log directories.
* We can get rid of edge cases in the transition in or out
of multi log dir configuration, where the Controller would
have to update a lot of replica assignments.
* The OfflineLogDirs field in the broker registration is no longer
necessary.

So I'm updating the KIP with the following changes:

- "Directory" is no longer a tagged field in PartitionRecord
  and PartitionChangeRecord
- "OfflineLogDirs" is removed from BrokerRegistrationRequest,
  RegisterBrokerRecord and BrokerRegistrationChangeRecord
- "OnlineLogDirs" is renamed to "LogDirs" in BrokerRegistrationRequest,
  RegisterBrokerRecord and BrokerRegistrationChangeRecord

--
Igor

Re: Apache Kafka 3.6.0 release

[Divij Vaidya]

Ismael,
Thank you for checking.
Multiple other folks have validated after I left the comment here that
it doesn't impact log truncation and hence won't lead to data loss. I
agree that it's not a blocker.

(ref: https://github.com/apache/kafka/pull/14457)

--
Divij Vaidya

On Wed, Sep 27, 2023 at 8:50 PM Ismael Juma  wrote:
>
> Doesn't look like a blocker to me.
>
> Ismael
>
> On Wed, Sep 27, 2023 at 2:36 AM Divij Vaidya 
> wrote:
>
> > Hey team
> >
> > I need help in determining whether
> > https://github.com/apache/kafka/pull/14457 is a release blocker bug or
> > not. If someone is familiar with replication protocol (on the log
> > diverange and reconciliation process), please add your comments on the
> > PR.
> >
> > --
> > Divij Vaidya
> >
> > On Wed, Sep 27, 2023 at 10:43 AM Divij Vaidya 
> > wrote:
> > >
> > > A community member reported another bug in TS feature in 3.6 -
> > > https://issues.apache.org/jira/browse/KAFKA-15511
> > >
> > > I don't consider it as a blocker for release because the bug occurs in
> > > rare situations when the index on disk or in a remote store is
> > > corrupted and fails a sanity check.
> > > Sharing it here as an FYI.
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya 
> > wrote:
> > > >
> > > > Found a bug while testing TS feature in 3.6 -
> > > > https://issues.apache.org/jira/browse/KAFKA-15481
> > > >
> > > > I don't consider it as a blocker for release since it's a concurrency
> > > > bug that should occur rarely for a feature which is early access.
> > > > Sharing it here as FYI in case someone else thinks differently.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > > On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana <
> > satish.dugg...@gmail.com> wrote:
> > > > >
> > > > > Thanks Divij for raising a PR for doc formatting issue.
> > > > >
> > > > > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya, 
> > wrote:
> > > > >
> > > > > > Hey Satish
> > > > > >
> > > > > > I filed a PR to fix the website formatting bug in 3.6
> > documentation -
> > > > > > https://github.com/apache/kafka/pull/14419
> > > > > > Please take a look when you get a chance.
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > > On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton
> > 
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > I think this qualifies as a blocker. This API has been around
> > for years
> > > > > > now
> > > > > > > and, while we don't document it as not exposing duplicates*, it
> > has come
> > > > > > > with that implicit contract since its inception. More
> > importantly, it has
> > > > > > > also never exposed plugins that cannot be used on the worker.
> > This change
> > > > > > > in behavior not only introduces duplicates*, it causes
> > unreachable
> > > > > > plugins
> > > > > > > to be displayed. With this in mind, it seems to qualify pretty
> > clearly
> > > > > > as a
> > > > > > > regression and we should not put out a release that includes it.
> > > > > > >
> > > > > > > * - Really, these aren't duplicates; rather, they're multiple
> > copies of
> > > > > > the
> > > > > > > same plugin that come from different locations on the worker
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana <
> > satish.dugg...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > > Is this API documented that it does not return duplicate
> > entries?
> > > > > > > >
> > > > > > > > Can we also get an opinion from PMC/Committers who have
> > KafkaConnect
> > > > > > > > expertise on whether this issue is a release blocker?
> > > > > > > >
> > > > > > > > If we agree that it is not a release blocker then we can have a
> > > > > > > > release note clarifying this behaviour and add a reference to
> > the JIRA
> > > > > > > > that follows up on the possible solutions.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Satish.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, 19 Sept 2023 at 03:29, Greg Harris
> > > > > > 
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hey Satish,
> > > > > > > > >
> > > > > > > > > After investigating further, I believe that this is a
> > regression, but
> > > > > > > > > mostly a cosmetic one.
> > > > > > > > > I don't think there is significant risk of breaking clients
> > with this
> > > > > > > > > change, but it would be confusing for users, so I'd still
> > like to get
> > > > > > > > > the fix into the next RC.
> > > > > > > > > I've opened a PR here:
> > https://github.com/apache/kafka/pull/14398
> > > > > > and
> > > > > > > > > I'll work to get it merged promptly.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris <
> > greg.har...@aiven.io>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Satish,
> > > > > > 

[GitHub] [kafka-site] tombentley commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

tombentley commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1738021296

   Thanks for the contribution @qingwei91! 
   
   @divijvaidya thanks, updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-15515) Remove duplicated integration tests for new consumer

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-15515:
--

 Summary: Remove duplicated integration tests for new consumer
 Key: KAFKA-15515
 URL: https://issues.apache.org/jira/browse/KAFKA-15515
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Lianet Magrans


This task involves removing the temporary `PlaintextAsyncConsumer` file 
containing duplicated integration tests for the new consumer. The copy was 
generated to catch regressions and validate functionality in the new consumer 
while in development. It should be deleted when the new consumer is fully 
implemented and the existing integration tests (`PlaintextConsumerTest`) can be 
executed for both implementations.
 
Context:
 
For the current KafkaConsumer, a set of integration tests exist in the file 
PlaintextConsumerTest. Those tests cannot be executed as such for the new 
consumer implementation for 2 main reasons
- the new consumer is being developed as a new PrototypeAsyncConsumer class, in 
parallel to the existing KafkaConsumer. 
- the new consumer is under development, so it does not support all the 
consumer functionality yet. 
 
In order to be able to run the subsets of tests that the new consumer supports 
while the implementation completes, it was decided to :  
 - to make a copy of the `PlaintextAsyncConsumer` class, named 
PlaintextAsyncConsumer.
- leave all the existing integration tests that cover the simple consumer case 
unchanged, and disable the tests that are not yet supported by the new 
consumer. Disabled tests will be enabled as the async consumer
evolves.
 
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] tombentley merged pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

tombentley merged PR #410:
URL: https://github.com/apache/kafka-site/pull/410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2237

[Apache Jenkins Server]

See 

Re: Apache Kafka 3.6.0 release

[Ismael Juma]

Doesn't look like a blocker to me.

Ismael

On Wed, Sep 27, 2023 at 2:36 AM Divij Vaidya 
wrote:

> Hey team
>
> I need help in determining whether
> https://github.com/apache/kafka/pull/14457 is a release blocker bug or
> not. If someone is familiar with replication protocol (on the log
> diverange and reconciliation process), please add your comments on the
> PR.
>
> --
> Divij Vaidya
>
> On Wed, Sep 27, 2023 at 10:43 AM Divij Vaidya 
> wrote:
> >
> > A community member reported another bug in TS feature in 3.6 -
> > https://issues.apache.org/jira/browse/KAFKA-15511
> >
> > I don't consider it as a blocker for release because the bug occurs in
> > rare situations when the index on disk or in a remote store is
> > corrupted and fails a sanity check.
> > Sharing it here as an FYI.
> >
> > --
> > Divij Vaidya
> >
> > On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya 
> wrote:
> > >
> > > Found a bug while testing TS feature in 3.6 -
> > > https://issues.apache.org/jira/browse/KAFKA-15481
> > >
> > > I don't consider it as a blocker for release since it's a concurrency
> > > bug that should occur rarely for a feature which is early access.
> > > Sharing it here as FYI in case someone else thinks differently.
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana <
> satish.dugg...@gmail.com> wrote:
> > > >
> > > > Thanks Divij for raising a PR for doc formatting issue.
> > > >
> > > > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya, 
> wrote:
> > > >
> > > > > Hey Satish
> > > > >
> > > > > I filed a PR to fix the website formatting bug in 3.6
> documentation -
> > > > > https://github.com/apache/kafka/pull/14419
> > > > > Please take a look when you get a chance.
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > > On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton
> 
> > > > > wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I think this qualifies as a blocker. This API has been around
> for years
> > > > > now
> > > > > > and, while we don't document it as not exposing duplicates*, it
> has come
> > > > > > with that implicit contract since its inception. More
> importantly, it has
> > > > > > also never exposed plugins that cannot be used on the worker.
> This change
> > > > > > in behavior not only introduces duplicates*, it causes
> unreachable
> > > > > plugins
> > > > > > to be displayed. With this in mind, it seems to qualify pretty
> clearly
> > > > > as a
> > > > > > regression and we should not put out a release that includes it.
> > > > > >
> > > > > > * - Really, these aren't duplicates; rather, they're multiple
> copies of
> > > > > the
> > > > > > same plugin that come from different locations on the worker
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana <
> satish.dugg...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > > Is this API documented that it does not return duplicate
> entries?
> > > > > > >
> > > > > > > Can we also get an opinion from PMC/Committers who have
> KafkaConnect
> > > > > > > expertise on whether this issue is a release blocker?
> > > > > > >
> > > > > > > If we agree that it is not a release blocker then we can have a
> > > > > > > release note clarifying this behaviour and add a reference to
> the JIRA
> > > > > > > that follows up on the possible solutions.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Satish.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, 19 Sept 2023 at 03:29, Greg Harris
> > > > > 
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hey Satish,
> > > > > > > >
> > > > > > > > After investigating further, I believe that this is a
> regression, but
> > > > > > > > mostly a cosmetic one.
> > > > > > > > I don't think there is significant risk of breaking clients
> with this
> > > > > > > > change, but it would be confusing for users, so I'd still
> like to get
> > > > > > > > the fix into the next RC.
> > > > > > > > I've opened a PR here:
> https://github.com/apache/kafka/pull/14398
> > > > > and
> > > > > > > > I'll work to get it merged promptly.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris <
> greg.har...@aiven.io>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Satish,
> > > > > > > > >
> > > > > > > > > While validating 3.6.0-rc0, I noticed this regression as
> compared
> > > > > to
> > > > > > > > > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> > > > > > > > >
> > > > > > > > > Impact: The `connector-plugins` endpoint lists duplicates
> which may
> > > > > > > > > cause confusion for users, or poor behavior in clients.
> > > > > > > > > Using the other REST API endpoints appears unaffected.
> > > > > > > > > I'll open a PR for this later today.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Thu, Sep 14, 

[GitHub] [kafka-site] qingwei91 commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

qingwei91 commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1737867753

   Hi @mimaison , sure, I've incorporated the change. Would be nice to merge 
this if everything is ok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2236

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #71

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15514) Controller-side replica management changes

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15514:
---

 Summary: Controller-side replica management changes
 Key: KAFKA-15514
 URL: https://issues.apache.org/jira/browse/KAFKA-15514
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


The new "Assignments" field replaces the "Replicas" field in PartitionRecord 
and PartitionChangeRecord.

On the controller side, any changes to partitions need to consider both fields.
 * ISR updates
 * Partiton reassignments & reverts
 * Partition creation



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15489) split brain in KRaft cluster

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen reopened KAFKA-15489:
---

> split brain in KRaft cluster 
> -
>
> Key: KAFKA-15489
> URL: https://issues.apache.org/jira/browse/KAFKA-15489
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft
>Affects Versions: 3.5.1
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.6.0
>
>
> I found in the current KRaft implementation, when network partition happened 
> between the current controller leader and the other controller nodes, the 
> "split brain" issue will happen. It causes 2 leaders will exist in the 
> controller cluster, and 2 inconsistent sets of metadata will return to the 
> clients.
>  
> *Root cause*
> In 
> [KIP-595|https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595:ARaftProtocolfortheMetadataQuorum-Vote],
>  we said A voter will begin a new election under three conditions:
> 1. If it fails to receive a FetchResponse from the current leader before 
> expiration of quorum.fetch.timeout.ms
> 2. If it receives a EndQuorumEpoch request from the current leader
> 3. If it fails to receive a majority of votes before expiration of 
> quorum.election.timeout.ms after declaring itself a candidate.
> And that's exactly what the current KRaft's implementation.
>  
> However, when the leader is isolated from the network partition, there's no 
> way for it to resign from the leadership and start a new election. So the 
> leader will always be the leader even though all other nodes are down. And 
> this makes the split brain issue possible.
> When reading further in the KIP-595, I found we indeed considered this 
> situation and have solution for that. in [this 
> section|https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP595:ARaftProtocolfortheMetadataQuorum-LeaderProgressTimeout],
>  it said:
> {quote}In the pull-based model, however, say a new leader has been elected 
> with a new epoch and everyone has learned about it except the old leader 
> (e.g. that leader was not in the voters anymore and hence not receiving the 
> BeginQuorumEpoch as well), then that old leader would not be notified by 
> anyone about the new leader / epoch and become a pure "zombie leader", as 
> there is no regular heartbeats being pushed from leader to the follower. This 
> could lead to stale information being served to the observers and clients 
> inside the cluster.
> {quote}
> {quote}To resolve this issue, we will piggy-back on the 
> "quorum.fetch.timeout.ms" config, such that if the leader did not receive 
> Fetch requests from a majority of the quorum for that amount of time, it 
> would begin a new election and start sending VoteRequest to voter nodes in 
> the cluster to understand the latest quorum. If it couldn't connect to any 
> known voter, the old leader shall keep starting new elections and bump the 
> epoch.
> {quote}
>  
> But we missed this implementation in current KRaft.
>  
> *The flow is like this:*
> 1. 3 controller nodes, A(leader), B(follower), C(follower)
> 2. network partition happened between [A] and [B, C].
> 3. B and C starts new election since fetch timeout expired before receiving 
> fetch response from leader A.
> 4. B (or C) is elected as a leader in new epoch, while A is still the leader 
> in old epoch.
> 5. broker D creates a topic "new", and updates to leader B.
> 6. broker E describe topic "new", but got nothing because it is connecting to 
> the old leader A.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jira issues for CVEs in dependencies?

[Arnout Engelen]

Hello,

Kafka has the gradle dependency-check plugin configured, which can detect
when CVEs are issued for dependencies of the project. Now, more often than
not, those CVEs don't actually affect Kafka, but generally it's good to at
least consider them: every once in a while they might justify expediting an
update or applying a mitigation.

Currently I suspect few contributors ever run the report manually, and I
don't think it's published somewhere. The result is that people from
outside the project are posting Jira issues when these CVEs pop up in their
security scanning tooling, which seems like a missed opportunity.

Would it be interesting to (eventually automatically) create JIRA issues
for any CVEs flagged by dependency-check? I don't think that would create a
"dependabot-style" overwhelming amount of tickets: there's currently 9 CVEs
flagged when you exclude the :jmh-benchmarks subproject. It's not a problem
to make these JIRA tickets publicly available: given anyone can run that
report, and Kafka is not impacted by most CVEs in dependencies, we don't
consider the mere existence of those CVEs as sensitive information. When
someone looks into them and finds Kafka is impacted, it might be better to
continue the conversation on security@kafka.a.o. When people ask us about
CVEs flagged by their dependency scanners, we could point them to those
issues.

Looking further ahead, it would be great to have the conclusions of these
discussions in machine-readable form. For trunk, this could initially be
the dependencycheck suppressions file[0] for CVEs where Kafka is not
impacted. It might also be interesting to publish SBOM and VEX/VDR
descriptions where we can explicitly say we are or are not affected - if
there's sufficient interest, both for trunk and for currently-supported
releases.

I'd be happy to try things out and learn what might work best for Kafka!


Kind regards,

-- 
Arnout Engelen
ASF Security Response

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

[Viktor Somogyi-Vass]

Hi Elkhan,

I don't see the config you introduced on the KIP. Otherwise it looks
good to me. If no one has objections or anything to add then I think you
can start the vote after updating the KIP.

>I added the configurable poll interval as an optional - I believe it is
useful to let users to configure it but I doubt whether it would actually
be used.

Yes, it definitely won't be a high priority config but it's defensive. In
case of very large clusters I can imagine users might want longer periods
to avoid any unnecessary pressure from MM2.

Viktor

On Wed, Sep 27, 2023 at 12:50 AM Elxan Eminov 
wrote:

> Hi Team,
> Just pinging this
>
> Thanks,
> Elkhan
>
> On Thu, 21 Sep 2023 at 23:09, Elxan Eminov 
> wrote:
>
> > Hi Viktor and huqedi,
> > I've made the changes to the KIP.
> > I added the configurable poll interval as an optional - I believe it is
> > useful to let users to configure it but I doubt whether it would actually
> > be used.
> > Please let me know your thoughts on this,
> > Thanks!
> >
> > On Sat, 26 Aug 2023 at 15:49, Elxan Eminov 
> > wrote:
> >
> >> Relatively minor change with a new metric for MM2
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-971%3A+Expose+replication-offset-lag+MirrorMaker2+metric
> >>
> >
>

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

[Mayank Shekhar Narula]

Adding to Crispin. The new micro-benchmark shows improvements of 88% in
p99.9 with the KIP changes Vs baseline(& rejected alternate). Its
hypothesised improvements are seen as KIP avoids a full metadata refresh(Vs
baseline/rejected alternate), and the full metadata refresh can be slow due
to metadata reconvergence delay at the server(post leadership-change of
partitions). Extending this logic, KIP changes would be beneficial in
scenarios where full metadata refresh can be slow. Potential example would
be, metadata RPC is slowed due to head-of-line blocking by another slow RPC
in front, say a produce RPC(possibly slow due to churn in the ISR).

This new benchmark is in addition to the previously done benchmark of roll
simulation, where improvements upto 5% were seen.

Please do a review, as the voting thread is live.

Thanks!

On Wed, Sep 20, 2023 at 4:43 PM Crispin Bernier
 wrote:

> Hi,
>
> I've updated the KIP with benchmark results focusing more directly on the
> redirect case, please review and +1 in the voting thread if convinced.
>
> Thank you,
> Crispin
>
> On Wed, Jul 26, 2023 at 11:13 PM Luke Chen  wrote:
>
> > Thanks for adding the benchmark results, Crispin!
> > IMO, 2~5% performance improvement is small, but given the change is
> small,
> > cost is also small (only append endpoint info when
> NOT_LEADER_OR_FOLLOWER..
> > etc error), I think it is worth doing it.
> >
> > Thank you.
> > Luke
> >
> > On Wed, Jul 26, 2023 at 12:33 AM Ismael Juma  wrote:
> >
> > > Thanks Crispin!
> > >
> > > Ismael
> > >
> > > On Mon, Jul 24, 2023 at 8:16 PM Crispin Bernier
> > >  wrote:
> > >
> > > > I updated the wiki to include both results along with their average.
> > > >
> > > > Thank you,
> > > > Crispin
> > > >
> > > > On Mon, Jul 24, 2023 at 10:58 AM Ismael Juma 
> > wrote:
> > > >
> > > > > Hi Crispin,
> > > > >
> > > > > One additional question, the wiki says "The results are averaged
> > over 2
> > > > > runs.". Can you please provide some measure of variance in the
> > > > > distribution, i.e. were both results similar to each other for both
> > > > cases?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Jul 21, 2023 at 11:31 AM Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Thanks for the update Crispin - very helpful to have actual
> > > performance
> > > > > > data. 2-5% for the default configuration is a bit on the low side
> > for
> > > > > this
> > > > > > kind of proposal.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Jul 20, 2023 at 11:33 PM Crispin Bernier
> > > > > >  wrote:
> > > > > >
> > > > > >> Benchmark numbers have been posted on the KIP, please review.
> > > > > >>
> > > > > >> On 2023/07/20 13:03:00 Mayank Shekhar Narula wrote:
> > > > > >> > Jun
> > > > > >> >
> > > > > >> > Thanks for the feedback.
> > > > > >> >
> > > > > >> > Numbers to follow.
> > > > > >> >
> > > > > >> > If we don't plan to
> > > > > >> > > bump up the FetchResponse version, we could just remove the
> > > > > reference
> > > > > >> to
> > > > > >> > > version 16.
> > > > > >> >
> > > > > >> > Fixed.
> > > > > >> >
> > > > > >> > On Thu, Jul 20, 2023 at 1:28 AM Jun Rao
> > >  > > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Hi, Mayank,
> > > > > >> > >
> > > > > >> > > Thanks for the KIP. I agree with others that it would be
> > useful
> > > to
> > > > > >> see the
> > > > > >> > > performance results. Otherwise, just a minor comment. If we
> > > don't
> > > > > >> plan to
> > > > > >> > > bump up the FetchResponse version, we could just remove the
> > > > > reference
> > > > > >> to
> > > > > >> > > version 16.
> > > > > >> > >
> > > > > >> > > Jun
> > > > > >> > >
> > > > > >> > > On Wed, Jul 19, 2023 at 2:31 PM Mayank Shekhar Narula <
> > > > > >> > > mayanks.nar...@gmail.com> wrote:
> > > > > >> > >
> > > > > >> > > > Luke
> > > > > >> > > >
> > > > > >> > > > Thanks for the interest in the KIP.
> > > > > >> > > >
> > > > > >> > > > But what if the consumer was fetching from the follower?
> > > > > >> > > >
> > > > > >> > > > We already include `PreferredReadReplica` in the fetch
> > > response.
> > > > > >> > > > > Should we put the node info of PreferredReadReplica
> under
> > > this
> > > > > >> case,
> > > > > >> > > > > instead of the leader's info?
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > > PreferredReadReplica is the decided on the leader. Looking
> > at
> > > > the
> > > > > >> Java
> > > > > >> > > > client code, AbstractFetch::selectReadReplica, first fetch
> > > > request
> > > > > >> goes
> > > > > >> > > to
> > > > > >> > > > Leader of the partition -> Sends back PreferredReadReplica
> > ->
> > > > Next
> > > > > >> fetch
> > > > > >> > > > uses PreferredReadReplica. So as long as leader is
> > available,
> > > > > >> > > > PreferredReadReplica would be found in subsequent fetches.
> > > > > >> > > >
> > > > > >> > > > Also, under this case, should we include the leader's info
> > in
> > > > the
> > > > > >> > > response?
> > > > > >> > > >
> > 

Re: [VOTE] KIP-951: Leader discovery optimisations for the client

[Mayank Shekhar Narula]

Reviving this thread, as the discussion thread has been updated.

On Fri, Jul 28, 2023 at 11:29 AM Mayank Shekhar Narula <
mayanks.nar...@gmail.com> wrote:

> Thanks Jose.
>
> On Thu, Jul 27, 2023 at 5:46 PM José Armando García Sancio
>  wrote:
>
>> The KIP LGTM. Thanks for the design. I am looking forward to the
>> implementation.
>>
>> +1 (binding).
>>
>> Thanks!
>> --
>> -José
>>
>
>
> --
> Regards,
> Mayank Shekhar Narula
>


-- 
Regards,
Mayank Shekhar Narula

[jira] [Resolved] (KAFKA-15177) MirrorMaker 2 should implement the alterOffsets KIP-875 API

[Yash Mayya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yash Mayya resolved KAFKA-15177.

Fix Version/s: 3.6.0
   Resolution: Fixed

> MirrorMaker 2 should implement the alterOffsets KIP-875 API
> ---
>
> Key: KAFKA-15177
> URL: https://issues.apache.org/jira/browse/KAFKA-15177
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect, mirrormaker
>Reporter: Yash Mayya
>Assignee: Chris Egerton
>Priority: Minor
> Fix For: 3.6.0
>
>
> The {{MirrorSourceConnector}} class should implement the new alterOffsets API 
> added in 
> [KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect].
>  We could also implement the API in 
> {{MirrorCheckpointConnector}} and 
> {{MirrorHeartbeatConnector}} to prevent external modification of offsets 
> since the operation wouldn't really make sense in their case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Viktor Somogyi-Vass]

Hi Krishna,

Thanks for the answer. I've seen multiple such images where environment
variables are used and I think they are generally good but it's unsafe for
passing around secrets, jaas configs and so on. Perhaps for secrets we
could recommend using the file config provider. Then users can create and
mount secured properties file(s) with configs that are considered as
secrets. What do you think? Did you already have something in your mind
regarding this?

Thanks,
Viktor

On Tue, Sep 26, 2023 at 3:05 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi Ismael,
> Apologies for missing the mailing list in the last reply.
>
> Thank you for the suggestions.
> Just to clarify, the sizes mentioned in the previous email are of the
> uncompressed base images, not the resulting Apache Kafka docker images:
>
>1. eclipse-temurin:17-jre -- 263MB (They should release JRE 21 images
>soon)
>2. registry.access.redhat.com/ubi8/openjdk-17-runtime:1.17-1.1693366274
>-- 375MB
>
> Regards,
> Krishna
>
>
> On Tue, Sep 26, 2023 at 9:24 AM Ismael Juma  wrote:
>
> > Hi Krishna,
> >
> > Looks like you sent the response to me and not the mailing list,
> > please include the mailing list in the replies. Comments below.
> >
> > On Mon, Sep 25, 2023 at 11:45 AM Krishna Agarwal <
> > krishna0608agar...@gmail.com> wrote:
> >
> >> Hi Ismael,
> >> Thanks for the questions.
> >>
> >>1. We intend to support only the latest Java supported by Apache
> >>Kafka(As per this documentation
> >> Apache Kafka
> currently
> >>supports Java 8, Java 11, and Java 17) which currently is Java 17. If
> >>Apache Kafka supports Java 21 in the future, we will align with it.
> >>
> >> We are already building and testing with Java 21 (
> > https://github.com/apache/kafka/pull/14451 updates `README.md` to
> > indicate that). By 3.7.0 (the next release), we'll have Java 21 as one of
> > the officially supported versions. I think we should start with that
> > version for both docker image KIPs.
> >
> >>
> >>1. For users seeking a Docker image with an alternative Java version,
> >>they will have the flexibility to build their own Docker image
> utilising
> >>the Dockerfiles we provide. In our documentation, we will provide
> clear
> >>guidance on the designated base images for various Java versions.
> >>
> >> This sounds good to me. We should include these details as part of the
> > KIP and also the documentation for the docker images. More specifically,
> we
> > should state that we will update the Java major version as part of minor
> > Apache Kafka releases. The implication is that users who include broker
> > plugins alongside the broker should use custom images to ensure their
> > custom code is not broken by Java upgrades.
> >
> >>
> >>1. Apache Kafka only requires JRE, not JDK, for operation. Utilizing
> >>a base image with only JRE, rather than JDK, is a logical choice as
> it
> >>significantly reduces the size of the docker image.
> >>Upon further investigation, I discovered the eclipse-temurin
> >><
> https://hub.docker.com/layers/library/eclipse-temurin/17-jre/images/sha256-d1dfb065ae433fe1b43ac7e50a1ed03660f487c73ec256c686b126c37fd4d086?context=explore
> >
> >>docker image, which is notably smaller than Redhat’s ubi8 docker
> image (263
> >>MB vs 375 MB). Additionally, the fact that Apache Flink relies on
> >>eclipse-temurin base images
> >><
> https://github.com/apache/flink-docker/blob/master/1.17/scala_2.12-java11-ubuntu/Dockerfile#L19
> >
> >>further increases our confidence in their dependability(Will make
> this
> >>change in the KIP).
> >>
> >> Yes, eclipse-temurin looks like a good choice to me. Nice size
> reduction!
> >
> >>
> >>1. I'll conduct comparisons between our docker image and existing
> >>ones, and incorporate the findings into the KIP. I'll keep you
> posted on
> >>the same.
> >>
> >> Excellent, thanks!
> >
> > Ismael
> >
> >
> >> On Wed, Sep 20, 2023 at 11:26 PM Ismael Juma  wrote:
> >>
> >>> Hi Krishna,
> >>>
> >>> Thanks for the KIP. A few quick questions:
> >>>
> >>> 1. Since this will only be available for Kafka 3.7 in the best case, I
> >>> suggest we go with Java 21 instead of Java 17. Also, we should be clear
> >>> about Java version expectations. Are we allowed to change the Java
> >>> version
> >>> whenever we want? Or can it only be done in major versions? I prefer
> the
> >>> former option by default, but a subset of users may have their own
> >>> plugins
> >>> running within the broker. Such users may need to have control over the
> >>> Java version they use - how would they achieve tha
> >>> 2. We state that ubi-minimal would be the base image - it would be good
> >>> to
> >>> provide some details on why this was the choice.
> >>> 3. How does this docker image compare to the existing popular images?
> >>> Perhaps we can compare with the 3 most popular 

[jira] [Created] (KAFKA-15513) KRaft cluster fails with SCRAM authentication enabled for control-plane

[migruiz4 (Jira)]

migruiz4 created KAFKA-15513:


 Summary: KRaft cluster fails with SCRAM authentication enabled for 
control-plane
 Key: KAFKA-15513
 URL: https://issues.apache.org/jira/browse/KAFKA-15513
 Project: Kafka
  Issue Type: Bug
Reporter: migruiz4


We have observed a scenario where a KRaft cluster fails to bootstrap when using 
SCRAM authentication for controller-to-controller communications.

The steps to reproduce are simple:
 * Deploy (at least) 2 Kafka servers using latest version 3.5.1.
 * Configure a KRaft cluster, where the controller listener uses SASL_PLAINTEXT 
+ SCRAM-SHA-256 or SCRAM-SHA-512. In my case, I'm using the recommended in-line 
jaas config 
'{{{}listener.name..scram-sha-512.sasl.jaas.config{}}}'
 * Run 'kafka-storage.sh' in both nodes using option '--add-scram' to create 
the SCRAM user.

When initialized, Controllers will fail to connect to each other with an 
authentication error:

 
{code:java}
[2023-08-01 11:12:45,295] ERROR [kafka-1-raft-outbound-request-thread]: Failed 
to send the following request due to authentication error: 
ClientRequest(expectResponse=true, 
callback=kafka.raft.KafkaNetworkChannel$$Lambda$687/0x7f27d443fc60@2aba6075,
 destination=0, correlationId=129, clientId=raft-client-1, 
createdTimeMs=1690888364960, 
requestBuilder=VoteRequestData(clusterId='abcdefghijklmnopqrstug', 
topics=[TopicData(topicName='__cluster_metadata', 
partitions=[PartitionData(partitionIndex=0, candidateEpoch=4, candidateId=1, 
lastOffsetEpoch=0, lastOffset=0)])])) (kafka.raft.RaftSendThread) {code}
Some additional details about the scenario that we tested out:
 *  Controller listener does work when configured with SASL+PLAIN
 * The issue only affects the Controller listener, SCRAM users created using 
the same method work for data-plane listeners and inter-broker listeners.

 

Below you can find the exact configuration and command used to deploy:
 * server.properties

{code:java}
listeners=INTERNAL://:9092,CLIENT://:9091,CONTROLLER://:9093
advertised.listeners=INTERNAL://kafka-0:9092,CLIENT://:9091
listener.security.protocol.map=INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/bitnami/kafka/data
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.retention.check.interval.ms=30
controller.listener.names=CONTROLLER
controller.quorum.voters=0@kafka-0:9093,1@kafka-1:9093
inter.broker.listener.name=INTERNAL
node.id=0
process.roles=controller,broker
sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
sasl.mechanism.controller.protocol=SCRAM-SHA-512
listener.name.controller.sasl.enabled.mechanisms=SCRAM-SHA-512
listener.name.controller.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
 required username="controller_user" password="controller_password";{code}

 * kafka-storage.sh command

{code:java}
kafka-storage.sh format --config /path/to/server.properties --ignore-formatted 
--cluster-id abcdefghijklmnopqrstuv --add-scram 
SCRAM-SHA-512=[name=controller_user,password=controller_password] {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15512) EOL zookeeper component in kafka

[Aleksandra Drobnjak (Jira)]

Aleksandra Drobnjak created KAFKA-15512:
---

 Summary: EOL zookeeper component in kafka
 Key: KAFKA-15512
 URL: https://issues.apache.org/jira/browse/KAFKA-15512
 Project: Kafka
  Issue Type: Task
Affects Versions: 3.5.1, 3.4.1, 3.5.0, 3.4.0
Reporter: Aleksandra Drobnjak


Component: *zookeeper  3.6.3* in kafka 3.4.0, EOL:2022-12-30  Days EOL:270

https://endoflife.date/zookeeper

https://archive.apache.org/dist/kafka/3.4.0/kafka_2.12-3.4.0.tgz

No newer version of zookeeper in public kafka currently available.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Divij Vaidya]

Hey team

I need help in determining whether
https://github.com/apache/kafka/pull/14457 is a release blocker bug or
not. If someone is familiar with replication protocol (on the log
diverange and reconciliation process), please add your comments on the
PR.

--
Divij Vaidya

On Wed, Sep 27, 2023 at 10:43 AM Divij Vaidya  wrote:
>
> A community member reported another bug in TS feature in 3.6 -
> https://issues.apache.org/jira/browse/KAFKA-15511
>
> I don't consider it as a blocker for release because the bug occurs in
> rare situations when the index on disk or in a remote store is
> corrupted and fails a sanity check.
> Sharing it here as an FYI.
>
> --
> Divij Vaidya
>
> On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya  wrote:
> >
> > Found a bug while testing TS feature in 3.6 -
> > https://issues.apache.org/jira/browse/KAFKA-15481
> >
> > I don't consider it as a blocker for release since it's a concurrency
> > bug that should occur rarely for a feature which is early access.
> > Sharing it here as FYI in case someone else thinks differently.
> >
> > --
> > Divij Vaidya
> >
> > On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana  
> > wrote:
> > >
> > > Thanks Divij for raising a PR for doc formatting issue.
> > >
> > > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya,  
> > > wrote:
> > >
> > > > Hey Satish
> > > >
> > > > I filed a PR to fix the website formatting bug in 3.6 documentation -
> > > > https://github.com/apache/kafka/pull/14419
> > > > Please take a look when you get a chance.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > > On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton 
> > > > wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > I think this qualifies as a blocker. This API has been around for 
> > > > > years
> > > > now
> > > > > and, while we don't document it as not exposing duplicates*, it has 
> > > > > come
> > > > > with that implicit contract since its inception. More importantly, it 
> > > > > has
> > > > > also never exposed plugins that cannot be used on the worker. This 
> > > > > change
> > > > > in behavior not only introduces duplicates*, it causes unreachable
> > > > plugins
> > > > > to be displayed. With this in mind, it seems to qualify pretty clearly
> > > > as a
> > > > > regression and we should not put out a release that includes it.
> > > > >
> > > > > * - Really, these aren't duplicates; rather, they're multiple copies 
> > > > > of
> > > > the
> > > > > same plugin that come from different locations on the worker
> > > > >
> > > > > Best,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana 
> > > > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > > Is this API documented that it does not return duplicate entries?
> > > > > >
> > > > > > Can we also get an opinion from PMC/Committers who have KafkaConnect
> > > > > > expertise on whether this issue is a release blocker?
> > > > > >
> > > > > > If we agree that it is not a release blocker then we can have a
> > > > > > release note clarifying this behaviour and add a reference to the 
> > > > > > JIRA
> > > > > > that follows up on the possible solutions.
> > > > > >
> > > > > > Thanks,
> > > > > > Satish.
> > > > > >
> > > > > >
> > > > > > On Tue, 19 Sept 2023 at 03:29, Greg Harris
> > > > 
> > > > > > wrote:
> > > > > > >
> > > > > > > Hey Satish,
> > > > > > >
> > > > > > > After investigating further, I believe that this is a regression, 
> > > > > > > but
> > > > > > > mostly a cosmetic one.
> > > > > > > I don't think there is significant risk of breaking clients with 
> > > > > > > this
> > > > > > > change, but it would be confusing for users, so I'd still like to 
> > > > > > > get
> > > > > > > the fix into the next RC.
> > > > > > > I've opened a PR here: https://github.com/apache/kafka/pull/14398
> > > > and
> > > > > > > I'll work to get it merged promptly.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris 
> > > > > > > 
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > While validating 3.6.0-rc0, I noticed this regression as 
> > > > > > > > compared
> > > > to
> > > > > > > > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> > > > > > > >
> > > > > > > > Impact: The `connector-plugins` endpoint lists duplicates which 
> > > > > > > > may
> > > > > > > > cause confusion for users, or poor behavior in clients.
> > > > > > > > Using the other REST API endpoints appears unaffected.
> > > > > > > > I'll open a PR for this later today.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Thanks Justine for the update. I saw in the morning that these
> > > > > > changes
> > > > > > > > > are pushed to trunk and 3.6.
> > > > > > > > >
> > > > > > > > > ~Satish.
> > > > > > > > >
> > > > > 

[GitHub] [kafka-site] mimaison commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

mimaison commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1736988414

   @tombentley It would have been nice to understand the .htaccess 
inconsistency but the duplication is minor and this PR really simplifies 
testing the website so I'm happy to merge it.
   
   @qingwei91 Can you integrate the change Tom suggested?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Apache Kafka 3.6.0 release

[Divij Vaidya]

A community member reported another bug in TS feature in 3.6 -
https://issues.apache.org/jira/browse/KAFKA-15511

I don't consider it as a blocker for release because the bug occurs in
rare situations when the index on disk or in a remote store is
corrupted and fails a sanity check.
Sharing it here as an FYI.

--
Divij Vaidya

On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya  wrote:
>
> Found a bug while testing TS feature in 3.6 -
> https://issues.apache.org/jira/browse/KAFKA-15481
>
> I don't consider it as a blocker for release since it's a concurrency
> bug that should occur rarely for a feature which is early access.
> Sharing it here as FYI in case someone else thinks differently.
>
> --
> Divij Vaidya
>
> On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana  
> wrote:
> >
> > Thanks Divij for raising a PR for doc formatting issue.
> >
> > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya,  wrote:
> >
> > > Hey Satish
> > >
> > > I filed a PR to fix the website formatting bug in 3.6 documentation -
> > > https://github.com/apache/kafka/pull/14419
> > > Please take a look when you get a chance.
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton 
> > > wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > I think this qualifies as a blocker. This API has been around for years
> > > now
> > > > and, while we don't document it as not exposing duplicates*, it has come
> > > > with that implicit contract since its inception. More importantly, it 
> > > > has
> > > > also never exposed plugins that cannot be used on the worker. This 
> > > > change
> > > > in behavior not only introduces duplicates*, it causes unreachable
> > > plugins
> > > > to be displayed. With this in mind, it seems to qualify pretty clearly
> > > as a
> > > > regression and we should not put out a release that includes it.
> > > >
> > > > * - Really, these aren't duplicates; rather, they're multiple copies of
> > > the
> > > > same plugin that come from different locations on the worker
> > > >
> > > > Best,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana  > > >
> > > > wrote:
> > > >
> > > > > Hi Greg,
> > > > > Is this API documented that it does not return duplicate entries?
> > > > >
> > > > > Can we also get an opinion from PMC/Committers who have KafkaConnect
> > > > > expertise on whether this issue is a release blocker?
> > > > >
> > > > > If we agree that it is not a release blocker then we can have a
> > > > > release note clarifying this behaviour and add a reference to the JIRA
> > > > > that follows up on the possible solutions.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > >
> > > > > On Tue, 19 Sept 2023 at 03:29, Greg Harris
> > > 
> > > > > wrote:
> > > > > >
> > > > > > Hey Satish,
> > > > > >
> > > > > > After investigating further, I believe that this is a regression, 
> > > > > > but
> > > > > > mostly a cosmetic one.
> > > > > > I don't think there is significant risk of breaking clients with 
> > > > > > this
> > > > > > change, but it would be confusing for users, so I'd still like to 
> > > > > > get
> > > > > > the fix into the next RC.
> > > > > > I've opened a PR here: https://github.com/apache/kafka/pull/14398
> > > and
> > > > > > I'll work to get it merged promptly.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris 
> > > > > wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > While validating 3.6.0-rc0, I noticed this regression as compared
> > > to
> > > > > > > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> > > > > > >
> > > > > > > Impact: The `connector-plugins` endpoint lists duplicates which 
> > > > > > > may
> > > > > > > cause confusion for users, or poor behavior in clients.
> > > > > > > Using the other REST API endpoints appears unaffected.
> > > > > > > I'll open a PR for this later today.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Thanks Justine for the update. I saw in the morning that these
> > > > > changes
> > > > > > > > are pushed to trunk and 3.6.
> > > > > > > >
> > > > > > > > ~Satish.
> > > > > > > >
> > > > > > > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hi Satish,
> > > > > > > > > We were able to merge
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> > > > > > > > > and pick to 3.6.
> > > > > > > > >
> > > > > > > > > Hopefully nothing more from me on this release.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Luke for the update.
> > > > > > > > > >
> > > > > > > > > > 

[jira] [Created] (KAFKA-15511) Exception not handled correctly if indexFile is corrupted.

[Arpit Goyal (Jira)]

Arpit Goyal created KAFKA-15511:
---

 Summary:  Exception not handled correctly if indexFile is 
corrupted.
 Key: KAFKA-15511
 URL: https://issues.apache.org/jira/browse/KAFKA-15511
 Project: Kafka
  Issue Type: Bug
Reporter: Arpit Goyal
Assignee: Arpit Goyal
 Attachments: Screenshot 2023-09-27 at 1.01.58 PM.png, Screenshot 
2023-09-27 at 1.14.22 PM.png

I was simulating  a code flow where  there is a possibility of inconsistency 
between RemoteCache and fileStored on disk.


1. OffsetIndex Corrupt File already exist on disk 
2. There is no entry exist in the RemoteIndexCache 
3.  Call getIndexEntry
4. As File already exists on the disk and corrupted , indexSanityCheck  will 
throws "*CorruptIndexException*"

5. But the code flow in RemoteIndexCache  catches only 
*"CorruptRecordException".*

Ideally it should catch *CorruptIndexException* instead of  
*CorruptRecordException*.

Impact - Functionality is break on the above code flow , and  it will not able 
to auto-recover and overwriting the corrupted  index file. 

Check the screenshot attached for more reference. 

cc [~divijvaidya] [~satish.duggana] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)