[jira] [Created] (KAFKA-14106) Fetcher thread was shutdown and all fetched partitions are lost.

[Yang Ling (Jira)]

Yang Ling created KAFKA-14106:
-

 Summary: Fetcher thread was shutdown and all fetched partitions 
are lost.
 Key: KAFKA-14106
 URL: https://issues.apache.org/jira/browse/KAFKA-14106
 Project: Kafka
  Issue Type: Bug
  Components: replication
Affects Versions: 3.0.0, 2.2.2
Reporter: Yang Ling


Dynamic changes of listeners will lead into out of sync. Our operation is as 
following:
 # Broker is started and listening on a ip-address.
 # Create some topics.
 # Change listening to a domain name via dynamic-configuration for some reason.
 # Create some new topics.
 # Produce message into any older topics.
 # All topics, produced in step 5, are out of sync.

Following is major logs：
{panel}
[2022-07-23 15:30:53,282] INFO [ReplicaFetcherManager on broker 0] Added 
fetcher to broker BrokerEndPoint(id=2, host=168.1.3.88:9092) for partitions 
Map(test-11 -> (offset=0, leaderEpoch=0), test-5 -> (offset=0, leaderEpoch=0), 
test-8 -> (offset=0, leaderEpoch=0), test-2 -> (offset=0, leaderEpoch=0)) 
(kafka.server.ReplicaFetcherManager)
[2022-07-25 15:01:51,581] INFO [ReplicaFetcher replicaId=0, leaderId=2, 
fetcherId=0] Shutting down (kafka.server.ReplicaFetcherThread)
[2022-07-25 18:14:05,297] INFO [ReplicaFetcherManager on broker 0]Added fetcher 
to broker BrokerEndPoint(id=2, host=kafka-server-1:9092) for partitions 
Map(test2-6 -> (offset=0, leaderEpoch=0), test2-0 -> (offset=0, leaderEpoch=0), 
test2-3 -> (offset=0, leaderEpoch=0), test2-9 -> (offset=0, leaderEpoch=0)) 
(kafka.server.ReplicaFetcherManager)
{panel}

After read source code. We found following code in AbstractFetcherManager:
{code:scala}
def addFetcherForPartitions(partitionAndOffsets: Map[TopicPartition, 
InitialFetchState]) {
...
  for ((brokerAndFetcherId, initialFetchOffsets) <- partitionsPerFetcher) {
val brokerIdAndFetcherId = 
BrokerIdAndFetcherId(brokerAndFetcherId.broker.id, brokerAndFetcherId.fetcherId)
val fetcherThread = fetcherThreadMap.get(brokerIdAndFetcherId) match {
  case Some(currentFetcherThread) if currentFetcherThread.sourceBroker 
== brokerAndFetcherId.broker =>
currentFetcherThread
  case Some(f) =>
f.shutdown() // - marked
addAndStartFetcherThread(brokerAndFetcherId, brokerIdAndFetcherId)
  case None =>
addAndStartFetcherThread(brokerAndFetcherId, brokerIdAndFetcherId)
}
}
  }
...
}
{code}

As marked code defined, if sourceBroker is changed, in our case, the older 
fetcher thread will be shutdown and a new fetcher thread will be created using 
new sourceBroker. In this way, all of the fetched partitions in older fetcher 
thread will be lost.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Committer: Chris Egerton

[Kumud Kumar Srivatsava Tirupati]

Congratulations Chris!

On Tue, 26 Jul, 2022, 7:11 AM deng ziming,  wrote:

> Congratulations Chris !
>
> --
> Ziming
>
> > On Jul 26, 2022, at 5:01 AM, Matthias J. Sax  wrote:
> >
> > Congrats! Well deserved!
> >
> > -Matthias
> >
> > On 7/25/22 1:08 PM, Bill Bejeck wrote:
> >> Congrats Chris!
> >> -Bill
> >> On Mon, Jul 25, 2022 at 3:58 PM Jorge Esteban Quilcate Otoya <
> >> quilcate.jo...@gmail.com> wrote:
> >>> Congratulations Chris!
> >>>
> >>> On Mon, 25 Jul 2022 at 20:27, Robin Moffatt  >
> >>> wrote:
> >>>
>  Congrats Chris!
> 
> 
>  --
> 
>  Robin Moffatt | Principal Developer Advocate | ro...@confluent.io |
> >>> @rmoff
> 
> 
>  On Mon, 25 Jul 2022 at 17:26, Mickael Maison 
> >>> wrote:
> 
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Chris Egerton as a committer,
> and
> > we are excited to announce that he accepted!
> >
> > Chris has been contributing to Kafka since 2017. He has made over 80
> > commits mostly around Kafka Connect. His most notable contributions
> > include KIP-507: Securing Internal Connect REST Endpoints and
> KIP-618:
> > Exactly-Once Support for Source Connectors.
> >
> > He has been an active participant in discussions and reviews on the
> > mailing lists and on Github.
> >
> > Thanks for all of your contributions Chris. Congratulations!
> >
> > -- Mickael, on behalf of the Apache Kafka PMC
> >
> 
> >>>
>
>

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[Sagar]

Hello Guozhang,

Thank you so much for the doc on Kafka Streams. Sure, I would do the
analysis and come up with such a document.

Thanks!
Sagar.

On Tue, Jul 26, 2022 at 4:47 AM Guozhang Wang  wrote:

> Hello Sagar,
>
> It would be great if you could come back with some analysis on how to
> implement the Connect side integration with the new protocol; so far
> besides leveraging on the new "protocol type" we did not yet think through
> the Connect side implementations. For Streams here's a draft of integration
> plan:
>
> https://docs.google.com/document/d/17PNz2sGoIvGyIzz8vLyJTJTU2rqnD_D9uHJnH9XARjU/edit#heading=h.pdgirmi57dvn
> just FYI for your analysis on Connect.
>
> On Tue, Jul 19, 2022 at 10:48 PM Sagar  wrote:
>
> > Hi David,
> >
> > Thank you for your response. The reason I thought connect can also fit
> into
> > this new scheme is that even today the connect uses a WorkerCoordinator
> > extending from AbstractCoordinator to empower rebalances of
> > tasks/connectors. The WorkerCoordinator sets the protocolType() to
> connect
> > and uses the metadata() method by plumbing into JoinGroupRequestProtocol.
> >
> > I think the changes to support connect would be similar at a high level
> to
> > the changes in streams mainly because of the Client side assignors being
> > used in both. At an implementation level, we might need to make a lot of
> > changes to get onto this new assignment protocol like enhancing the
> > JoinGroup request/response and SyncGroup and using ConsumerGroupHeartbeat
> > API etc again on similar lines to streams (or there might be
> deviations). I
> > would try to perform a detailed analysis of the same  and we can have a
> > separate discussion thread for that as that would derail this discussion
> > thread. Let me know if that sounds good to you.
> >
> > Thanks!
> > Sagar.
> >
> >
> >
> > On Fri, Jul 15, 2022 at 5:47 PM David Jacot  >
> > wrote:
> >
> > > Hi Sagar,
> > >
> > > Thanks for your comments.
> > >
> > > 1) Yes. That refers to `Assignment#error`. Sure, I can mention it.
> > >
> > > 2) The idea is to transition C from his current assignment to his
> > > target assignment when he can move to epoch 3. When that happens, the
> > > member assignment is updated and persisted with all its assigned
> > > partitions even if they are not all revoked yet. In other words, the
> > > member assignment becomes the target assignment. This is basically an
> > > optimization to avoid having to write all the changes to the log. The
> > > examples are based on the persisted state so I understand the
> > > confusion. Let me see if I can improve this in the description.
> > >
> > > 3) Regarding Connect, it could reuse the protocol with a client side
> > > assignor if it fits in the protocol. The assignment is about
> > > topicid-partitions + metadata, could Connect fit into this?
> > >
> > > Best,
> > > David
> > >
> > > On Fri, Jul 15, 2022 at 1:55 PM Sagar 
> wrote:
> > > >
> > > > Hi David,
> > > >
> > > > Thanks for the KIP. I just had minor observations:
> > > >
> > > > 1) In the Assignment Error section in Client Side mode Assignment
> > > process,
> > > > you mentioned => `In this case, the client side assignor can return
> an
> > > > error to the group coordinator`. In this case are you referring to
> the
> > > > Assignor returning an AssignmentError that's listed down towards the
> > end?
> > > > If yes, do you think it would make sense to mention this explicitly
> > here?
> > > >
> > > > 2) In the Case Studies section, I have a slight confusion, not sure
> if
> > > > others have the same. Consider this step:
> > > >
> > > > When B heartbeats, the group coordinator transitions him to epoch 3
> > > because
> > > > B has no partitions to revoke. It persists the change and reply.
> > > >
> > > >- Group (epoch=3)
> > > >   - A
> > > >   - B
> > > >   - C
> > > >- Target Assignment (epoch=3)
> > > >   - A - partitions=[foo-0]
> > > >   - B - partitions=[foo-2]
> > > >   - C - partitions=[foo-1]
> > > >- Member Assignment
> > > >   - A - epoch=2, partitions=[foo-0, foo-1]
> > > >   - B - epoch=3, partitions=[foo-2]
> > > >   - C - epoch=3, partitions=[foo-1]
> > > >
> > > > When C heartbeats, it transitions to epoch 3 but cannot get foo-1
> yet.
> > > >
> > > > Here,it's mentioned that member C can't get the foo-1 partition yet,
> > but
> > > > based on the description above, it seems it already has it. Do you
> > think
> > > it
> > > > would be better to remove it and populate it only when it actually
> gets
> > > it?
> > > > I see this in a lot of other places, so have I understood it
> > incorrectly
> > > ?
> > > >
> > > >
> > > > Regarding connect , it might be out of scope of this discussion, but
> > from
> > > > what I understood it would probably be running in client side
> assignor
> > > mode
> > > > even on the new rebalance protocol as it has its own Custom
> > > Assignors(Eager
> > > > and IncrementalCooperative).
> > > >
> > > 

Re: [ANNOUNCE] New Committer: Chris Egerton

[deng ziming]

Congratulations Chris !

--
Ziming

> On Jul 26, 2022, at 5:01 AM, Matthias J. Sax  wrote:
> 
> Congrats! Well deserved!
> 
> -Matthias
> 
> On 7/25/22 1:08 PM, Bill Bejeck wrote:
>> Congrats Chris!
>> -Bill
>> On Mon, Jul 25, 2022 at 3:58 PM Jorge Esteban Quilcate Otoya <
>> quilcate.jo...@gmail.com> wrote:
>>> Congratulations Chris!
>>> 
>>> On Mon, 25 Jul 2022 at 20:27, Robin Moffatt 
>>> wrote:
>>> 
 Congrats Chris!
 
 
 --
 
 Robin Moffatt | Principal Developer Advocate | ro...@confluent.io |
>>> @rmoff
 
 
 On Mon, 25 Jul 2022 at 17:26, Mickael Maison 
>>> wrote:
 
> Hi all,
> 
> The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> we are excited to announce that he accepted!
> 
> Chris has been contributing to Kafka since 2017. He has made over 80
> commits mostly around Kafka Connect. His most notable contributions
> include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> Exactly-Once Support for Source Connectors.
> 
> He has been an active participant in discussions and reviews on the
> mailing lists and on Github.
> 
> Thanks for all of your contributions Chris. Congratulations!
> 
> -- Mickael, on behalf of the Apache Kafka PMC
> 
 
>>> 

Re: [DISCUSS] KIP-844: Transactional State Stores

[Guozhang Wang]

Hello Alex,

Thanks for the updated KIP, I looked over it and browsed the WIP and just
have a couple meta thoughts:

1) About the param passed into the `recover()` function: it seems to me
that the semantics of "recover(offset)" is: recover this state to a
transaction boundary which is at least the passed-in offset. And the only
possibility that the returned offset is different than the passed-in offset
is that if the previous failure happens after we've done all the commit
procedures except writing the new checkpoint, in which case the returned
offset would be larger than the passed-in offset. Otherwise it should
always be equal to the passed-in offset, is that right?

2) It seems the only use for the "transactional()" function is to determine
if we can update the checkpoint file while in EOS. But the purpose of the
checkpoint file's offsets is just to tell "the local state's current
snapshot's progress is at least the indicated offsets" anyways, and with
this KIP maybe we would just do:

a) when in ALOS, upon failover: we set the starting offset as
checkpointed-offset, then restore() from changelog till the end-offset.
This way we may restore some records twice.
b) when in EOS, upon failover: we first call recover(checkpointed-offset),
then set the starting offset as the returned offset (which may be larger
than checkpointed-offset), then restore until the end-offset.

So why not also:
c) we let the `commit()` function to also return an offset, which indicates
"checkpointable offsets".
d) for existing non-transactional stores, we just have a default
implementation of "commit()" which is simply a flush, and returns a
sentinel value like -1. Then later if we get checkpointable offsets -1, we
do not write the checkpoint. Upon clean shutting down we can just
checkpoint regardless of the returned value from "commit".
e) for existing non-transactional stores, we just have a default
implementation of "recover()" which is to wipe out the local store and
return offset 0 if the passed in offset is -1, otherwise if not -1 then it
indicates a clean shutdown in the last run, can this function is just a
no-op.

In that case, we would not need the "transactional()" function anymore,
since for non-transactional stores their behaviors are still wrapped in the
`commit / recover` function pairs.

I have not completed the thorough pass on your WIP PR, so maybe I could
come up with some more feedback later, but just let me know if my
understanding above is correct or not?


Guozhang




On Thu, Jul 14, 2022 at 7:01 AM Alexander Sorokoumov
 wrote:

> Hi,
>
> I updated the KIP with the following changes:
> * Replaced in-memory batches with the secondary-store approach as the
> default implementation to address the feedback about memory pressure as
> suggested by Sagar and Bruno.
> * Introduced StateStore#commit and StateStore#recover methods as an
> extension of the rollback idea. @Guozhang, please see the comment below on
> why I took a slightly different approach than you suggested.
> * Removed mentions of changes to IQv1 and IQv2. Transactional state stores
> enable reading committed in IQ, but it is really an independent feature
> that deserves its own KIP. Conflating them unnecessarily increases the
> scope for discussion, implementation, and testing in a single unit of work.
>
> I also published a prototype - https://github.com/apache/kafka/pull/12393
> that implements changes described in the proposal.
>
> Regarding explicit rollback, I think it is a powerful idea that allows
> other StateStore implementations to take a different path to the
> transactional behavior rather than keep 2 state stores. Instead of
> introducing a new commit token, I suggest using a changelog offset that
> already 1:1 corresponds to the materialized state. This works nicely
> because Kafka Stream first commits an AK transaction and only then
> checkpoints the state store, so we can use the changelog offset to commit
> the state store transaction.
>
> I called the method StateStore#recover rather than StateStore#rollback
> because a state store might either roll back or forward depending on the
> specific point of the crash failure.Consider the write algorithm in Kafka
> Streams is:
> 1. write stuff to the state store
> 2. producer.sendOffsetsToTransaction(token); producer.commitTransaction();
> 3. flush
> 4. checkpoint
>
> Let's consider 3 cases:
> 1. If the crash failure happens between #2 and #3, the state store rolls
> back and replays the uncommitted transaction from the changelog.
> 2. If the crash failure happens during #3, the state store can roll forward
> and finish the flush/commit.
> 3. If the crash failure happens between #3 and #4, the state store should
> do nothing during recovery and just proceed with the checkpoint.
>
> Looking forward to your feedback,
> Alexander
>
> On Wed, Jun 8, 2022 at 12:16 AM Alexander Sorokoumov <
> asorokou...@confluent.io> wrote:
>
> > Hi,
> >
> > As a status update, I did the following changes to 

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[Guozhang Wang]

Hello Luke,

Thanks for the detailed feedback! I tried to incorporate / answer some of
them inline below.

On Thu, Jul 21, 2022 at 6:39 AM Luke Chen  wrote:

> Hi David,
>
> So excited to see the consumer protocol will be renewed!
> Thanks for David, Guozhang, and Jason for creating this great proposal!
>
> Some comments about the protocol:
> 1. Will the member reconciling jump to the latest target assignment? For
> example, member epoch are A:1, B:2, but now the group is in target
> assignment epoch 4. Will the members jump to the target assignment result
> in epoch 4 directly, or they should first sync to epoch 2, then 3, and 4? I
> tried to find out if there's any problem if we jump to 4, but I can't. So,
> I think we'll go to epoch 4 directly, right?
>
> The short answer is "yes" :) But the a bit longer answer is that, such
reconciliation logic that may skip some epochs are controlled by the
brokers, not the clients. The clients should be simply dumb and takes in
the indicated (delta) assignment as long as its epoch is no smaller than
its current one. The brokers, having full knowledge of where everyone is at
the moment, and the latest target assignment, would decide which delta
assignment to send, step by step, and which assignment could be skipped.


> 2. For the ConsumerGroupPrepareAssignment and
> ConsumerGroupInstallAssignment API, when member epoch doesn't match, it'll
> retry (and hope) in next heartbeat response, the member epoch will be
> bumped. But in ConsumerGroupHeartbeat, Upon receiving the
> FENCED_MEMBER_EPOCH error, the consumer abandon all its partitions and
> rejoins with the same member id and the epoch 0, unless the partitions
> owned by the members are a subset of the target partitions. I'm wondering
> why can't we return the latest member epoch to the member in
> ConsumerGroupHeartbeat response no matter if the owned partitions are
> subset of target partitions or not. After all, the member is still one of
> the group member, the assignment for the member is still valid, right? This
> way, the ConsumerGroupPrepareAssignment and ConsumerGroupInstallAssignment
> API can piggyback it to retry in next heartbeat response. WDYT?
>
> The semantics of these two classes of requests are different: for
ConsumerGroupPrepareAssignment / ConsumerGroupInstallAssignment it's not
really indicating the consumer is actually "fenced", but only to indicate
"your information is probably stale, so just stay put and wait for the
renewed info". Note that these two requests are only for clients to try
collecting information or making a change to the group assignment, so
even fai that request does not mean the member is no longer part of the
group. BTW that reminds me for these two requests maybe we should use a
different error code than FENCED_MEMBER_EPOCH since it's a bit confusing,
maybe "STALE_MEMBER_EPOCH"?

For ConsumerGroupHeartbeat, the fencing purpose is only to make sure that
any partitions that are assigned to others should not be retained at the
"fenced member". So I think we can make the assignment response in
ConsumerGroupHeartbeat idempotent, and only return the fatal error code if
the consumer's "owned partitions" has any overlaps with other members'
current assignment, and otherwise we just proceed with the new assignment.



> 3. When receiving COMPUTE_ASSIGNMENT error in ConsumerGroupHeartbeat API,
> the consumer starts the assignment process. But what if somehow the member
> didn't send out ConsumerGroupPrepareAssignment request, what would we do? I
> think we'll wait for rebalance timeout and kick the member out of the
> group, right? And then, what will the group coordinator do? I think it'll
> select another member to be the leader and return COMPUTE_ASSIGNMENT error
> in that member's heartbeat response, right? Maybe we should add that into
> KIP.
>
The broker has a timer for this process, and if it did not receive the
assignment (i.e. it is until the ConsumerGroupInstallAssignment is
received; note that for some assignor they may not need to always refresh
their information via ConsumerGroupPrepareAssignment upon computing the new
assignment) in time, then the broker would pick another member and ask it
to COMPUTE_ASSIGNMENT.

But, we may not try to kick the member out of the group based on that
timer, but only based on the heartbeat timer. I.e. let's say there's one
member who keeps heartbeating but never want to compute the new assignment
even when being asked, it would still be remained in the group.

We can clarify this a bit more in the doc.

>
> Some typos:
> 4. In "Assignment Process" section:
> If the selected assignor does exist, the group coordinator will reject the
> heartbeat with an UNSUPPORTED_ASSIGNOR error.
> -> I think it should be "does not exist"
>
> Ack!


> 5. In "JoinGroup Handling" section:
> If the member has revoked all its partitions or the required partitions,
> the member can transition to its next epoch. The current assignment become
> the current 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #16

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 568745 lines...]
[2022-07-25T23:49:29.112Z] > Task :connect:api:testJar
[2022-07-25T23:49:29.112Z] > Task :connect:api:testSrcJar
[2022-07-25T23:49:29.112Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-07-25T23:49:29.112Z] > Task :connect:json:publishToMavenLocal
[2022-07-25T23:49:29.112Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-07-25T23:49:29.112Z] > Task :connect:api:publishToMavenLocal
[2022-07-25T23:49:31.535Z] 
[2022-07-25T23:49:31.535Z] > Task :streams:javadoc
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:854:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-07-25T23:49:31.535Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-07-25T23:49:31.535Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.3_2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-07-25T23:49:31.535Z] 

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[Guozhang Wang]

Hello Sagar,

It would be great if you could come back with some analysis on how to
implement the Connect side integration with the new protocol; so far
besides leveraging on the new "protocol type" we did not yet think through
the Connect side implementations. For Streams here's a draft of integration
plan:
https://docs.google.com/document/d/17PNz2sGoIvGyIzz8vLyJTJTU2rqnD_D9uHJnH9XARjU/edit#heading=h.pdgirmi57dvn
just FYI for your analysis on Connect.

On Tue, Jul 19, 2022 at 10:48 PM Sagar  wrote:

> Hi David,
>
> Thank you for your response. The reason I thought connect can also fit into
> this new scheme is that even today the connect uses a WorkerCoordinator
> extending from AbstractCoordinator to empower rebalances of
> tasks/connectors. The WorkerCoordinator sets the protocolType() to connect
> and uses the metadata() method by plumbing into JoinGroupRequestProtocol.
>
> I think the changes to support connect would be similar at a high level to
> the changes in streams mainly because of the Client side assignors being
> used in both. At an implementation level, we might need to make a lot of
> changes to get onto this new assignment protocol like enhancing the
> JoinGroup request/response and SyncGroup and using ConsumerGroupHeartbeat
> API etc again on similar lines to streams (or there might be deviations). I
> would try to perform a detailed analysis of the same  and we can have a
> separate discussion thread for that as that would derail this discussion
> thread. Let me know if that sounds good to you.
>
> Thanks!
> Sagar.
>
>
>
> On Fri, Jul 15, 2022 at 5:47 PM David Jacot 
> wrote:
>
> > Hi Sagar,
> >
> > Thanks for your comments.
> >
> > 1) Yes. That refers to `Assignment#error`. Sure, I can mention it.
> >
> > 2) The idea is to transition C from his current assignment to his
> > target assignment when he can move to epoch 3. When that happens, the
> > member assignment is updated and persisted with all its assigned
> > partitions even if they are not all revoked yet. In other words, the
> > member assignment becomes the target assignment. This is basically an
> > optimization to avoid having to write all the changes to the log. The
> > examples are based on the persisted state so I understand the
> > confusion. Let me see if I can improve this in the description.
> >
> > 3) Regarding Connect, it could reuse the protocol with a client side
> > assignor if it fits in the protocol. The assignment is about
> > topicid-partitions + metadata, could Connect fit into this?
> >
> > Best,
> > David
> >
> > On Fri, Jul 15, 2022 at 1:55 PM Sagar  wrote:
> > >
> > > Hi David,
> > >
> > > Thanks for the KIP. I just had minor observations:
> > >
> > > 1) In the Assignment Error section in Client Side mode Assignment
> > process,
> > > you mentioned => `In this case, the client side assignor can return an
> > > error to the group coordinator`. In this case are you referring to the
> > > Assignor returning an AssignmentError that's listed down towards the
> end?
> > > If yes, do you think it would make sense to mention this explicitly
> here?
> > >
> > > 2) In the Case Studies section, I have a slight confusion, not sure if
> > > others have the same. Consider this step:
> > >
> > > When B heartbeats, the group coordinator transitions him to epoch 3
> > because
> > > B has no partitions to revoke. It persists the change and reply.
> > >
> > >- Group (epoch=3)
> > >   - A
> > >   - B
> > >   - C
> > >- Target Assignment (epoch=3)
> > >   - A - partitions=[foo-0]
> > >   - B - partitions=[foo-2]
> > >   - C - partitions=[foo-1]
> > >- Member Assignment
> > >   - A - epoch=2, partitions=[foo-0, foo-1]
> > >   - B - epoch=3, partitions=[foo-2]
> > >   - C - epoch=3, partitions=[foo-1]
> > >
> > > When C heartbeats, it transitions to epoch 3 but cannot get foo-1 yet.
> > >
> > > Here,it's mentioned that member C can't get the foo-1 partition yet,
> but
> > > based on the description above, it seems it already has it. Do you
> think
> > it
> > > would be better to remove it and populate it only when it actually gets
> > it?
> > > I see this in a lot of other places, so have I understood it
> incorrectly
> > ?
> > >
> > >
> > > Regarding connect , it might be out of scope of this discussion, but
> from
> > > what I understood it would probably be running in client side assignor
> > mode
> > > even on the new rebalance protocol as it has its own Custom
> > Assignors(Eager
> > > and IncrementalCooperative).
> > >
> > > Thanks!
> > >
> > > Sagar.
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Jul 15, 2022 at 5:00 PM David Jacot
>  > >
> > > wrote:
> > >
> > > > Thanks Hector! Our goal is to move forward with specialized API
> > > > instead of relying on one generic API. For Connect, we can apply the
> > > > exact same pattern and reuse/share the core implementation on the
> > > > server side. For the schema registry, I think that we should consider
> > > > having a 

[jira] [Created] (KAFKA-14105) Remove quorum.all_non_upgrade for system tests

[Ron Dagostino (Jira)]

Ron Dagostino created KAFKA-14105:
-

 Summary: Remove quorum.all_non_upgrade for system tests
 Key: KAFKA-14105
 URL: https://issues.apache.org/jira/browse/KAFKA-14105
 Project: Kafka
  Issue Type: Task
  Components: kraft, system tests
Reporter: Ron Dagostino


We defined `all_non_upgrade = [zk, remote_kraft]` in `quorum.py` to encapsulate 
the quorum(s) that we want system tests to generally run with when they are 
unrelated to upgrading.  The idea was that we would just annotate tests with 
that and then we would be able to change the definition of it as we move 
through and beyond the KRaft bridge release.  But it is confusing, and 
search-and-replace is cheap -- especially if we are only doing it once or twice 
over the course of the project.  So we should eliminate the definition of 
`quorum.all_non_upgrade` (which was intended to be mutable over the course of 
the project) in favor of something like `zk_and_remote_kraft`, which will 
forever list ZK and REMOTE_KRAFT.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Committer: Chris Egerton

[Matthias J. Sax]

Congrats! Well deserved!

-Matthias

On 7/25/22 1:08 PM, Bill Bejeck wrote:

Congrats Chris!

-Bill

On Mon, Jul 25, 2022 at 3:58 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:


Congratulations Chris!

On Mon, 25 Jul 2022 at 20:27, Robin Moffatt 
wrote:


Congrats Chris!


--

Robin Moffatt | Principal Developer Advocate | ro...@confluent.io |

@rmoff



On Mon, 25 Jul 2022 at 17:26, Mickael Maison 

wrote:



Hi all,

The PMC for Apache Kafka has invited Chris Egerton as a committer, and
we are excited to announce that he accepted!

Chris has been contributing to Kafka since 2017. He has made over 80
commits mostly around Kafka Connect. His most notable contributions
include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
Exactly-Once Support for Source Connectors.

He has been an active participant in discussions and reviews on the
mailing lists and on Github.

Thanks for all of your contributions Chris. Congratulations!

-- Mickael, on behalf of the Apache Kafka PMC

[jira] [Created] (KAFKA-14104) Perform CRC validation on KRaft Batch Records and Snapshots

[Niket Goel (Jira)]

Niket Goel created KAFKA-14104:
--

 Summary: Perform CRC validation on KRaft Batch Records and 
Snapshots
 Key: KAFKA-14104
 URL: https://issues.apache.org/jira/browse/KAFKA-14104
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.2.0
Reporter: Niket Goel
 Fix For: 3.3


Today we stamp the BatchRecord header with a CRC [1] and verify that CRC before 
the log is written to disk [2]. The CRC checks should also be verified when the 
records are read back from disk. The same procedure should be followed for 
KRaft snapshots as well.

[1] 
[https://github.com/apache/kafka/blob/6b76c01cf895db0651e2cdcc07c2c392f00a8ceb/clients/src/main/java/org/apache/kafka/common/record/DefaultRecordBatch.java#L501=]
 

[2] 
[https://github.com/apache/kafka/blob/679e9e0cee67e7d3d2ece204a421ea7da31d73e9/core/src/main/scala/kafka/log/UnifiedLog.scala#L1143]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14078) Replica fetches to follower should return NOT_LEADER error

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-14078.
-
Resolution: Fixed

> Replica fetches to follower should return NOT_LEADER error
> --
>
> Key: KAFKA-14078
> URL: https://issues.apache.org/jira/browse/KAFKA-14078
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.3.0
>
>
> After the fix for KAFKA-13837, if a follower receives a request from another 
> replica, it will return UNKNOWN_LEADER_EPOCH even if the leader epoch 
> matches. We need to do epoch leader/epoch validation first before we check 
> whether we have a valid replica.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Committer: Chris Egerton

[Bill Bejeck]

Congrats Chris!

-Bill

On Mon, Jul 25, 2022 at 3:58 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Congratulations Chris!
>
> On Mon, 25 Jul 2022 at 20:27, Robin Moffatt 
> wrote:
>
> > Congrats Chris!
> >
> >
> > --
> >
> > Robin Moffatt | Principal Developer Advocate | ro...@confluent.io |
> @rmoff
> >
> >
> > On Mon, 25 Jul 2022 at 17:26, Mickael Maison 
> wrote:
> >
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> > > we are excited to announce that he accepted!
> > >
> > > Chris has been contributing to Kafka since 2017. He has made over 80
> > > commits mostly around Kafka Connect. His most notable contributions
> > > include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> > > Exactly-Once Support for Source Connectors.
> > >
> > > He has been an active participant in discussions and reviews on the
> > > mailing lists and on Github.
> > >
> > > Thanks for all of your contributions Chris. Congratulations!
> > >
> > > -- Mickael, on behalf of the Apache Kafka PMC
> > >
> >
>

Re: [ANNOUNCE] New Committer: Chris Egerton

[Jorge Esteban Quilcate Otoya]

Congratulations Chris!

On Mon, 25 Jul 2022 at 20:27, Robin Moffatt 
wrote:

> Congrats Chris!
>
>
> --
>
> Robin Moffatt | Principal Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Mon, 25 Jul 2022 at 17:26, Mickael Maison  wrote:
>
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> > we are excited to announce that he accepted!
> >
> > Chris has been contributing to Kafka since 2017. He has made over 80
> > commits mostly around Kafka Connect. His most notable contributions
> > include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> > Exactly-Once Support for Source Connectors.
> >
> > He has been an active participant in discussions and reviews on the
> > mailing lists and on Github.
> >
> > Thanks for all of your contributions Chris. Congratulations!
> >
> > -- Mickael, on behalf of the Apache Kafka PMC
> >
>

[GitHub] [kafka-site] C0urante commented on pull request #428: Add Chris Egerton to committers

[GitBox]

C0urante commented on PR #428:
URL: https://github.com/apache/kafka-site/pull/428#issuecomment-1194548517

   Thanks Bill!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] C0urante merged pull request #428: Add Chris Egerton to committers

[GitBox]

C0urante merged PR #428:
URL: https://github.com/apache/kafka-site/pull/428


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #427: KAFKA-13868: Replace YouTube embedded video with hyperlinks

[GitBox]

bbejeck commented on PR #427:
URL: https://github.com/apache/kafka-site/pull/427#issuecomment-1194536588

   Thanks for this fix @divijvaidya 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck merged pull request #427: KAFKA-13868: Replace YouTube embedded video with hyperlinks

[GitBox]

bbejeck merged PR #427:
URL: https://github.com/apache/kafka-site/pull/427


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] New Committer: Chris Egerton

[Robin Moffatt]

Congrats Chris!


-- 

Robin Moffatt | Principal Developer Advocate | ro...@confluent.io | @rmoff


On Mon, 25 Jul 2022 at 17:26, Mickael Maison  wrote:

> Hi all,
>
> The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> we are excited to announce that he accepted!
>
> Chris has been contributing to Kafka since 2017. He has made over 80
> commits mostly around Kafka Connect. His most notable contributions
> include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> Exactly-Once Support for Source Connectors.
>
> He has been an active participant in discussions and reviews on the
> mailing lists and on Github.
>
> Thanks for all of your contributions Chris. Congratulations!
>
> -- Mickael, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Chris Egerton

[Guozhang Wang]

Congratulations Chris!

Guozhang

On Mon, Jul 25, 2022 at 10:45 AM Tom Bentley  wrote:

> Congratulations Chris!
>
> On Mon, 25 Jul 2022 at 18:40, Jeremy Custenborder  >
> wrote:
>
> > Congrats Chris!
> >
> > On Mon, Jul 25, 2022 at 12:07 PM Sagar 
> wrote:
> > >
> > > Congratulations Chris !
> > >
> > > On Mon, 25 Jul 2022 at 10:32 PM, Viktor Somogyi-Vass
> > >  wrote:
> > >
> > > > Congrats Chris!
> > > >
> > > > On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
> > > >  wrote:
> > > >
> > > > > Congratulations!
> > > > >
> > > > > --
> > > > > Matthew de Detrich
> > > > > Aiven Deutschland GmbH
> > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > m: +491603708037
> > > > > w: aiven.io e: matthew.dedetr...@aiven.io
> > > > > On 25. Jul 2022, 18:26 +0200, Mickael Maison  >,
> > > > wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > The PMC for Apache Kafka has invited Chris Egerton as a
> committer,
> > and
> > > > > > we are excited to announce that he accepted!
> > > > > >
> > > > > > Chris has been contributing to Kafka since 2017. He has made over
> > 80
> > > > > > commits mostly around Kafka Connect. His most notable
> contributions
> > > > > > include KIP-507: Securing Internal Connect REST Endpoints and
> > KIP-618:
> > > > > > Exactly-Once Support for Source Connectors.
> > > > > >
> > > > > > He has been an active participant in discussions and reviews on
> the
> > > > > > mailing lists and on Github.
> > > > > >
> > > > > > Thanks for all of your contributions Chris. Congratulations!
> > > > > >
> > > > > > -- Mickael, on behalf of the Apache Kafka PMC
> > > > >
> > > >
> >
> >
>


-- 
-- Guozhang

Re: [ANNOUNCE] New Committer: Chris Egerton

[Tom Bentley]

Congratulations Chris!

On Mon, 25 Jul 2022 at 18:40, Jeremy Custenborder 
wrote:

> Congrats Chris!
>
> On Mon, Jul 25, 2022 at 12:07 PM Sagar  wrote:
> >
> > Congratulations Chris !
> >
> > On Mon, 25 Jul 2022 at 10:32 PM, Viktor Somogyi-Vass
> >  wrote:
> >
> > > Congrats Chris!
> > >
> > > On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
> > >  wrote:
> > >
> > > > Congratulations!
> > > >
> > > > --
> > > > Matthew de Detrich
> > > > Aiven Deutschland GmbH
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > m: +491603708037
> > > > w: aiven.io e: matthew.dedetr...@aiven.io
> > > > On 25. Jul 2022, 18:26 +0200, Mickael Maison ,
> > > wrote:
> > > > > Hi all,
> > > > >
> > > > > The PMC for Apache Kafka has invited Chris Egerton as a committer,
> and
> > > > > we are excited to announce that he accepted!
> > > > >
> > > > > Chris has been contributing to Kafka since 2017. He has made over
> 80
> > > > > commits mostly around Kafka Connect. His most notable contributions
> > > > > include KIP-507: Securing Internal Connect REST Endpoints and
> KIP-618:
> > > > > Exactly-Once Support for Source Connectors.
> > > > >
> > > > > He has been an active participant in discussions and reviews on the
> > > > > mailing lists and on Github.
> > > > >
> > > > > Thanks for all of your contributions Chris. Congratulations!
> > > > >
> > > > > -- Mickael, on behalf of the Apache Kafka PMC
> > > >
> > >
>
>

Re: [ANNOUNCE] New Committer: Chris Egerton

[Randall Hauch]

Congratulations, Chris!

On Mon, Jul 25, 2022 at 12:31 PM Jeremy Custenborder <
jcustenbor...@gmail.com> wrote:

> Congrats Chris!
>
> On Mon, Jul 25, 2022 at 12:07 PM Sagar  wrote:
> >
> > Congratulations Chris !
> >
> > On Mon, 25 Jul 2022 at 10:32 PM, Viktor Somogyi-Vass
> >  wrote:
> >
> > > Congrats Chris!
> > >
> > > On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
> > >  wrote:
> > >
> > > > Congratulations!
> > > >
> > > > --
> > > > Matthew de Detrich
> > > > Aiven Deutschland GmbH
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > m: +491603708037
> > > > w: aiven.io e: matthew.dedetr...@aiven.io
> > > > On 25. Jul 2022, 18:26 +0200, Mickael Maison ,
> > > wrote:
> > > > > Hi all,
> > > > >
> > > > > The PMC for Apache Kafka has invited Chris Egerton as a committer,
> and
> > > > > we are excited to announce that he accepted!
> > > > >
> > > > > Chris has been contributing to Kafka since 2017. He has made over
> 80
> > > > > commits mostly around Kafka Connect. His most notable contributions
> > > > > include KIP-507: Securing Internal Connect REST Endpoints and
> KIP-618:
> > > > > Exactly-Once Support for Source Connectors.
> > > > >
> > > > > He has been an active participant in discussions and reviews on the
> > > > > mailing lists and on Github.
> > > > >
> > > > > Thanks for all of your contributions Chris. Congratulations!
> > > > >
> > > > > -- Mickael, on behalf of the Apache Kafka PMC
> > > >
> > >
>

[GitHub] [kafka-site] divijvaidya commented on pull request #427: KAFKA-13868: Replace YouTube embedded video with hyperlinks

[GitBox]

divijvaidya commented on PR #427:
URL: https://github.com/apache/kafka-site/pull/427#issuecomment-1194393568

   @bbejeck perhaps you would like to review this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] divijvaidya opened a new pull request, #427: KAFKA-13868: Replace YouTube embedded video with hyperlinks

[GitBox]

divijvaidya opened a new pull request, #427:
URL: https://github.com/apache/kafka-site/pull/427

   As per the [discussion in the 
community](https://lists.apache.org/thread/p24xvbf8nkvxpbj668vc0g3x3lojsnk4), 
we want to replace the embedded YouTube videos with hyperlinks to satisfy the 
[ASF privacy policy](https://privacy.apache.org/faq/committers.html).
   
   This code change replaces the embedded videos from two of the pages on the 
website with hyperlinks.
   
   **Before**
   ![Screenshot 2022-07-25 at 19 31 
50](https://user-images.githubusercontent.com/71267/180839003-da0f967c-019b-449e-a7c3-3bbac37611dd.png)
   ![Screenshot 2022-07-25 at 19 32 
13](https://user-images.githubusercontent.com/71267/180839020-b37a118d-b8ca-480e-8832-bd19b29cfbdd.png)
   
   **After**
   ![Screenshot 2022-07-25 at 19 32 
01](https://user-images.githubusercontent.com/71267/180839040-591e67df-8053-4633-9c35-52d7fd32fd0c.png)
   ![Screenshot 2022-07-25 at 19 32 
23](https://user-images.githubusercontent.com/71267/180839061-355b3e06-1e9c-40da-89b0-aefd95ee5be5.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] New Committer: Chris Egerton

[Jeremy Custenborder]

Congrats Chris!

On Mon, Jul 25, 2022 at 12:07 PM Sagar  wrote:
>
> Congratulations Chris !
>
> On Mon, 25 Jul 2022 at 10:32 PM, Viktor Somogyi-Vass
>  wrote:
>
> > Congrats Chris!
> >
> > On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
> >  wrote:
> >
> > > Congratulations!
> > >
> > > --
> > > Matthew de Detrich
> > > Aiven Deutschland GmbH
> > > Immanuelkirchstraße 26, 10405 Berlin
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > m: +491603708037
> > > w: aiven.io e: matthew.dedetr...@aiven.io
> > > On 25. Jul 2022, 18:26 +0200, Mickael Maison ,
> > wrote:
> > > > Hi all,
> > > >
> > > > The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> > > > we are excited to announce that he accepted!
> > > >
> > > > Chris has been contributing to Kafka since 2017. He has made over 80
> > > > commits mostly around Kafka Connect. His most notable contributions
> > > > include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> > > > Exactly-Once Support for Source Connectors.
> > > >
> > > > He has been an active participant in discussions and reviews on the
> > > > mailing lists and on Github.
> > > >
> > > > Thanks for all of your contributions Chris. Congratulations!
> > > >
> > > > -- Mickael, on behalf of the Apache Kafka PMC
> > >
> >

Re: [ANNOUNCE] New Committer: Chris Egerton

[Israel Ekpo]

Congrats Chris.

Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/


On Mon, Jul 25, 2022 at 12:26 PM Mickael Maison  wrote:

> Hi all,
>
> The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> we are excited to announce that he accepted!
>
> Chris has been contributing to Kafka since 2017. He has made over 80
> commits mostly around Kafka Connect. His most notable contributions
> include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> Exactly-Once Support for Source Connectors.
>
> He has been an active participant in discussions and reviews on the
> mailing lists and on Github.
>
> Thanks for all of your contributions Chris. Congratulations!
>
> -- Mickael, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New Committer: Chris Egerton

[Sagar]

Congratulations Chris !

On Mon, 25 Jul 2022 at 10:32 PM, Viktor Somogyi-Vass
 wrote:

> Congrats Chris!
>
> On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
>  wrote:
>
> > Congratulations!
> >
> > --
> > Matthew de Detrich
> > Aiven Deutschland GmbH
> > Immanuelkirchstraße 26, 10405 Berlin
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > m: +491603708037
> > w: aiven.io e: matthew.dedetr...@aiven.io
> > On 25. Jul 2022, 18:26 +0200, Mickael Maison ,
> wrote:
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> > > we are excited to announce that he accepted!
> > >
> > > Chris has been contributing to Kafka since 2017. He has made over 80
> > > commits mostly around Kafka Connect. His most notable contributions
> > > include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> > > Exactly-Once Support for Source Connectors.
> > >
> > > He has been an active participant in discussions and reviews on the
> > > mailing lists and on Github.
> > >
> > > Thanks for all of your contributions Chris. Congratulations!
> > >
> > > -- Mickael, on behalf of the Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New Committer: Chris Egerton

[Viktor Somogyi-Vass]

Congrats Chris!

On Mon, Jul 25, 2022, 18:33 Matthew Benedict de Detrich
 wrote:

> Congratulations!
>
> --
> Matthew de Detrich
> Aiven Deutschland GmbH
> Immanuelkirchstraße 26, 10405 Berlin
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> m: +491603708037
> w: aiven.io e: matthew.dedetr...@aiven.io
> On 25. Jul 2022, 18:26 +0200, Mickael Maison , wrote:
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> > we are excited to announce that he accepted!
> >
> > Chris has been contributing to Kafka since 2017. He has made over 80
> > commits mostly around Kafka Connect. His most notable contributions
> > include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> > Exactly-Once Support for Source Connectors.
> >
> > He has been an active participant in discussions and reviews on the
> > mailing lists and on Github.
> >
> > Thanks for all of your contributions Chris. Congratulations!
> >
> > -- Mickael, on behalf of the Apache Kafka PMC
>

Re: [DISCUSS] KIP-853: KRaft Voters Change

[José Armando García Sancio]

Thanks for the feedback Colin. I updated the KIP with your suggestions
and replied to your comments below.

Colin McCabe wrote:
> 1. It seems like the proposal is to have a UUID per partition directory on 
> the voter. If I understand correctly, this is sometimes referred to as 
> "VoterUUID" and sometimes as "ReplicaUUID." The latter seems more accurate, 
> since a single voter could have multiple of these IDs, in a situation where 
> we had multiple Raft topics. So it would be good to standardize on that. 
> Also, I didn't see a description of how this would be stored in the log 
> directory. That would be good to add.

Conceptually, these are all replica UUID. In the protocol (Kafka
message) we use different field names depending on the context. This
KIP and KIP-595 uses the prefixes "Candidate", "Replica", "Voter" and
"Voted" depending on the context. I went through the KIP and made some
changes to make this consistent. Diff:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=217391519=24=23

> 2. When we originally did the Raft and Quorum Controller KIPs, one 
> contentious topic was node IDs. We eventually settled on the idea that broker 
> and controller IDs were in the same ID space. So you can't (for example) have 
> a broker 3 that is in a separate JVM from controller 3. This is pretty easy 
> to enforce with a static configuration, but it seems like it will be harder 
> to do dynamically.
>
> I would like to keep this invariant. This probably requires us to reject 
> attempts to add a new quorum voter which duplicates a broker ID (except in 
> the special case of co-location!) Similarly, we should reject broker 
> registrations that duplicate an unrelated controller ID. The broker's 
> incarnation ID is the key to doing this, I think. But that requires us to 
> send the incarnation ID in many of these RPCs.
>

I think there are two cases that we want to protect.
1. The admin attempts to add a voter that is a broker using the
AddVoter RPC. The KIP protects against this case by only having
controllers (or replicas that support being voters) send the
ReplicaUuid to the cluster metadata leader. Since the AddVoter RPC
requires both ID and UUID to be specified, brokers cannot be added as
voters. I thought about adding another field to the Fetch and
FetchSnapshot request to communicate to the leader if the sending
replica supports becoming a voter but decided against it to not
increase the already large number of fields in the Fetch request.

2. The admin starts a broker that has the same ID as a voter. Kafka
currently protects against this by validating the broker configuration
and fails the validation of the node is a broker and has an id that is
enumerated in controller.quorum.voters. When using dynamic
configuration the controller.quorum.voters configuration will be
empty. In the implementation we can still protect against this case by
passing enough information to the KafkaRaftClient. For example,if we
pass the boolean "canBeVoter" to the KRaft implement then check that
its replica id is not in the voter set. If it is then it will shut
down.

> 3. Is it really necessary to put the endpoint information into the 
> AddVoterRecord? It seems like that could be figured out at runtime, like we 
> do today. If we do need it, it seems particularly weird for it to be 
> per-partition (will we have a separate TCP port for each Raft partition?) I 
> also don't know why we'd want multiple endpoints. We have that for the broker 
> because the endpoints have different uses, but that isn't the case here.

Today, voters/controllers use the hosts and ports specified in
controller.quorum.voters to establish a leader. Brokers use the hosts
and ports in controller.quorum.voters to discover the leaders. The
requirements are the following:
1. Voters/controllers need to know the endpoint of all of the voters.
This is required because at a minimum the voters need to send a Vote
request to the majority of the voters to establish leadership.
2. Brokers need to discover the leader from a list of nodes.

The endpoints in AddVoterRecord are so that the voters can discover
each other. The configuration controllers.quorum.bootstrap.servers is
so that observers (brokers and new voters) which are not required to
have replicated the voters and AddVoterRecord can discover the leader.

I noticed that I forgot to add endpoint information in all of the RPCs
that return the current leader. This is needed because it is not
guaranteed that the replica will have an AddVoterRecord for the leader
id returned in the response.

> The original rationale for multiple endpoints on the controllers was to 
> support migration from PLAINTEXT to SSL (or whatever). But that only requires 
> multiple listeners to be active on the receive side, not send side. A single 
> voter never needs more than one endpoint to contact a peer.

I agree. I removed the array ([]) and made it a single endpoint.

> Overall, I think we'd be 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #15

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 568652 lines...]
[2022-07-25T16:20:27.180Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-5b86ccfc-d1f5-46af-aa76-057289955875-GlobalStreamThread"
[2022-07-25T16:20:28.283Z] 
[2022-07-25T16:20:28.283Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutDownClientIfGlobalStreamThreadWantsToReplaceThread PASSED
[2022-07-25T16:20:28.283Z] 
[2022-07-25T16:20:28.283Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceSingleThread STARTED
[2022-07-25T16:20:31.903Z] 
[2022-07-25T16:20:31.903Z] Exception: java.lang.AssertionError thrown from the 
UncaughtExceptionHandler in thread 
"appId_StreamsUncaughtExceptionHandlerIntegrationTestnull-81db38c2-ff15-4516-a91b-b559356660a6-StreamThread-1"
[2022-07-25T16:20:33.005Z] 
[2022-07-25T16:20:33.005Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldReplaceSingleThread PASSED
[2022-07-25T16:20:33.005Z] 
[2022-07-25T16:20:33.005Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownMultipleThreadApplication STARTED
[2022-07-25T16:20:57.787Z] 
[2022-07-25T16:20:57.787Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownMultipleThreadApplication PASSED
[2022-07-25T16:20:57.787Z] 
[2022-07-25T16:20:57.787Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClient STARTED
[2022-07-25T16:20:59.831Z] 
[2022-07-25T16:20:59.831Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownClient PASSED
[2022-07-25T16:20:59.831Z] 
[2022-07-25T16:20:59.831Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownSingleThreadApplication STARTED
[2022-07-25T16:21:11.739Z] 
[2022-07-25T16:21:11.739Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldShutdownSingleThreadApplication PASSED
[2022-07-25T16:21:11.739Z] 
[2022-07-25T16:21:11.739Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldEmitSameRecordAfterFailover STARTED
[2022-07-25T16:21:23.807Z] 
[2022-07-25T16:21:23.807Z] 
org.apache.kafka.streams.integration.StreamsUncaughtExceptionHandlerIntegrationTest
 > shouldEmitSameRecordAfterFailover PASSED
[2022-07-25T16:21:28.174Z] 
[2022-07-25T16:21:28.174Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[at_least_once] STARTED
[2022-07-25T16:22:29.241Z] 
[2022-07-25T16:22:29.242Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[at_least_once] PASSED
[2022-07-25T16:22:29.242Z] 
[2022-07-25T16:22:29.242Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once] STARTED
[2022-07-25T16:23:30.510Z] 
[2022-07-25T16:23:30.511Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once] PASSED
[2022-07-25T16:23:30.511Z] 
[2022-07-25T16:23:30.511Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once_v2] STARTED
[2022-07-25T16:24:31.602Z] 
[2022-07-25T16:24:31.602Z] 
org.apache.kafka.streams.integration.SuppressionDurabilityIntegrationTest > 
shouldRecoverBufferAfterShutdown[exactly_once_v2] PASSED
[2022-07-25T16:24:31.602Z] 
[2022-07-25T16:24:31.602Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] STARTED
[2022-07-25T16:24:50.060Z] 
[2022-07-25T16:24:50.060Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] PASSED
[2022-07-25T16:24:50.060Z] 
[2022-07-25T16:24:50.060Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] STARTED
[2022-07-25T16:25:11.236Z] 
[2022-07-25T16:25:11.236Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] PASSED
[2022-07-25T16:25:11.236Z] 
[2022-07-25T16:25:11.236Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] STARTED
[2022-07-25T16:25:32.803Z] 
[2022-07-25T16:25:32.803Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] PASSED
[2022-07-25T16:25:32.803Z] 
[2022-07-25T16:25:32.803Z] 
org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching 

Re: [ANNOUNCE] New Committer: Chris Egerton

[Matthew Benedict de Detrich]

Congratulations!

--
Matthew de Detrich
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491603708037
w: aiven.io e: matthew.dedetr...@aiven.io
On 25. Jul 2022, 18:26 +0200, Mickael Maison , wrote:
> Hi all,
>
> The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> we are excited to announce that he accepted!
>
> Chris has been contributing to Kafka since 2017. He has made over 80
> commits mostly around Kafka Connect. His most notable contributions
> include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> Exactly-Once Support for Source Connectors.
>
> He has been an active participant in discussions and reviews on the
> mailing lists and on Github.
>
> Thanks for all of your contributions Chris. Congratulations!
>
> -- Mickael, on behalf of the Apache Kafka PMC

Re: [ANNOUNCE] New Committer: Chris Egerton

[Josep Prat]

Congrats Chris!

———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Mon, Jul 25, 2022, 18:26 Mickael Maison  wrote:

> Hi all,
>
> The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> we are excited to announce that he accepted!
>
> Chris has been contributing to Kafka since 2017. He has made over 80
> commits mostly around Kafka Connect. His most notable contributions
> include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> Exactly-Once Support for Source Connectors.
>
> He has been an active participant in discussions and reviews on the
> mailing lists and on Github.
>
> Thanks for all of your contributions Chris. Congratulations!
>
> -- Mickael, on behalf of the Apache Kafka PMC
>

[ANNOUNCE] New Committer: Chris Egerton

[Mickael Maison]

Hi all,

The PMC for Apache Kafka has invited Chris Egerton as a committer, and
we are excited to announce that he accepted!

Chris has been contributing to Kafka since 2017. He has made over 80
commits mostly around Kafka Connect. His most notable contributions
include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
Exactly-Once Support for Source Connectors.

He has been an active participant in discussions and reviews on the
mailing lists and on Github.

Thanks for all of your contributions Chris. Congratulations!

-- Mickael, on behalf of the Apache Kafka PMC

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Justine Olshan]

Hey Bill,
Thanks! I was just going to say that hopefully
transactional.id.expiration.ms would also be over the delivery timeout. :)
Thanks for the +1!

Justine

On Mon, Jul 25, 2022 at 9:17 AM Bill Bejeck  wrote:

> Hi Justine,
>
> I just took another look at the KIP, and I realize my question/suggestion
> about default values has already been addressed in the `Compatibility`
> section.
>
> I'm +1 on the KIP.
>
> -Bill
>
> On Thu, Jul 21, 2022 at 6:20 PM Bill Bejeck  wrote:
>
> > Hi Justine,
> >
> > Thanks for the well written KIP, this looks like it will be a useful
> > addition.
> >
> > Overall the KIP looks good to me, I have one question/comment.
> >
> > You mentioned that setting the `producer.id.expiration.ms` less than the
> > delivery timeout could lead to duplicates, which makes sense.  To help
> > avoid this situation, do we want to consider a default value that is the
> > same as the delivery timeout?
> >
> > Thanks again for the KIP.
> >
> > Bill
> >
> > On Thu, Jul 21, 2022 at 4:54 PM Justine Olshan
> >  wrote:
> >
> >> Hey all!
> >>
> >> I'd like to start a discussion on my proposal to separate time-based
> >> producer ID expiration from transactional ID expiration by introducing a
> >> new configuration.
> >>
> >> The KIP Is pretty small and simple, but will be helpful in controlling
> >> memory usage in brokers -- especially now that by default producers are
> >> idempotent and create producer ID state.
> >>
> >> Please take a look and leave any comments you may have!
> >>
> >> KIP:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-854+Separate+configuration+for+producer+ID+expiry
> >> JIRA: https://issues.apache.org/jira/browse/KAFKA-14097
> >>
> >> Thanks!
> >> Justine
> >>
> >
>

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Bill Bejeck]

Hi Justine,

I just took another look at the KIP, and I realize my question/suggestion
about default values has already been addressed in the `Compatibility`
section.

I'm +1 on the KIP.

-Bill

On Thu, Jul 21, 2022 at 6:20 PM Bill Bejeck  wrote:

> Hi Justine,
>
> Thanks for the well written KIP, this looks like it will be a useful
> addition.
>
> Overall the KIP looks good to me, I have one question/comment.
>
> You mentioned that setting the `producer.id.expiration.ms` less than the
> delivery timeout could lead to duplicates, which makes sense.  To help
> avoid this situation, do we want to consider a default value that is the
> same as the delivery timeout?
>
> Thanks again for the KIP.
>
> Bill
>
> On Thu, Jul 21, 2022 at 4:54 PM Justine Olshan
>  wrote:
>
>> Hey all!
>>
>> I'd like to start a discussion on my proposal to separate time-based
>> producer ID expiration from transactional ID expiration by introducing a
>> new configuration.
>>
>> The KIP Is pretty small and simple, but will be helpful in controlling
>> memory usage in brokers -- especially now that by default producers are
>> idempotent and create producer ID state.
>>
>> Please take a look and leave any comments you may have!
>>
>> KIP:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-854+Separate+configuration+for+producer+ID+expiry
>> JIRA: https://issues.apache.org/jira/browse/KAFKA-14097
>>
>> Thanks!
>> Justine
>>
>

[jira] [Created] (KAFKA-14103) Check for hostnames in CoreUtils.scala

[Matthew de Detrich (Jira)]

Matthew de Detrich created KAFKA-14103:
--

 Summary: Check for hostnames in CoreUtils.scala
 Key: KAFKA-14103
 URL: https://issues.apache.org/jira/browse/KAFKA-14103
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Matthew de Detrich


In the process of working on [https://github.com/apache/kafka/pull/11478] we 
realized that Kafka does not do any hostname validation when parsing listener 
configurations. It would be ideal to investigate hostname validation so that we 
can eagerly short-circuit on invalid hostnames rather than the current 
behaviour (this needs to be verified). 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14093) Flaky ExactlyOnceSourceIntegrationTest.testFencedLeaderRecovery

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-14093.

Fix Version/s: 3.3.0
 Assignee: Chris Egerton
   Resolution: Fixed

> Flaky ExactlyOnceSourceIntegrationTest.testFencedLeaderRecovery
> ---
>
> Key: KAFKA-14093
> URL: https://issues.apache.org/jira/browse/KAFKA-14093
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Mickael Maison
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: 
> org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testFencedLeaderRecovery.test.stdout
>
>
> org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest > 
> testFencedLeaderRecovery FAILED
> java.lang.AssertionError: expected 
> org.apache.kafka.connect.runtime.rest.errors.ConnectRestException to be 
> thrown, but nothing was thrown



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Transactions, delivery timeout and changing transactional producer behavior

[Viktor Somogyi-Vass]

Hi Luke & Artem,

We prepared the fix, would you please help in getting a committer-reviewer
to get this issue resolved?

Thanks,
Viktor

On Fri, Jul 8, 2022 at 12:57 PM Dániel Urbán  wrote:

> Submitted a PR with the fix: https://github.com/apache/kafka/pull/12392
> In the PR I tried keeping the producer in a usable state after the forced
> bump. I understand that it might be the cleanest solution, but the only
> other option I know of is to transition into a fatal state, meaning that
> the producer has to be recreated after a delivery timeout. I think that is
> still fine compared to the out-of-order messages.
>
> Looking forward to your reviews,
> Daniel
>
> Dániel Urbán  ezt írta (időpont: 2022. júl. 7., Cs,
> 12:04):
>
> > Thanks for the feedback, I created
> > https://issues.apache.org/jira/browse/KAFKA-14053 and started working on
> > a PR.
> >
> > Luke, for the workaround, we used the transaction admin tool released in
> > 3.0 to "abort" these hanging batches manually.
> > Naturally, the cluster health should be stabilized. This issue popped up
> > most frequently around times when some partitions went into a few minute
> > window of unavailability. The infinite retries on the producer side
> caused
> > a situation where the last retry was still in-flight, but the delivery
> > timeout was triggered on the client side. We reduced the retries and
> > increased the delivery timeout to avoid such situations.
> > Still, the issue can occur in other scenarios, for example a client
> > queueing up many batches in the producer buffer, and causing those
> batches
> > to spend most of the delivery timeout window in the client memory.
> >
> > Thanks,
> > Daniel
> >
> > Luke Chen  ezt írta (időpont: 2022. júl. 7., Cs,
> 5:13):
> >
> >> Hi Daniel,
> >>
> >> Thanks for reporting the issue, and the investigation.
> >> I'm curious, so, what's your workaround for this issue?
> >>
> >> I agree with Artem, it makes sense. Please file a bug in JIRA.
> >> And looking forward to your PR! :)
> >>
> >> Thank you.
> >> Luke
> >>
> >> On Thu, Jul 7, 2022 at 3:07 AM Artem Livshits
> >>  wrote:
> >>
> >> > Hi Daniel,
> >> >
> >> > What you say makes sense.  Could you file a bug and put this info
> there
> >> so
> >> > that it's easier to track?
> >> >
> >> > -Artem
> >> >
> >> > On Wed, Jul 6, 2022 at 8:34 AM Dániel Urbán 
> >> wrote:
> >> >
> >> > > Hello everyone,
> >> > >
> >> > > I've been investigating some transaction related issues in a very
> >> > > problematic cluster. Besides finding some interesting issues, I had
> >> some
> >> > > ideas about how transactional producer behavior could be improved.
> >> > >
> >> > > My suggestion in short is: when the transactional producer
> encounters
> >> an
> >> > > error which doesn't necessarily mean that the in-flight request was
> >> > > processed (for example a client side timeout), the producer should
> not
> >> > send
> >> > > an EndTxnRequest on abort, but instead it should bump the producer
> >> epoch.
> >> > >
> >> > > The long description about the issue I found, and how I came to the
> >> > > suggestion:
> >> > >
> >> > > First, the description of the issue. When I say that the cluster is
> >> "very
> >> > > problematic", I mean all kinds of different issues, be it infra
> (disks
> >> > and
> >> > > network) or throughput (high volume producers without fine tuning).
> >> > > In this cluster, Kafka transactions are widely used by many
> producers.
> >> > And
> >> > > in this cluster, partitions get "stuck" frequently (few times every
> >> > week).
> >> > >
> >> > > The exact meaning of a partition being "stuck" is this:
> >> > >
> >> > > On the client side:
> >> > > 1. A transactional producer sends X batches to a partition in a
> single
> >> > > transaction
> >> > > 2. Out of the X batches, the last few get sent, but are timed out
> >> thanks
> >> > to
> >> > > the delivery timeout config
> >> > > 3. producer.flush() is unblocked due to all batches being "finished"
> >> > > 4. Based on the errors reported in the producer.send() callback,
> >> > > producer.abortTransaction() is called
> >> > > 5. Then producer.close() is also invoked with a 5s timeout (this
> >> > > application does not reuse the producer instances optimally)
> >> > > 6. The transactional.id of the producer is never reused (it was
> >> random
> >> > > generated)
> >> > >
> >> > > On the partition leader side (what appears in the log segment of the
> >> > > partition):
> >> > > 1. The batches sent by the producer are all appended to the log
> >> > > 2. But the ABORT marker of the transaction was appended before the
> >> last 1
> >> > > or 2 batches of the transaction
> >> > >
> >> > > On the transaction coordinator side (what appears in the transaction
> >> > state
> >> > > partition):
> >> > > The transactional.id is present with the Empty state.
> >> > >
> >> > > These happenings result in the following:
> >> > > 1. The partition leader handles the first batch after the ABORT
> >> marker as
> >> > > 

[jira] [Created] (KAFKA-14102) (SASL/OAUTHBEARER) multiple applications in one JVM process, only the first started app can consume messages

[Shuo Chen (Jira)]

Shuo Chen created KAFKA-14102:
-

 Summary: (SASL/OAUTHBEARER) multiple applications in one JVM 
process, only the first started app can consume messages
 Key: KAFKA-14102
 URL: https://issues.apache.org/jira/browse/KAFKA-14102
 Project: Kafka
  Issue Type: Bug
  Components: clients, KafkaConnect
Affects Versions: 3.0.1
Reporter: Shuo Chen


We have 2 web applications (A and B) will consume messages from the same Kafka 
Server,  so they have the same configurations:

 
{code:java}
security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule
 required; 
sasl.login.callback.handler.class=MyOauth2AuthenticateCallbackHandler 
jaas.enabled=true{code}
 

 

A and B deployed together in one Tomcat server (means they are in JVM process), 
startup  sequential is A -> B,  then we find B cannot consume the message with 
following exception:

 
{code:java}
[2022-07-22 02:52:45,184] [ INFO] 6 
[org.springframework.kafka.KafkaListenerEndpointContainer#5-0-C-1] 
o.a.k.c.n.SaslChannelBuilder             -  - [Consumer 
clientId=consumer-XXX-7d8650290c70c1fc3da6305099bde64c-1, 
groupId=XXX-7d8650290c70c1fc3da6305099bde64c] Failed to create channel due to 
org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure 
SaslClientAuthenticator
Caused by: java.lang.IllegalArgumentException: Callback handler must be 
castable to org.apache.kafka.common.security.auth.AuthenticateCallbackHandler: 
org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler
at 
org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient$OAuthBearerSaslClientFactory.createSaslClient(OAuthBearerSaslClient.java:182)
 ~[kafka-clients-3.0.1.jar:?]
at javax.security.sasl.Sasl.createSaslClient(Sasl.java:420) ~[?:1.8.0_332]
at 
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslClient$0(SaslClientAuthenticator.java:219)
 ~[kafka-clients-3.0.1.jar:?]
... suppressed 2 lines
at 
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:215)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.(SaslClientAuthenticator.java:206)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:286)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:228)
 ~[kafka-clients-3.0.1.jar:?]
at org.apache.kafka.common.network.KafkaChannel.(KafkaChannel.java:143) 
~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:236)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
 ~[kafka-clients-3.0.1.jar:?]
at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329) 
~[kafka-clients-3.0.1.jar:?]
at org.apache.kafka.common.network.Selector.connect(Selector.java:256) 
~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:981) 
~[kafka-clients-3.0.1.jar:?]
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) 
~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1152)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1040)
 ~[kafka-clients-3.0.1.jar:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549) 
~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:227)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:164)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:258)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:483)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1262)
 ~[kafka-clients-3.0.1.jar:?]
at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231) 
~[kafka-clients-3.0.1.jar:?]
at 

[jira] [Created] (KAFKA-14101) Flaky ExactlyOnceSourceIntegrationTest.testConnectorBoundary

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14101:
--

 Summary: Flaky 
ExactlyOnceSourceIntegrationTest.testConnectorBoundary
 Key: KAFKA-14101
 URL: https://issues.apache.org/jira/browse/KAFKA-14101
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Mickael Maison
 Attachments: 
org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testConnectorBoundary.test.stdout

I hit this one while running the tests on your branch from 
https://github.com/apache/kafka/pull/12429

org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest > 
testConnectorBoundary FAILED
java.lang.AssertionError: Committed records should exclude 
connector-aborted transactions expected:<[1, 3, 4, 5, 9, 10, 11, 12, 13, 14, 
15, 16, 17, 18, 19, 20, 21, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 
88, 89]> but was:<[4, 5, 10, 13, 16, 18, 20, 37, 39, 40, 46, 47, 49, 54, 59, 
64, 65, 68, 70, 71, 77, 83, 85, 89, 146, 148, 153, 154, 157, 158, 159, 163, 
165, 169, 175, 176, 178, 183, 184, 185, 187, 188, 191, 196, 199, 211, 216, 217, 
218, 222, 223, 229, 232, 238, 244, 251, 255, 259, 261, 269, 272, 274, 275, 276, 
277, 278, 279, 285, 291, 293, 296, 299]>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at 
org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testConnectorBoundary(ExactlyOnceSourceIntegrationTest.java:456)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14100) Upgrade vulnerable dependencies Kafka version 3.1.1 July 2022

[Shivakumar (Jira)]

Shivakumar created KAFKA-14100:
--

 Summary: Upgrade vulnerable dependencies Kafka version 3.1.1  July 
2022
 Key: KAFKA-14100
 URL: https://issues.apache.org/jira/browse/KAFKA-14100
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Shivakumar
 Fix For: 3.0.1, 3.2.0, 3.1.1


|Packages|Package Version|CVSS|Fix Status|
|com.fasterxml.jackson.core_jackson-databind| 2.10.5.1| 7.5| fixed in 2.14, 
2.13.1, 2.12.6|
| | | | |

Our security scan detected the above vulnerabilities

upgrade to correct versions for fixing vulnerabilities



--
This message was sent by Atlassian Jira
(v8.20.10#820010)