[jira] [Created] (KAFKA-15285) https://github.com/apache/kafka/pull/13990#issuecomment-1659256150

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15285:
--

 Summary: 
https://github.com/apache/kafka/pull/13990#issuecomment-1659256150
 Key: KAFKA-15285
 URL: https://issues.apache.org/jira/browse/KAFKA-15285
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana


storage module is intermittently failing for the last couple of weeks as 
mentioned in PR 
[thread|https://github.com/apache/kafka/pull/13990#issuecomment-1659256150].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-759: Unneeded repartition canceling

[Matthias J. Sax]

+1 (binding)

On 7/11/23 11:16 AM, Shay Lin wrote:

Hi all,

I'd like to call a vote on KIP-759: Unneeded repartition canceling
The KIP has been under discussion for quite some time(two years). This is a
valuable optimization for advanced users. I hope we can push this toward
the finish line this time.

Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-759%3A+Unneeded+repartition+canceling

Best,
Shay

[jira] [Created] (KAFKA-15284) Determine group protocol at client initialization

[Kirk True (Jira)]

Kirk True created KAFKA-15284:
-

 Summary: Determine group protocol at client initialization
 Key: KAFKA-15284
 URL: https://issues.apache.org/jira/browse/KAFKA-15284
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


At client initialization, we need to determine which of the 
{{ConsumerDelegate}} implementations to use:
 # {{LegacyKafkaConsumerDelegate}}
 # {{AsyncKafkaConsumerDelegate}}

There are conditions defined by KIP-848 that determine client eligibility to 
use the new protocol.

Known tasks:
 * Determine at what point in the {{Consumer}} initialization the network 
communication should happen
 * Determine what RPCs to invoke in order to determine eligibility (API 
versions, IBP version, etc.)
 * Implement the network client lifecycle (startup, communication, shutdown, 
etc.)
 * Determine the fallback path in case the client is not eligible to use the 
protocol



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15283) Add support for topic ID-related Consumer changes

[Kirk True (Jira)]

Kirk True created KAFKA-15283:
-

 Summary: Add support for topic ID-related Consumer changes
 Key: KAFKA-15283
 URL: https://issues.apache.org/jira/browse/KAFKA-15283
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True


Currently, {{KafkaConsumer}} keeps track of topic IDs in the in-memory 
{{ConsumerMetadata}} object, and they are provided to the {{FETCH}} and 
{{METADATA}} RPC calls.

With KIP-848 the OffsetFetch and OffsetCommit will start using topic IDs in the 
same way, so the new client implementation will provide it when issuing those 
requests. Topic names should continue to be supported as needed by the 
{{{}AdminClient{}}}.

We should also review/clean-up the support for topic names in requests such as 
the {{METADATA}} request (currently supporting topic names as well as topic IDs 
on the client side).

Tasks include:
 * Introduce Topic ID in existing OffsetFetch and OffsetCommit API that will be 
upgraded on the server to support topic ID
 * Check topic ID propagation internally in the client based on RPCs including 
it.
 * Review existing support for topic name for potential clean if not needed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15282) Implement client support for KIP-848 client-side assignors

[Kirk True (Jira)]

Kirk True created KAFKA-15282:
-

 Summary: Implement client support for KIP-848 client-side assignors
 Key: KAFKA-15282
 URL: https://issues.apache.org/jira/browse/KAFKA-15282
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


This includes:
 * Validate the client’s configuration for assignor selection
 * Validate the request/response from the {{ConsumerGroupHeartbeat}} RPC call

This task is part of the work to implement support for the new KIP-848 consumer 
group protocol.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15281) Implement the groupMetadata Consumer API

[Kirk True (Jira)]

Kirk True created KAFKA-15281:
-

 Summary: Implement the groupMetadata Consumer API
 Key: KAFKA-15281
 URL: https://issues.apache.org/jira/browse/KAFKA-15281
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


The threading refactor project needs to implement the {{groupMetadata()}} API 
call once support for the KIP-848 protocol is implemented.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15280) Implement client support for KIP-848 server-side assignors

[Kirk True (Jira)]

Kirk True created KAFKA-15280:
-

 Summary: Implement client support for KIP-848 server-side assignors
 Key: KAFKA-15280
 URL: https://issues.apache.org/jira/browse/KAFKA-15280
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


This includes:
 * Validate the client’s configuration for assignor selection
 * Validate the request/response from the {{ConsumerGroupHeartbeat}} RPC call

This task is part of the work to implement support for the new KIP-848 consumer 
group protocol.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15279) Implement client support for KIP-848 assignment RPCs

[Kirk True (Jira)]

Kirk True created KAFKA-15279:
-

 Summary: Implement client support for KIP-848 assignment RPCs
 Key: KAFKA-15279
 URL: https://issues.apache.org/jira/browse/KAFKA-15279
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


The protocol introduces three new RPCs that the client uses to communicate with 
the broker:
 # 
[ConsumerGroupHeartbeat|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-ConsumerGroupHeartbeatAPI]

 # 
[ConsumerGroupPrepareAssignment|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-ConsumerGroupPrepareAssignmentAPI]

 # 
[ConsumerGroupInstallAssignment|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-ConsumerGroupInstallAssignmentAPI]

Support for ConsumerGroupHeartbeat is handled by KAFKA-15278. This task is to 
implement the ConsumerGroupAssignmentRequestManager to handle the second and 
third RPCs on the above list.

This task is part of the work to implement support for the new KIP-848 consumer 
group protocol.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15278) Implement client support for KIP-848 ConsumerGroupHeartbeat protocol RPC

[Kirk True (Jira)]

Kirk True created KAFKA-15278:
-

 Summary: Implement client support for KIP-848 
ConsumerGroupHeartbeat protocol RPC
 Key: KAFKA-15278
 URL: https://issues.apache.org/jira/browse/KAFKA-15278
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


The protocol introduces three new RPCs that the client uses to communicate with 
the broker:
 # 
[ConsumerGroupHeartbeat|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-ConsumerGroupHeartbeatAPI]

 # 
[ConsumerGroupPrepareAssignment|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-ConsumerGroupPrepareAssignmentAPI]

 # 
[ConsumerGroupInstallAssignment|https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-ConsumerGroupInstallAssignmentAPI]

The necessary Java code that represents the {{ConsumerGroupHeartbeatRequest}} 
and {{ConsumerGroupHeartbeatResponse}} are already present in the codebase. It 
is assumed that the scaffolding for the other two will come along in time.
 * Implement {{ConsumerGroupRequestManager }}
 * Ensure that {{DefaultBackgroundThread}} correctly calculates I/O timeouts so 
that the heartbeat occurs within the {{group.consumer.session.timeout.ms}} 
interval regardless of other {{RequestManager}} instance activity

This task is part of the work to implement support for the new KIP-848 consumer 
group protocol.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15277) Design & implement support for internal Consumer delegates

[Kirk True (Jira)]

Kirk True created KAFKA-15277:
-

 Summary: Design & implement support for internal Consumer delegates
 Key: KAFKA-15277
 URL: https://issues.apache.org/jira/browse/KAFKA-15277
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


As mentioned above, there are presently two different, coexisting 
implementations of the {{Consumer}} interface: {{KafkaConsumer}} ("old") and 
{{PrototypeAsyncConsumer}} ("new"). Eventually, these will be reorganized using 
the delegation pattern. The top-level {{KafkaConsumer}} that implements the old 
protocol will be renamed as {{LegacyKafkaConsumer}} and 
{{PrototypeAsyncConsumer}} will be renamed as {{{}AsyncKafkaConsumer{}}}. It is 
assumed that neither {{AsyncKafkaConsumer}} nor {{LegacyKafkaConsumer}} will be 
top-level implementations of {{{}Consumer{}}}, but will likely implement an 
internal interface that is better suited to the needs of the top-level 
{{{}KafkaConsumer{}}}.

Provide the Java client support for the consumer delegates, including:
 * Create {{ConsumerDelegate}} interface
 * Clone {{{}KafkaConsumer{}}}, rename as {{LegacyKafkaConsumerDelegate}} and 
refactor to implement {{ConsumerDelegate}}
 * Rename {{PrototypeAsyncConsumer}} to {{AsyncKafkaConsumerDelegate}} and 
refactor to implement the {{ConsumerDelegate}} interface
 * Refactor the (original) {{KafkaConsumer}} to remove the core implementation, 
instead delegating to the {{{}ConsumerDelegate{}}}, which will be hard-coded to 
use {{LegacyKafkaConsumerDelegate}}

This task is part of the work to implement support for the new KIP-848 consumer 
group protocol.{{{}
{}}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15276) Implement partition assignment reconciliation

[Kirk True (Jira)]

Kirk True created KAFKA-15276:
-

 Summary: Implement partition assignment reconciliation
 Key: KAFKA-15276
 URL: https://issues.apache.org/jira/browse/KAFKA-15276
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


Provide the Java client support for the consumer group partition assignment 
logic, including:
 * Calculate the difference between the current partition assignment and that 
returned in the {{ConsumerGroupHeartbeatResponse}} RPC response
 * Ensure we handle the case where changes to the assignment take multiple 
passes of {{RequestManager.poll()}}
 * Integrate the mechanism to invoke the user’s rebalance callback

This task is part of the work to implement support for the new KIP-848 consumer 
group protocol.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15275) Consumer group membership state machine

[Kirk True (Jira)]

Kirk True created KAFKA-15275:
-

 Summary: Consumer group membership state machine
 Key: KAFKA-15275
 URL: https://issues.apache.org/jira/browse/KAFKA-15275
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Kirk True
Assignee: Kirk True


Provide the Java client support for the consumer group member state machine, 
including:
 * Define the states of the client member, based on the heartbeat 
{{ConsumerGroupHeartbeat}} data structure & state transitions
 * Determine the valid transitions between those states
 * Implement logic to ensure that the data in the 
{{ConsumerGroupHeartbeatResponse}} is validated by the state machine, including 
throwing the correct exception types

This task is part of the work to implement support for the new KIP-848 consumer 
group protocol.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

[Shay Lin]

Hi all,

It's been a few days, if there is no further comments or questions I'd like
to call for a vote. There is an existing VOTE thread if you search for
KIP-759.

Thank you,
Shay

On Wed, Jul 26, 2023 at 7:30 PM Shay Lin  wrote:

> Very good catch, Matthias. I updated the KIP to state that the new
> DSLOperation will return a new, mutated KStream.
>
> Thank you,
> Shay
>
> On Wed, Jul 26, 2023 at 6:13 PM Matthias J. Sax  wrote:
>
>> One last question. What should happen for the following case:
>>
>> KStream myStream = build.stream(...).map(...);
>> myStream.markAsPartiitoned().groupByKey().aggregate(...);
>> myStream.join(...)
>>
>> The question is about the "fan-out" pattern. `myStream`, which is marked
>> for partitioning, is fed into two downstream operations. Thus, it's
>> clear that the aggregation won't trigger a rebalance. However, the
>> fan-out happens before `markAsRepartiitoned` and thus I would assume
>> that the join would trigger a repartitioning?
>>
>> This question is important, because if we follow what I said above,
>> `markAsRepartiitoned` returns a new KStream object, but does mutate the
>> upstream KStream object, what is semantically two different things. It
>> also has an impact on how we need to implement the feature. The KIP
>> should explicitly explain this case.
>>
>>
>> -Matthias
>>
>> On 7/26/23 4:58 PM, Shay Lin wrote:
>> > Hi John,
>> >
>> > Thanks for your reply. I updated the KIP to reflect the changes we
>> > discussed in the thread today.
>> > #1 is duly noted, I learned from the examples Sophie sent earlier! =)
>> >
>> > In the new version, I also talked about why IQ and joins will not work
>> with
>> > the interface and talked about the mitigation. The proposal
>> > now specifically states we are solving the unneeded partition problem
>> when
>> > IQ or join does not coexist in the kafka streams. In the concerns
>> section,
>> > the proposal talks about having a reverse mapping would make this new
>> > interface compatible with IQ and join again but is subject to demand.
>> >
>> > Let me know what you think. Thanks!
>> > Shay
>> >
>> >
>> >
>> > On Wed, Jul 26, 2023 at 2:35 PM John Roesler 
>> wrote:
>> >
>> >> Hello Shay,
>> >>
>> >> Thanks for the KIP!
>> >>
>> >> I just took a look in preparation to vote, and there are two small-ish
>> >> things that I'd like to fix first. Apologies if this stuff has already
>> come
>> >> up in the discussion thread; I only skimmed it.
>> >>
>> >> 1. The KIP only mentions the name of the method instead of providing a
>> >> code snippet showing exactly what the method signature will be in the
>> >> interface. Normally, KIPs do the latter because it removes all
>> ambiguity
>> >> from the proposal. It also gives you an opportunity to write down the
>> >> Javadoc you would add to the method instead of just mentioning the
>> points
>> >> that you plan to document.
>> >>
>> >> 2. The KIP lists some concerns, but not what you will do to mitigate
>> them.
>> >> For example, the concern about IQ not behaving correctly. Will you
>> disable
>> >> the use of the implicit partitioner downstream of one of these
>> >> cancellations? Or provide a new interface to supply the "reverse
>> mapping"
>> >> you mentioned? Or include documentation in the Javadoc for how to deal
>> with
>> >> the situation? I think there are a range of options for each of those
>> >> concerns, and we should state up front what we plan to do.
>> >>
>> >> Thanks again!
>> >> -John
>> >>
>> >> On 2023/07/24 20:33:05 Sophie Blee-Goldman wrote:
>> >>> Thanks Shay! You and Matthias have convinced me, I'm happy with the
>> >> current
>> >>> proposal. I think once you make the minor
>> >>> updates to the KIP document this will be ready for voting again.
>> >>>
>> >>> Cheers,
>> >>> Sophie
>> >>>
>> >>> On Mon, Jul 24, 2023 at 8:26 AM Shay Lin  wrote:
>> >>>
>>  Hi Sophie and Matthias, thanks for your comments and replies.
>> 
>>  1. Scope of change: KStreams only or KStreams/KTable
>>  I took some time to digest your points, looking through how KStreams
>>  triggers repartitions today. I noticed that `repartitionRequired`is a
>> >> flag
>>  in KStreamImpl etc and not in KTableImpl etc. When I look further, in
>> >> the
>>  case of KTable, instead of passing in a boolean flag, a repartition
>> >> node `
>>  TableRepartitionMapNode` is directly created. I went back and
>> >> referenced
>>  the two issue tickets KAFKA-10844 and KAFKA-4835, both requests were
>>  focused on KStreams, i.e. not to change the partition why the input
>> >> streams
>>  are already correctly keyed. Is it possible that in the case of
>> KTable,
>>  users always intend to repartition (change key) when they call on
>>  aggregate? -- (this was written before I saw Matthias's comment)
>> 
>>  Overall, based on the tickets, I see the benefit of doing a contained
>>  change focusing on KStreams, i.e. repartitionRequired, which would
>> >> solve
>> >

[DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams

[Florin Akermann]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams

Re: Feedback of using the 'confluent-kafka-go' for reliablity and performance

[Kenneth Eversole]

Hello,

Xinli,

I can only speak for Cloudflare, but we use various versions of the Sarama
Go Client for various production workloads and they seem to function quite
well.

I am curious what your reasoning for wanting to replace Sarama?

Kenneth Eversole

On Fri, Jul 28, 2023 at 1:58 PM Xinli shang  wrote:

> Hi all,
>
> We are considering replacing the Sarama Go client with confluent-kafka-go
>  which is supported
> and
> seems promising. Regarding reliability and performance, does anybody want
> to share the experience? Whether or not it is the experience in production
> or testing, they are all appreciated.
>
> --
> Xinli Shang
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2056

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-954: expand default DSL store configuration to custom types

[Almog Gavra]

Thanks everyone! Marking this KIP as accepted with +1s from Bruno, John &
Guozhang.

@Bruno, yes I think that's a great idea! I was going to update the KIP but
missed that part, I'll do it now as I assume that's not controversial.

Cheers,
Almog

On Mon, Jul 31, 2023 at 1:47 AM Bruno Cadonna  wrote:

> Hi Almog,
>
> +1 (binding)
>
> Last question: Did you consider to let interface DslStoreSuppliers
> extend Configurable?
>
> Thanks,
> Bruno
>
>
> On 7/30/23 5:53 AM, John Roesler wrote:
> > Thanks for the KIP, Almog!
> >
> > I'm +1 (binding)
> >
> > I've reviewed the KIP and skimmed the discussion thread. I think this is
> going to be a very nice improvement.
> >
> > Thanks,
> > -John
> >
> > On Sat, Jul 29, 2023, at 13:26, Guozhang Wang wrote:
> >> Thanks Almog! I made a pass over the updated wiki and have no more
> questions. +1
> >>
> >> Guozhang
> >>
> >> On Wed, Jul 26, 2023 at 8:14 AM Almog Gavra 
> wrote:
> >>>
> >>> Hello Everyone,
> >>>
> >>> Opening the voting for KIP-954. The discussion is converging, but
> please
> >>> feel free to chime in on the last few conversation points if you aren't
> >>> happy with where it settled.
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-954%3A+expand+default+DSL+store+configuration+to+custom+types
> >>>
> >>> Cheers,
> >>> Almog
>

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

[Jack Tomy]

Hey Andrew, Sagar

Thanks. I'm travelling so sorry for being brief and getting back late.

1. For the first concern, that is moving in a direction of server side
partitioner, the idea seems very much promising but I believe we still have
a long way to go. Since the proposal/design for the same is still not
available, it's hard for me to defend my proposal.
2.  For the second concern:
 2.1 Loss of order in messages, I believe the ordering of messages is never
promised and the partitioner has no requirement to ensure the same. It is
upto the user to implement/use a partitioner which ensures ordering based
on keys.
2.2 Key deciding the partitioner, It is totally up to the user to decide
the partition regardless of the key, we are also passing the value to the
partitioner. Even the existing implementation receives the value which lets
the user decide the partition based on value.
2.3 Sending to a specific partition, for this, I need to be aware of the
total number of partitions, but if I can do that same in partitioner, the
cluster param gives me all the information I want.

I would also quote a line from KIP-82 where headers were added to the
serializer : The payload is traditionally for the business object, and *headers
are traditionally used for transport routing*, filtering etc. So I
believe when a user wants to add some routing information (in this case
which set of partitions to go for), headers seem to be the right place.



On Sat, Jul 29, 2023 at 8:48 PM Sagar  wrote:

> Hi Andrew,
>
> Thanks for your comments.
>
> 1) Yes that makes sense and that's what even would expect to see as well. I
> just wanted to highlight that we might still need a way to let client side
> partitioning logic be present as well. Anyways, I am good on this point.
> 2) The example provided does seem achievable by simply attaching the
> partition number in the ProducerRecord. I guess if we can't find any
> further examples which strengthen the case of this partitioner, it might be
> harder to justify adding it.
>
>
> Thanks!
> Sagar.
>
> On Fri, Jul 28, 2023 at 2:05 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
> > Hi Sagar,
> > Thanks for your comments.
> >
> > 1) Server-side partitioning doesn’t necessarily mean that there’s only
> one
> > way to do it. It just means that the partitioning logic runs on the
> broker
> > and
> > any configuration of partitioning applies to the broker’s partitioner. If
> > we ever
> > see a KIP for this, that’s the kind of thing I would expect to see.
> >
> > 2) In the priority example in the KIP, there is a kind of contract
> between
> > the
> > producers and consumers so that some records can be processed before
> > others regardless of the order in which they were sent. The producer
> > wants to apply special significance to a particular header to control
> which
> > partition is used. I would simply achieve this by setting the partition
> > number
> > in the ProducerRecord at the time of sending.
> >
> > I don’t think the KIP proposes adjusting the built-in partitioner or
> > adding to AK
> > a new one that uses headers in the partitioning decision. So, any
> > configuration
> > for a partitioner that does support headers would be up to the
> > implementation
> > of that specific partitioner. Partitioner implements Configurable.
> >
> > I’m just providing an alternative view and I’m not particularly opposed
> to
> > the KIP.
> > I just don’t think it quite merits the work involved to get it voted and
> > merged.
> > As an aside, a long time ago, I created a small KIP that was never
> adopted
> > and I didn’t push it because I eventually didn’t need it.
> >
> > Thanks,
> > Andrew
> >
> > > On 28 Jul 2023, at 05:15, Sagar  wrote:
> > >
> > > Hey Andrew,
> > >
> > > Thanks for the review. Since I had reviewed the KIP I thought I would
> > also
> > > respond. Of course Jack has the final say on this since he wrote the
> KIP.
> > >
> > > 1) This is an interesting point and I hadn't considered it. The
> > > comparison with KIP-848 is a valid one but even within that KIP, it
> > allows
> > > client side partitioning for power users like Streams. So while we
> would
> > > want to move away from client side partitioner as much as possible, we
> > > still shouldn't do away completely with Client side partitioning and
> end
> > up
> > > being in a state of inflexibility for different kinds of usecases. This
> > is
> > > my opinion though and you have more context on Clients, so would like
> to
> > > know your thoughts on this.
> > >
> > > 2) Regarding this, I assumed that since the headers are already part of
> > the
> > > consumer records they should have access to the headers and if there
> is a
> > > contract b/w the applications producing and the application consuming,
> > that
> > > decisioning should be transparent. Was my assumption incorrect? But as
> > you
> > > rightly pointed out header based partitioning with keys is going to
> lead
> > to
> > > surprising results. Ass

[jira] [Resolved] (KAFKA-15252) Task is not stopped until the poll interval passes in case of task restarting.

[Nikita (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita resolved KAFKA-15252.

Resolution: Duplicate

> Task is not stopped until the poll interval passes in case of task restarting.
> --
>
> Key: KAFKA-15252
> URL: https://issues.apache.org/jira/browse/KAFKA-15252
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Nikita
>Priority: Major
>
> We face a problem with restarting of the tasks, sometimes it leads to 
> resource leak. 
> We used the jdbc source connector and noticed an increasing of count of 
> opened sessions on Vertica side. But this problem is applicable for all 
> databases and possibly for all source connectors.
> Our case is the next: 
> 1) Run jdbc source connector (io.confluent.connect.jdbc.JdbcSourceConnector) 
> and set poll.interval.ms (8640) > task.shutdown.graceful.timeout.ms (it's 
> the property on Kafka-connect side, we set 1)
> 2) Send POST /connectors//tasks//restart
> ER: count of session is the same as before restart
> AR: count of session increases
> The main problem is when 
> org.apache.kafka.connect.runtime.Worker#stopAndAwaitTasks(java.util.Collection)
>   method is called it doesn't stop a source task itself. 
> The source task stops only if polling process stops on source task side. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15274) support moving files to be deleted to other directories

[jianbin.chen (Jira)]

jianbin.chen created KAFKA-15274:


 Summary: support moving files to be deleted to other directories
 Key: KAFKA-15274
 URL: https://issues.apache.org/jira/browse/KAFKA-15274
 Project: Kafka
  Issue Type: Improvement
Reporter: jianbin.chen


Hello everyone, I am a Kafka user from China. Our company operates in public 
clouds overseas, such as AWS, Ali Cloud, and Huawei Cloud. We face a large 
amount of data exchange and business message delivery every day. Daily messages 
consume a significant amount of disk space. Purchasing the corresponding 
storage capacity on these cloud providers incurs substantial costs, especially 
for SSDs with ultra-high IOPS. High IOPS is very effective for disaster 
recovery, especially in the event of a sudden broker failure where storage 
space becomes full or memory space is exhausted leading to OOM kills. This high 
IOPS storage greatly improves data recovery efficiency, forcing us to adopt 
smaller storage specifications with high IO to save costs. Particularly, cloud 
providers only allow capacity expansion but not reduction.

Currently, we have come up with a solution and would like to contribute it to 
the community for discussion. When we need to delete logs, I can purchase S3 or 
Minio storage from services like AWS and mount it to my brokers. When a log 
needs to be deleted, we can decide how it leaves the broker. The default is to 
delete it directly, while the move option moves it to S3. Since most of the 
deleted data is cold data that won't be used in the short term, this approach 
improves the retention period of historical data while maintaining good cost 
control.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-714: Client metrics and observability

[Andrew Schofield]

Hi Milind,
Thanks for your question.

On reflection, I agree that INVALID_RECORD is most likely to be caused by a
problem in the serialization in the client. I have changed the client action in 
this case
to “Log an error and stop pushing metrics”.

I have updated the KIP text accordingly.

Thanks,
Andrew

> On 31 Jul 2023, at 12:09, Milind Luthra  wrote:
>
> Hi Andrew,
> Thanks for the clarifications.
>
> About 2b:
> In case a client has a bug while serializing, it might be difficult for the
> client to recover from that without code changes. In that, it might be good
> to just log the INVALID_RECORD as an error, and treat the error as fatal
> for the client (only fatal in terms of sending the metrics, the client can
> keep functioning otherwise). What do you think?
>
> Thanks
> Milind
>
> On Mon, Jul 24, 2023 at 8:18 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
>> Hi Milind,
>> Thanks for your questions about the KIP.
>>
>> 1) I did some archaeology and looked at historical versions of the KIP. I
>> think this is
>> just a mistake. 5 minutes is the default metric push interval. 30 minutes
>> is a mystery
>> to me. I’ve updated the KIP.
>>
>> 2) I think there are two situations in which INVALID_RECORD might occur.
>> a) The client might perhaps be using a content-type that the broker does
>> not support.
>> The KIP mentions content-type as a future extension, but there’s only one
>> supported
>> to start with. Until we have multiple content-types, this seems out of
>> scope. I think a
>> future KIP would add another error code for this.
>> b) The client might perhaps have a bug which means the metrics payload is
>> malformed.
>> Logging a warning and attempting the next metrics push on the push
>> interval seems
>> appropriate.
>>
>> UNKNOWN_SUBSCRIPTION_ID would indeed be handled by making an immediate
>> GetTelemetrySubscriptionsRequest.
>>
>> UNSUPPORTED_COMPRESSION_TYPE seems like either a client bug or perhaps
>> a situation in which a broker sends a compression type in a
>> GetTelemetrySubscriptionsResponse
>> which is subsequently not supported when its used with a
>> PushTelemetryRequest.
>> We do want the client to have the opportunity to get an up-to-date list of
>> supported
>> compression types. I think an immediate GetTelemetrySubscriptionsRequest
>> is appropriate.
>>
>> 3) If a client attempts a subsequent handshake with a Null
>> ClientInstanceId, the
>> receiving broker may not already know the client's existing
>> ClientInstanceId. If the
>> receiving broker knows the existing ClientInstanceId, it simply responds
>> the existing
>> value back to the client. If it does not know the existing
>> ClientInstanceId, it will create
>> a new client instance ID and respond with that.
>>
>> I will update the KIP with these clarifications.
>>
>> Thanks,
>> Andrew
>>
>>> On 17 Jul 2023, at 14:21, Milind Luthra 
>> wrote:
>>>
>>> Hi Andrew, thanks for this KIP.
>>>
>>> I had a few questions regarding the "Error handling" section.
>>>
>>> 1. It mentions that "The 5 and 30 minute retries are to eventually
>> trigger
>>> a retry and avoid having to restart clients if the cluster metrics
>>> configuration is disabled temporarily, e.g., by operator error, rolling
>>> upgrades, etc."
>>> But this 30 min interval isn't mentioned anywhere else. What is it
>>> referring to?
>>>
>>> 2. For the actual errors:
>>> INVALID_RECORD : The action required is to "Log a warning to the
>>> application and schedule the next GetTelemetrySubscriptionsRequest to 5
>>> minutes". Why is this 5 minutes, and not something like PushIntervalMs?
>> And
>>> also, why are we scheduling a GetTelemetrySubscriptionsRequest in this
>>> case, if the serialization is broken?
>>> UNKNOWN_SUBSCRIPTION_ID , UNSUPPORTED_COMPRESSION_TYPE : just to confirm,
>>> the GetTelemetrySubscriptionsRequest needs to be scheduled immediately
>>> after the PushTelemetry response, correct?
>>>
>>> 3. For "Subsequent GetTelemetrySubscriptionsRequests must include the
>>> ClientInstanceId returned in the first response, regardless of broker":
>>> Will a broker error be returned in case some implementation of this KIP
>>> violates this accidentally and sends a request with ClientInstanceId =
>> Null
>>> even when it's been obtained already? Or will a new ClientInstanceId be
>>> returned without an error?
>>>
>>> Thanks!
>>>
>>> On Tue, Jun 13, 2023 at 8:38 PM Andrew Schofield <
>>> andrew_schofield_j...@outlook.com> wrote:
>>>
 Hi,
 I would like to start a new discussion thread on KIP-714: Client metrics
 and observability.



>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability

 I have edited the proposal significantly to reduce the scope. The
>> overall
 mechanism for client metric subscriptions is unchanged, but the
 KIP is now based on the existing client metrics, rather than introducing
 new metrics. The purpose remains helping cluster operators

Re: [DISCUSS] KIP-714: Client metrics and observability

[Milind Luthra]

Hi Andrew,
Thanks for the clarifications.

About 2b:
In case a client has a bug while serializing, it might be difficult for the
client to recover from that without code changes. In that, it might be good
to just log the INVALID_RECORD as an error, and treat the error as fatal
for the client (only fatal in terms of sending the metrics, the client can
keep functioning otherwise). What do you think?

Thanks
Milind

On Mon, Jul 24, 2023 at 8:18 PM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Milind,
> Thanks for your questions about the KIP.
>
> 1) I did some archaeology and looked at historical versions of the KIP. I
> think this is
> just a mistake. 5 minutes is the default metric push interval. 30 minutes
> is a mystery
> to me. I’ve updated the KIP.
>
> 2) I think there are two situations in which INVALID_RECORD might occur.
> a) The client might perhaps be using a content-type that the broker does
> not support.
> The KIP mentions content-type as a future extension, but there’s only one
> supported
> to start with. Until we have multiple content-types, this seems out of
> scope. I think a
> future KIP would add another error code for this.
> b) The client might perhaps have a bug which means the metrics payload is
> malformed.
> Logging a warning and attempting the next metrics push on the push
> interval seems
> appropriate.
>
> UNKNOWN_SUBSCRIPTION_ID would indeed be handled by making an immediate
> GetTelemetrySubscriptionsRequest.
>
> UNSUPPORTED_COMPRESSION_TYPE seems like either a client bug or perhaps
> a situation in which a broker sends a compression type in a
> GetTelemetrySubscriptionsResponse
> which is subsequently not supported when its used with a
> PushTelemetryRequest.
> We do want the client to have the opportunity to get an up-to-date list of
> supported
> compression types. I think an immediate GetTelemetrySubscriptionsRequest
> is appropriate.
>
> 3) If a client attempts a subsequent handshake with a Null
> ClientInstanceId, the
> receiving broker may not already know the client's existing
> ClientInstanceId. If the
> receiving broker knows the existing ClientInstanceId, it simply responds
> the existing
> value back to the client. If it does not know the existing
> ClientInstanceId, it will create
> a new client instance ID and respond with that.
>
> I will update the KIP with these clarifications.
>
> Thanks,
> Andrew
>
> > On 17 Jul 2023, at 14:21, Milind Luthra 
> wrote:
> >
> > Hi Andrew, thanks for this KIP.
> >
> > I had a few questions regarding the "Error handling" section.
> >
> > 1. It mentions that "The 5 and 30 minute retries are to eventually
> trigger
> > a retry and avoid having to restart clients if the cluster metrics
> > configuration is disabled temporarily, e.g., by operator error, rolling
> > upgrades, etc."
> > But this 30 min interval isn't mentioned anywhere else. What is it
> > referring to?
> >
> > 2. For the actual errors:
> > INVALID_RECORD : The action required is to "Log a warning to the
> > application and schedule the next GetTelemetrySubscriptionsRequest to 5
> > minutes". Why is this 5 minutes, and not something like PushIntervalMs?
> And
> > also, why are we scheduling a GetTelemetrySubscriptionsRequest in this
> > case, if the serialization is broken?
> > UNKNOWN_SUBSCRIPTION_ID , UNSUPPORTED_COMPRESSION_TYPE : just to confirm,
> > the GetTelemetrySubscriptionsRequest needs to be scheduled immediately
> > after the PushTelemetry response, correct?
> >
> > 3. For "Subsequent GetTelemetrySubscriptionsRequests must include the
> > ClientInstanceId returned in the first response, regardless of broker":
> > Will a broker error be returned in case some implementation of this KIP
> > violates this accidentally and sends a request with ClientInstanceId =
> Null
> > even when it's been obtained already? Or will a new ClientInstanceId be
> > returned without an error?
> >
> > Thanks!
> >
> > On Tue, Jun 13, 2023 at 8:38 PM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi,
> >> I would like to start a new discussion thread on KIP-714: Client metrics
> >> and observability.
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability
> >>
> >> I have edited the proposal significantly to reduce the scope. The
> overall
> >> mechanism for client metric subscriptions is unchanged, but the
> >> KIP is now based on the existing client metrics, rather than introducing
> >> new metrics. The purpose remains helping cluster operators
> >> investigate performance problems experienced by clients without
> requiring
> >> changes to the client application code or configuration.
> >>
> >> Thanks,
> >> Andrew
>
>

Re: [VOTE] KIP-954: expand default DSL store configuration to custom types

[Bruno Cadonna]

Hi Almog,

+1 (binding)

Last question: Did you consider to let interface DslStoreSuppliers 
extend Configurable?


Thanks,
Bruno


On 7/30/23 5:53 AM, John Roesler wrote:

Thanks for the KIP, Almog!

I'm +1 (binding)

I've reviewed the KIP and skimmed the discussion thread. I think this is going 
to be a very nice improvement.

Thanks,
-John

On Sat, Jul 29, 2023, at 13:26, Guozhang Wang wrote:

Thanks Almog! I made a pass over the updated wiki and have no more questions. +1

Guozhang

On Wed, Jul 26, 2023 at 8:14 AM Almog Gavra  wrote:


Hello Everyone,

Opening the voting for KIP-954. The discussion is converging, but please
feel free to chime in on the last few conversation points if you aren't
happy with where it settled.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-954%3A+expand+default+DSL+store+configuration+to+custom+types

Cheers,
Almog