Re: [DISCUSS] KIP-820: Extend KStream process with new Processor API

[Matthias J. Sax]

John, thanks for verifying source compatibility. My impression was that 
it should be source compatible, I was just not 100% sure.


The question about `processValues()` is really a hard one. Guozhang's 
point is very good one. Maybe we need to be pragmatic and accept the 
runtime check (even if I deeply hate this solution compare to a compile 
time check).


Other possibilities to address this issue might just become too ugly? It 
seems it would require to add a new `ValueProcessorContext` that offers 
a `#forward(ValueRecord)` method (with `ValueRecord` being a `Record` 
with immutable key? Not sure if we would be willing to go down this 
route? Personally, I would be ok with it, as a strongly prefer compile 
time checks and I am happy to extend the API surface area to achieve it 
-- however, I won't be surprised if others don't like this idea...




-Matthias

On 2/27/22 6:20 AM, Jorge Esteban Quilcate Otoya wrote:

Thanks, Guozhang.


Compared with reference checks and runtime exceptions for those who
mistakenly change the key, I think that enforcing everyone to `setValue`
may incur more costs..


This is a fair point. I agree that this may incur in more costs than key
checking.

Will hold for more feedback, but if we agree I will update the KIP during
the week.

Cheers,
Jorge.


On Sun, 27 Feb 2022 at 00:50, Guozhang Wang  wrote:


Hello folks,

Regarding the outstanding question, I'm actually a bit leaning towards the
second option since that `withKey()` itself always creates a new Record
object. This has a few implications:

* That we would have to discard the previous Record object to be GC'ed with
the new object --- note in practice, processing value does not mean you'd
have to replace the whole value with `withValue`, but maybe you just need
to manipulate some fields of the value object if it is a JSon / etc.
* It may become an obstacle for further runtime optimizations e.g. skip
serdes and interpret processing as direct byte manipulations.

Compared with reference checks and runtime exceptions for those who
mistakenly change the key, I think that enforcing everyone to `setValue`
may incur more costs..

Guozhang

On Fri, Feb 25, 2022 at 12:54 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:


Hi all,

Appreciate very much all the great feedback received so far.


After applying that interface change, I don't see any syntax

errors in our tests (which use those methods), and the
StreamBuilderTest still passes for me.

This is awesome John, thank you for your efforts here.


Jorge, do you mind clarifying these points in the Compatibility section

of your KIP?

+1. I have clarified the impact of changing the return type in the KIP.


I think the other outstanding question for you is whether
the output key type for processValues should be K or Void.

One thing I realized belatedly was that if we do set it to
Void, then users will actually have to override the key when
forwarding, like `record.withKey(null)`, whereas if we keep
it is K, all users have to do is not touch the key at all.


This is a tricky one.
On one hand, with Void type for key output, we force the users to cast to
Void and change the key to null,
though this can be documented on the API, so the users are aware of the
peculiarity of forwarding within `processValues`.
On the other hand, keeping the key type as output doesn't _require_ to do
any change of keys,
but this could lead to key-checking runtime exceptions.

I slightly inclined myself for the first option and change the type to
`Void`.
This will impose a bit of pain on the users to gain some type-safety and
avoid runtime exceptions.
We can justify this requirement as a way to prove that the key hasn't
changed.

Btw, thanks for this idea Matthias!


On Fri, 25 Feb 2022 at 17:10, John Roesler  wrote:


Oh, one more thing Jorge,

I think the other outstanding question for you is whether
the output key type for processValues should be K or Void. I
get the impression that all of us don't feel too strongly
about it, so I think the ball is in your court to consider
everyone's points and make a call (with justification).

One thing I realized belatedly was that if we do set it to
Void, then users will actually have to override the key when
forwarding, like `record.withKey(null)`, whereas if we keep
it as K, all users have to do is not touch the key at all.

Thanks,
-John

On Fri, 2022-02-25 at 11:07 -0600, John Roesler wrote:

Hello all,

I'll chime in again in the interest of trying to do a better
job of keeping KIPs moving forward...

Matthias raised some very good questions about whether the
change is really source compatible. I just checked out the
code and make the interface change that Jorge specified in
the KIP:


Modified methods:

KStream KStream#process(ProcessorSupplier
KOut, VOut> processorSupplier, String... stateStoreNames)

KStream KStream#process(ProcessorSupplier
KOut, VOut> processorSupplier, Named named, String...
stateStoreNames)

After applying 

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Artem Livshits]

Hi Jun,

2. Removed the option from the KIP.  Now the sticky partitioning threshold
is hardcoded to batch.size.

20. Added the corresponding wording to the KIP.

-Artem

On Thu, Mar 3, 2022 at 10:52 AM Jun Rao  wrote:

> Hi, Artem,
>
> Thanks for the reply.
>
> 1. Sounds good.
>
> 2. If we don't expect users to change it, we probably could just leave out
> the new config. In general, it's easy to add a new config, but hard to
> remove an existing config.
>
> 20. The two new configs enable.adaptive.partitioning and
> partition.availability.timeout.ms only apply to the two built-in
> partitioners DefaultPartitioner and UniformStickyPartitioner, right? It
> would be useful to document that in the KIP.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 3, 2022 at 9:47 AM Artem Livshits
>  wrote:
>
> > Hi Jun,
> >
> > Thank you for the suggestions.
> >
> > 1. As we discussed offline, we can hardcode the logic for
> > DefaultPartitioner and UniformStickyPartitioner in the KafkaProducer
> (i.e.
> > the DefaultPartitioner.partition won't get called, instead KafkaProducer
> > would check if the partitioner is an instance of DefaultPartitioner and
> > then run the actual partitioning logic itself).  Then the change to the
> > Partitioner wouldn't be required.  I'll update the KIP to reflect that.
> >
> > 2. I don't expect users to change this too often, as changing it would
> > require a bit of studying of the production patterns.  As a general
> > principle, if I can think of a model that requires a deviation from
> > default, I tend to add a configuration option.  It could be that it'll
> > never get used in practice, but I cannot prove that.  I'm ok with
> removing
> > the option, let me know what you think.
> >
> > -Artem
> >
> > On Mon, Feb 28, 2022 at 2:06 PM Jun Rao 
> wrote:
> >
> > > Hi, Artem,
> > >
> > > Thanks for the reply. A few more comments.
> > >
> > > 1. Since we control the implementation and the usage of
> > DefaultPartitioner,
> > > another way is to instantiate the DefaultPartitioner with a special
> > > constructor, which allows it to have more access to internal
> information.
> > > Then we could just change the behavior of  DefaultPartitioner such that
> > it
> > > can use the internal infoamtion when choosing the partition. This seems
> > > more intuitive than having DefaultPartitioner return -1 partition.
> > >
> > > 2. I guess partitioner.sticky.batch.size is introduced because the
> > > effective batch size could be less than batch.size and we want to align
> > > partition switching with the effective batch size. How would a user
> know
> > > the effective batch size to set partitioner.sticky.batch.size properly?
> > If
> > > the user somehow knows the effective batch size, does setting
> batch.size
> > to
> > > the effective batch size achieve the same result?
> > >
> > > 4. Thanks for the explanation. Makes sense to me.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Feb 25, 2022 at 8:26 PM Artem Livshits
> > >  wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > 1. Updated the KIP to add a couple paragraphs about implementation
> > > > necessities in the Proposed Changes section.
> > > >
> > > > 2. Sorry if my reply was confusing, what I meant to say (and I
> > elaborated
> > > > on that in point #3) is that there could be patterns for which 16KB
> > > > wouldn't be the most effective setting, thus it would be good to make
> > it
> > > > configurable.
> > > >
> > > > 4. We could use broker readiness timeout.  But I'm not sure it would
> > > > correctly model the broker load.  The problem is that latency is not
> an
> > > > accurate measure of throughput, we could have 2 brokers that have
> equal
> > > > throughput but one has higher latency (so it takes larger batches
> less
> > > > frequently, but still takes the same load).  Latency-based logic is
> > > likely
> > > > to send less data to the broker with higher latency.  Using the queue
> > > size
> > > > would adapt to throughput, regardless of latency (which could be
> just a
> > > > result of deployment topology), so that's the model chosen in the
> > > > proposal.  The partition.availability.timeout.ms logic approaches
> the
> > > > model
> > > > from a slightly different angle, say we have a requirement to deliver
> > > > messages via brokers that have a certain latency, then
> > > > partition.availability.timeout.ms could be used to tune that.
> Latency
> > > is
> > > > a
> > > > much more volatile metric than throughput (latency depends on
> external
> > > > load, on capacity, on deployment topology, on jitter in network, on
> > > jitter
> > > > in disk, etc.) and I think it would be best to leave latency-based
> > > > thresholds configurable to tune for the environment.
> > > >
> > > > -Artem
> > > >
> > > > On Wed, Feb 23, 2022 at 11:14 AM Jun Rao 
> > > wrote:
> > > >
> > > > > Hi, Artem,
> > > > >
> > > > > Thanks for the reply. A few more comments.
> > > > >
> > > > > 1. Perhaps you could elaborate 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #736

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #735

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 510374 lines...]
[2022-03-04T00:19:01.616Z] 1 warning
[2022-03-04T00:19:01.616Z] 
[2022-03-04T00:19:01.616Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-03-04T00:19:01.616Z] > Task :connect:api:jar UP-TO-DATE
[2022-03-04T00:19:01.616Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-03-04T00:19:01.616Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-03-04T00:19:01.616Z] > Task :connect:json:jar UP-TO-DATE
[2022-03-04T00:19:01.616Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-03-04T00:19:01.616Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-03-04T00:19:01.616Z] > Task :connect:json:publishToMavenLocal
[2022-03-04T00:19:01.616Z] > Task :connect:api:javadocJar
[2022-03-04T00:19:01.616Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-03-04T00:19:01.616Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-03-04T00:19:01.616Z] > Task :connect:api:testJar
[2022-03-04T00:19:02.552Z] > Task :connect:api:testSrcJar
[2022-03-04T00:19:02.552Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-03-04T00:19:02.552Z] > Task :connect:api:publishToMavenLocal
[2022-03-04T00:19:05.423Z] 
[2022-03-04T00:19:05.423Z] > Task :streams:javadoc
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-03-04T00:19:05.423Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:74:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:110:
 warning - Tag @link: reference not found: this#getResult()
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-03-04T00:19:05.423Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-03-04T00:19:05.424Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isFailure()
[2022-03-04T00:19:06.360Z] 10 warnings
[2022-03-04T00:19:07.555Z] 
[2022-03-04T00:19:07.555Z] > Task :streams:javadocJar
[2022-03-04T00:19:11.303Z] 
[2022-03-04T00:19:11.303Z] > Task :clients:javadoc
[2022-03-04T00:19:11.303Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-03-04T00:19:12.238Z] 1 warning
[2022-03-04T00:19:13.266Z] 
[2022-03-04T00:19:13.266Z] > Task :clients:javadocJar
[2022-03-04T00:19:15.019Z] 
[2022-03-04T00:19:15.019Z] > Task :clients:srcJar
[2022-03-04T00:19:15.019Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-03-04T00:19:15.019Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without 

[DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

[Sergio Daniel Troiano]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-824%3A+Allowing+dumping+segmentlogs+limiting+the+batches+in+the+output

[DISCUSS] KIP-824 Allowing dumping segmentlogs limiting the batches in the output

[Sergio Daniel Troiano]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-824%3A+Allowing+dumping+segmentlogs+limiting+the+batches+in+the+output

[jira] [Created] (KAFKA-13707) Refactor - unify the naming convention of in-flight

[Arvin Zheng (Jira)]

Arvin Zheng created KAFKA-13707:
---

 Summary: Refactor - unify the naming convention of in-flight 
 Key: KAFKA-13707
 URL: https://issues.apache.org/jira/browse/KAFKA-13707
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Arvin Zheng


Methods and variables that have in-flight in them are named in different 
styles, some treats in-flight as a single word while some doesn't, e.g. 
inFlight vs inflight or InFlight vs Inflight.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Jun Rao]

Hi, Artem,

Thanks for the reply.

1. Sounds good.

2. If we don't expect users to change it, we probably could just leave out
the new config. In general, it's easy to add a new config, but hard to
remove an existing config.

20. The two new configs enable.adaptive.partitioning and
partition.availability.timeout.ms only apply to the two built-in
partitioners DefaultPartitioner and UniformStickyPartitioner, right? It
would be useful to document that in the KIP.

Thanks,

Jun

On Thu, Mar 3, 2022 at 9:47 AM Artem Livshits
 wrote:

> Hi Jun,
>
> Thank you for the suggestions.
>
> 1. As we discussed offline, we can hardcode the logic for
> DefaultPartitioner and UniformStickyPartitioner in the KafkaProducer (i.e.
> the DefaultPartitioner.partition won't get called, instead KafkaProducer
> would check if the partitioner is an instance of DefaultPartitioner and
> then run the actual partitioning logic itself).  Then the change to the
> Partitioner wouldn't be required.  I'll update the KIP to reflect that.
>
> 2. I don't expect users to change this too often, as changing it would
> require a bit of studying of the production patterns.  As a general
> principle, if I can think of a model that requires a deviation from
> default, I tend to add a configuration option.  It could be that it'll
> never get used in practice, but I cannot prove that.  I'm ok with removing
> the option, let me know what you think.
>
> -Artem
>
> On Mon, Feb 28, 2022 at 2:06 PM Jun Rao  wrote:
>
> > Hi, Artem,
> >
> > Thanks for the reply. A few more comments.
> >
> > 1. Since we control the implementation and the usage of
> DefaultPartitioner,
> > another way is to instantiate the DefaultPartitioner with a special
> > constructor, which allows it to have more access to internal information.
> > Then we could just change the behavior of  DefaultPartitioner such that
> it
> > can use the internal infoamtion when choosing the partition. This seems
> > more intuitive than having DefaultPartitioner return -1 partition.
> >
> > 2. I guess partitioner.sticky.batch.size is introduced because the
> > effective batch size could be less than batch.size and we want to align
> > partition switching with the effective batch size. How would a user know
> > the effective batch size to set partitioner.sticky.batch.size properly?
> If
> > the user somehow knows the effective batch size, does setting batch.size
> to
> > the effective batch size achieve the same result?
> >
> > 4. Thanks for the explanation. Makes sense to me.
> >
> > Thanks,
> >
> > Jun
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Feb 25, 2022 at 8:26 PM Artem Livshits
> >  wrote:
> >
> > > Hi Jun,
> > >
> > > 1. Updated the KIP to add a couple paragraphs about implementation
> > > necessities in the Proposed Changes section.
> > >
> > > 2. Sorry if my reply was confusing, what I meant to say (and I
> elaborated
> > > on that in point #3) is that there could be patterns for which 16KB
> > > wouldn't be the most effective setting, thus it would be good to make
> it
> > > configurable.
> > >
> > > 4. We could use broker readiness timeout.  But I'm not sure it would
> > > correctly model the broker load.  The problem is that latency is not an
> > > accurate measure of throughput, we could have 2 brokers that have equal
> > > throughput but one has higher latency (so it takes larger batches less
> > > frequently, but still takes the same load).  Latency-based logic is
> > likely
> > > to send less data to the broker with higher latency.  Using the queue
> > size
> > > would adapt to throughput, regardless of latency (which could be just a
> > > result of deployment topology), so that's the model chosen in the
> > > proposal.  The partition.availability.timeout.ms logic approaches the
> > > model
> > > from a slightly different angle, say we have a requirement to deliver
> > > messages via brokers that have a certain latency, then
> > > partition.availability.timeout.ms could be used to tune that.  Latency
> > is
> > > a
> > > much more volatile metric than throughput (latency depends on external
> > > load, on capacity, on deployment topology, on jitter in network, on
> > jitter
> > > in disk, etc.) and I think it would be best to leave latency-based
> > > thresholds configurable to tune for the environment.
> > >
> > > -Artem
> > >
> > > On Wed, Feb 23, 2022 at 11:14 AM Jun Rao 
> > wrote:
> > >
> > > > Hi, Artem,
> > > >
> > > > Thanks for the reply. A few more comments.
> > > >
> > > > 1. Perhaps you could elaborate a bit more on how the producer
> > determines
> > > > the partition if the partitioner returns -1. This will help
> understand
> > > why
> > > > encapsulating that logic as a partitioner is not clean.
> > > >
> > > > 2. I am not sure that I understand this part. If 15.5KB is more
> > > efficient,
> > > > could we just set batch.size to 15.5KB?
> > > >
> > > > 4. Yes, we could add a switch (or a variant of the partitioner) for
> > > > enabling this behavior. Also, choosing 

[jira] [Created] (KAFKA-13706) org.apache.kafka.test.MockSelector doesn't remove closed connections from its 'ready' field

[Vincent Jiang (Jira)]

Vincent Jiang created KAFKA-13706:
-

 Summary: org.apache.kafka.test.MockSelector doesn't remove closed 
connections from its 'ready' field
 Key: KAFKA-13706
 URL: https://issues.apache.org/jira/browse/KAFKA-13706
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Reporter: Vincent Jiang


MockSelector.close(String id) method doesn't remove closed connection from 
"ready" field.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Spotting stale KIPs with a better "Under Discussion" table

[Kirk True]

Since this is on GitHub, you might look into GitHub Actions to implement the 
cron job to update the table.

Thanks!

On Wed, Mar 2, 2022, at 10:27 AM, Thomas Cooper wrote:
> Hi All,
> 
> I am hoping to get more involved in the upstream Kafka community. To that 
> end, I was trying to keep up with the KIPs that were currently under 
> discussion. However, I found it hard to keep track of what was and wasn't 
> being discussed and the progress they were making. Some KIPs appeared 
> abandoned but will still classed as "Under Discussion".
> 
> So, during a very rainy week on holiday, I created a tool (which I called 
> KIPper [[1](https://github.com/tomncooper/kipper)]) to parse the dev mailing 
> list archive and extract all KIP mentions. I paired this with information 
> parsed from the confluence (wiki) API to create an enriched table of the KIPs 
> Under Discussion [[2](https://tomncooper.github.io/kipper/)].
> 
> The table shows a "Status" for each KIP, which is based on the last time the 
> KIP was mentioned in the subject line of an email on the dev mailing list. 
> Green for within the last month, yellow for the last 3 months and red for 
> within the last year. If the status is black then it hasn't been mentioned in 
> over a year.
> 
> I also added vote information, but this is only indicative as it is based on 
> parsing the non-reply lines (without ">" in) of the email bodies so could 
> hold false positives.
> 
> In the spirit of the discussion on closing stale PRs 
> [[3](https://lists.apache.org/thread/66yj9m6tcyz8zqb3lqlbnr386bqwsopt)], it 
> might be a good idea to introduce a new KIP "state" alongside "Under 
> Discussion", "Accepted" and "Rejected" (and their numerous variants 
> [[4](https://github.com/tomncooper/kipper/blob/0bbb5595e79a9e075b0d2dc907c84693734d7846/kipper/wiki.py#L54)]).
>  Maybe a KIP with a black status and no votes could be moved to a "Stale" or 
> "Rejected" state?
> 
> The kipper page is statically generated at the moment so could be updated 
> every day with a cron job. The data used to create the page could also be 
> used to drive automation, perhaps emailing the KIPs author once a KIP hits 
> "Red" status and then automatically setting the state to stale once it turns 
> "Black"?
> 
> Anyway, I learned a lot making the tool and I now feel I have a better handle 
> on the state of various KIPs. I hope others find it useful. There is loads of 
> information to be harvested from the mailing list and wiki APIs so if any one 
> has any feature requests please post issues on the GH page. I had one 
> suggestion of performing sentiment analysis on the email bodies related to 
> each KIP, to get a feel of how the KIP was being received. But maybe that is 
> a step too far..
> 
> Cheers,
> 
> [1] https://github.com/tomncooper/kipper
> [2] https://tomncooper.github.io/kipper/
> [3] https://lists.apache.org/thread/66yj9m6tcyz8zqb3lqlbnr386bqwsopt
> [4] 
> https://github.com/tomncooper/kipper/blob/0bbb5595e79a9e075b0d2dc907c84693734d7846/kipper/wiki.py#L54
> 
> Tom Cooper
> 
> [@tomncooper](https://twitter.com/tomncooper) | https://tomcooper.dev

Re: Spotting stale KIPs with a better "Under Discussion" table

[Kirk True]

Hi Tom,

This is great! I've just begun reading through the KIPs and I've noticed the 
status on the KIP list and on the KIP itself isn't always up-to-date.

It would be great to have a "consistency checker" to verify high-level status 
details are consistent and up-to-date.

Great work!

Thanks,
Kirk

On Wed, Mar 2, 2022, at 10:49 PM, Luke Chen wrote:
> Hi Tom,
> 
> Thanks for the nice work!
> This is very helpful to understand the status of each KIP!
> And thanks to this tool, it reminds me of a KIP that needs one more vote :)
> 
> I agree we can automatically change the state of "black" KIPs, and move
> them into the "Discarded KIPs" or "Dormant/inactive KIPs" table
> 
> with notification to the KIP authors.
> And of course, the KIP authors can always revive those KIPs anytime.
> 
> However, I found there is a small bug that it would take the "draft" KIP as
> inactive KIP.
> I have filed into the github issue page.
> 
> Thank you.
> Luke
> 
> 
> 
> 
> 
> On Thu, Mar 3, 2022 at 2:28 AM Thomas Cooper  wrote:
> 
> > Hi All,
> >
> > I am hoping to get more involved in the upstream Kafka community. To that
> > end, I was trying to keep up with the KIPs that were currently under
> > discussion. However, I found it hard to keep track of what was and wasn't
> > being discussed and the progress they were making. Some KIPs appeared
> > abandoned but will still classed as "Under Discussion".
> >
> > So, during a very rainy week on holiday, I created a tool (which I called
> > KIPper [[1](https://github.com/tomncooper/kipper)]) to parse the dev
> > mailing list archive and extract all KIP mentions. I paired this with
> > information parsed from the confluence (wiki) API to create an enriched
> > table of the KIPs Under Discussion [[2](
> > https://tomncooper.github.io/kipper/)].
> >
> > The table shows a "Status" for each KIP, which is based on the last time
> > the KIP was mentioned in the subject line of an email on the dev mailing
> > list. Green for within the last month, yellow for the last 3 months and red
> > for within the last year. If the status is black then it hasn't been
> > mentioned in over a year.
> >
> > I also added vote information, but this is only indicative as it is based
> > on parsing the non-reply lines (without ">" in) of the email bodies so
> > could hold false positives.
> >
> > In the spirit of the discussion on closing stale PRs [[3](
> > https://lists.apache.org/thread/66yj9m6tcyz8zqb3lqlbnr386bqwsopt)], it
> > might be a good idea to introduce a new KIP "state" alongside "Under
> > Discussion", "Accepted" and "Rejected" (and their numerous variants [[4](
> > https://github.com/tomncooper/kipper/blob/0bbb5595e79a9e075b0d2dc907c84693734d7846/kipper/wiki.py#L54)]).
> > Maybe a KIP with a black status and no votes could be moved to a "Stale" or
> > "Rejected" state?
> >
> > The kipper page is statically generated at the moment so could be updated
> > every day with a cron job. The data used to create the page could also be
> > used to drive automation, perhaps emailing the KIPs author once a KIP hits
> > "Red" status and then automatically setting the state to stale once it
> > turns "Black"?
> >
> > Anyway, I learned a lot making the tool and I now feel I have a better
> > handle on the state of various KIPs. I hope others find it useful. There is
> > loads of information to be harvested from the mailing list and wiki APIs so
> > if any one has any feature requests please post issues on the GH page. I
> > had one suggestion of performing sentiment analysis on the email bodies
> > related to each KIP, to get a feel of how the KIP was being received. But
> > maybe that is a step too far..
> >
> > Cheers,
> >
> > [1] https://github.com/tomncooper/kipper
> > [2] https://tomncooper.github.io/kipper/
> > [3] https://lists.apache.org/thread/66yj9m6tcyz8zqb3lqlbnr386bqwsopt
> > [4]
> > https://github.com/tomncooper/kipper/blob/0bbb5595e79a9e075b0d2dc907c84693734d7846/kipper/wiki.py#L54
> >
> > Tom Cooper
> >
> > [@tomncooper](https://twitter.com/tomncooper) | https://tomcooper.dev
> 

[VOTE] 3.0.1 RC0

[Mickael Maison]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.0.1.

Apache Kafka 3.0.1 is a bugfix release and 29 issues have been fixed
since 3.0.0.

Release notes for the 3.0.1 release:
https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Thursday, March 10, 6pm GMT ***

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~mimaison/kafka-3.0.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~mimaison/kafka-3.0.1-rc0/javadoc/

* Tag to be voted upon (off 3.0 branch) is the 3.0.1 tag:
https://github.com/apache/kafka/releases/tag/3.0.1-rc0

* Documentation:
https://kafka.apache.org/30/documentation.html

* Protocol:
https://kafka.apache.org/30/protocol.html

* Successful Jenkins builds for the 3.0 branch:
I'll share a link once the build complete

/**

Thanks,
Mickael

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Artem Livshits]

Hi Jun,

Thank you for the suggestions.

1. As we discussed offline, we can hardcode the logic for
DefaultPartitioner and UniformStickyPartitioner in the KafkaProducer (i.e.
the DefaultPartitioner.partition won't get called, instead KafkaProducer
would check if the partitioner is an instance of DefaultPartitioner and
then run the actual partitioning logic itself).  Then the change to the
Partitioner wouldn't be required.  I'll update the KIP to reflect that.

2. I don't expect users to change this too often, as changing it would
require a bit of studying of the production patterns.  As a general
principle, if I can think of a model that requires a deviation from
default, I tend to add a configuration option.  It could be that it'll
never get used in practice, but I cannot prove that.  I'm ok with removing
the option, let me know what you think.

-Artem

On Mon, Feb 28, 2022 at 2:06 PM Jun Rao  wrote:

> Hi, Artem,
>
> Thanks for the reply. A few more comments.
>
> 1. Since we control the implementation and the usage of DefaultPartitioner,
> another way is to instantiate the DefaultPartitioner with a special
> constructor, which allows it to have more access to internal information.
> Then we could just change the behavior of  DefaultPartitioner such that it
> can use the internal infoamtion when choosing the partition. This seems
> more intuitive than having DefaultPartitioner return -1 partition.
>
> 2. I guess partitioner.sticky.batch.size is introduced because the
> effective batch size could be less than batch.size and we want to align
> partition switching with the effective batch size. How would a user know
> the effective batch size to set partitioner.sticky.batch.size properly? If
> the user somehow knows the effective batch size, does setting batch.size to
> the effective batch size achieve the same result?
>
> 4. Thanks for the explanation. Makes sense to me.
>
> Thanks,
>
> Jun
>
> Thanks,
>
> Jun
>
> On Fri, Feb 25, 2022 at 8:26 PM Artem Livshits
>  wrote:
>
> > Hi Jun,
> >
> > 1. Updated the KIP to add a couple paragraphs about implementation
> > necessities in the Proposed Changes section.
> >
> > 2. Sorry if my reply was confusing, what I meant to say (and I elaborated
> > on that in point #3) is that there could be patterns for which 16KB
> > wouldn't be the most effective setting, thus it would be good to make it
> > configurable.
> >
> > 4. We could use broker readiness timeout.  But I'm not sure it would
> > correctly model the broker load.  The problem is that latency is not an
> > accurate measure of throughput, we could have 2 brokers that have equal
> > throughput but one has higher latency (so it takes larger batches less
> > frequently, but still takes the same load).  Latency-based logic is
> likely
> > to send less data to the broker with higher latency.  Using the queue
> size
> > would adapt to throughput, regardless of latency (which could be just a
> > result of deployment topology), so that's the model chosen in the
> > proposal.  The partition.availability.timeout.ms logic approaches the
> > model
> > from a slightly different angle, say we have a requirement to deliver
> > messages via brokers that have a certain latency, then
> > partition.availability.timeout.ms could be used to tune that.  Latency
> is
> > a
> > much more volatile metric than throughput (latency depends on external
> > load, on capacity, on deployment topology, on jitter in network, on
> jitter
> > in disk, etc.) and I think it would be best to leave latency-based
> > thresholds configurable to tune for the environment.
> >
> > -Artem
> >
> > On Wed, Feb 23, 2022 at 11:14 AM Jun Rao 
> wrote:
> >
> > > Hi, Artem,
> > >
> > > Thanks for the reply. A few more comments.
> > >
> > > 1. Perhaps you could elaborate a bit more on how the producer
> determines
> > > the partition if the partitioner returns -1. This will help understand
> > why
> > > encapsulating that logic as a partitioner is not clean.
> > >
> > > 2. I am not sure that I understand this part. If 15.5KB is more
> > efficient,
> > > could we just set batch.size to 15.5KB?
> > >
> > > 4. Yes, we could add a switch (or a variant of the partitioner) for
> > > enabling this behavior. Also, choosing partitions based on broker
> > readiness
> > > can be made in a smoother way. For example, we could track the last
> time
> > a
> > > broker has drained any batches from the accumulator. We can then select
> > > partitions from brokers proportionally to the inverse of that time.
> This
> > > seems smoother than a cutoff based on a
> > partition.availability.timeout.ms
> > >  threshold.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Feb 18, 2022 at 5:14 PM Artem Livshits
> > >  wrote:
> > >
> > > > Hello Luke, Jun,
> > > >
> > > > Thank you for your feedback.  I've added the Rejected Alternative
> > section
> > > > that may clarify some of the questions w.r.t. returning -1.
> > > >
> > > > 1. I've elaborated on the -1 in the KIP.  The problem is 

Re: Proposal to add IBM Power (ppc64le) to apache kafka jenkins CI

[Mickael Maison]

Hi Abhijit,

Nobody raised concerns so I went ahead an opened a PR to add ppc64le
to the Kafka CI:
https://github.com/apache/kafka/pull/11833

As a starter I reused the same settings as the ARM build (only running
unitTest) and confirmed it's building fine. We can reconsider later if
we also want to run integrationTest.

Thanks,
Mickael

On Thu, Feb 10, 2022 at 9:10 AM Abhijit Mane  wrote:
>
> Hi Mickael,
>
> Thanks for validating the ppc64le VM spec for suitability of kafka builds.
>
> If there are no more comments, can I proceed with working with Infra
> on further steps?
> https://issues.apache.org/jira/browse/INFRA-22612
>
> Regards,
> Abhijit
>
> On Wed, Jan 26, 2022 at 8:40 PM Mickael Maison  
> wrote:
> >
> > Hi Abhijit,
> >
> > Thanks for offering a ppc64le VM. Looking at the INFRA ticket, it
> > looks like it was added to Jenkins successfully.
> > The spec looks appropriate for Kafka builds.
> >
> > Let's wait a few days to see if anybody has questions or concerns.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Mon, Jan 24, 2022 at 11:03 AM Abhijit Mane  wrote:
> > >
> > > Hello,
> > >
> > > We would like to work with the community to enable IBM Power (ppc64le)
> > > architecture support for Apache Jenkins CI so we can enable Apache
> > > Kafka on IBM Power systems. An IBM Power (ppc64le) VM available to the
> > > community for integrating into Jenkins cluster. The VM has also been
> > > set up as advised by Infra using the script below and verified.
> > > https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md
> > >
> > > ---
> > > Ubuntu 20.04 VM
> > > Root vol: 100GB,  Data vol: 500GB, mounted at /home/jenkins
> > > 8 cores / 16GB RAM
> > > Infra discussion with community:
> > > https://issues.apache.org/jira/browse/INFRA-22612
> > > ---
> > >
> > > Kindly review Jira to let me know if you have any questions/concerns
> > > and if it's okay to proceed further.
> > >
> > > Regards,
> > > Abhijit
> > >
> >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #82

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-714: Client Metrics and Observability

[Julien Chanaud]

+1
As a member of a team which operates several Kafka clusters, I am
unequipped when it comes to troubleshooting issues with project teams
that did not understand the importance of configuring client-side
monitoring.
Kafka represents a fraction of their work and they don't have enough
experience, time or interest in trying to understand the meaning behind
every metric.

I stand 100% behind what Colin stated back in June in the Discuss thread :

> Magnus and I explained a few times the reasons why it does matter. Within
> most organizations, there are usually several teams using clients, which
> are separate from the team which maintains the Kafka cluster. The Kafka
> team has the Kafka experts, which makes it the best place to centralize
> collecting and analyzing Kafka metrics.


Thanks for this KIP.

Le mer. 26 janv. 2022 à 16:01, rifer...@riferrei.com 
a écrit :

> +1
>
> I think this KIP solves a problem that has been around for some time with
> Kafka deployments, which is the ability to assess the current state of a
> Kafka architecture but looking at the whole picture. I also share other
> folks' concerns regarding adding runtime dependencies to the clients; this
> may be problematic for large deployments. Still, I think it is worth
> refactoring.
>
> IMHO, it is a fair trade-off.
>
> — Ricardo
>
> > On Jan 26, 2022, at 9:34 AM, Magnus Edenhill  wrote:
> >
> > Hi all,
> >
> > it's been a while and there's been some more discussions of the KIP which
> > have been
> > addressed on the KIP page.
> >
> > I think it's a good time to revive this vote thread and get things
> moving.
> >
> > We're currently at +3 (non-binding) and -1 (non-binding) votes.
> >
> > Regards,
> > Magnus
> >
> >
> > Den mån 1 nov. 2021 kl 21:19 skrev J Rivers :
> >
> >> +1
> >>
> >> Thank you for the KIP!
> >>
> >> Our organization runs kafka at large scale in a multi-tenant
> configuration.
> >> We actually have many other enterprises connecting up to our system to
> >> retrieve stream data. These feeds vary greatly in volume and velocity.
> The
> >> peak rates are a multiplicative factor of the nominal.  There is extreme
> >> skew in our datasets in a number of ways.
> >>
> >> We don't have time to work with every new internal/external client to
> tune
> >> their feeds. They need to be able to take one of the many kafka clients
> and
> >> go off to the races.
> >>
> >> Being able to retrieve client metrics would be invaluable here as it's
> hard
> >> and time consuming to communicate out of the enterprise walls.
> >>
> >> This KIP is important to us to expand the use of our datasets internally
> >> and outside the borders of the enterprise. Our clients like the
> performance
> >> and data safeties related to the kafka connection. The observability has
> >> been a problem...
> >>
> >> Jonathan Rivers
> >> jrivers...@gmail.com
> >>
> >>
> >>
> >>
> >> On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan 
> >> wrote:
> >>
> >>> -1
> >>>
> >>> Ryanne
> >>>
> >>> On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill 
> >> wrote:
> >>>
>  Hi all,
> 
>  I'd like to start a vote on KIP-714.
>  https://cwiki.apache.org/confluence/x/2xRRCg
> 
>  Discussion thread:
>  https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html
> 
>  Thanks,
>  Magnus
> 
> >>>
> >>
>
>

[jira] [Resolved] (KAFKA-13510) KIP-769: Connect APIs to list all connector plugins and retrieve their configuration definitions

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13510.

Resolution: Fixed

> KIP-769: Connect APIs to list all connector plugins and retrieve their 
> configuration definitions
> 
>
> Key: KAFKA-13510
> URL: https://issues.apache.org/jira/browse/KAFKA-13510
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 3.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #733

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[Luke Chen]

Thanks, Ziming!

So, now, this KIP vote passed with 3 binding +1 votes (David, Tom,
Guozhang) and 1 non-binding +1 vote (Ziming).
The vote will be closed.

Thanks again.
Luke

On Thu, Mar 3, 2022 at 5:00 PM deng ziming  wrote:

> Thank you Luke for this work,
> I’m +1(non-binding)
>
> --
> Best,
> Ziming Deng
>
> > On Dec 1, 2021, at 8:36 AM, Luke Chen  wrote:
> >
> > Hi all,
> >
> > I'd like to start the vote for KIP-792: Add "generation" field into
> > consumer protocol.
> >
> > The goal of this KIP is to allow the assignor/consumer coordinator to
> have
> > a way to identify the out-of-date members/assignments, to avoid rebalance
> > stuck issues in current protocol.
> >
> > Detailed description can be found here:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
>
>

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[deng ziming]

Thank you Luke for this work,
I’m +1(non-binding)

--
Best,
Ziming Deng

> On Dec 1, 2021, at 8:36 AM, Luke Chen  wrote:
> 
> Hi all,
> 
> I'd like to start the vote for KIP-792: Add "generation" field into
> consumer protocol.
> 
> The goal of this KIP is to allow the assignor/consumer coordinator to have
> a way to identify the out-of-date members/assignments, to avoid rebalance
> stuck issues in current protocol.
> 
> Detailed description can be found here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> 
> Any feedback is welcome.
> 
> Thank you.
> Luke

[jira] [Created] (KAFKA-13705) CoreUtils.swallow uses CoreUtils.logger but not logging parameter

[Kezhu Wang (Jira)]

Kezhu Wang created KAFKA-13705:
--

 Summary: CoreUtils.swallow uses CoreUtils.logger but not logging 
parameter
 Key: KAFKA-13705
 URL: https://issues.apache.org/jira/browse/KAFKA-13705
 Project: Kafka
  Issue Type: Bug
  Components: logging
Reporter: Kezhu Wang


# Javadoc says: {noformat}logging The logging instance to use for logging the 
thrown exception.{noformat}
# But {{logging}} parameter it is never used.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[Luke Chen]

Thanks David!

So, with 3 binding +1 votes (David, Tom, Guozhang), the vote passes.

Thanks everyone!

Luke

On Thu, Mar 3, 2022 at 4:34 PM David Jacot 
wrote:

> +1 (binding). Thanks for the KIP!
>
> On Fri, Jan 28, 2022 at 6:13 PM Tom Bentley  wrote:
> >
> > Hi Luke,
> >
> > Thanks for the KIP, +1 (binding).
> >
> > Kind regards,
> >
> > Tom
> >
> > On Wed, 19 Jan 2022 at 13:16, Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > Bump this thread to see if there are other comments to this KIP.
> > > So far, I have one +1 vote (binding), and need more votes.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Tue, Dec 21, 2021 at 10:33 AM Luke Chen  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Bump this thread to see if there are other comments to this KIP.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Fri, Dec 17, 2021 at 1:27 AM Colin McCabe 
> wrote:
> > > >
> > > >> Thanks for the explanation, Luke. That makes sense.
> > > >>
> > > >> best,
> > > >> Colin
> > > >>
> > > >> On Thu, Dec 9, 2021, at 13:31, Guozhang Wang wrote:
> > > >> > Thanks Luke, in that case I'm +1 on this KIP.
> > > >> >
> > > >> > On Thu, Dec 9, 2021 at 1:46 AM Luke Chen 
> wrote:
> > > >> >
> > > >> >> Hi Guozhang,
> > > >> >>
> > > >> >> Thanks for your comment.
> > > >> >>
> > > >> >> > we need to make sure the old-versioned leader would be able to
> > > >> ignore the
> > > >> >> new
> > > >> >> field during an upgrade e.g. without crashing.
> > > >> >>
> > > >> >> Yes, I understand. I'll be careful to make sure it won't crash
> the
> > > old
> > > >> >> versioned leader.
> > > >> >> But basically, it won't, because we appended the new field into
> the
> > > >> last of
> > > >> >> the ConsumerProtocolSubscription, which means, when
> read/deserialize
> > > >> the
> > > >> >> Subscription metadata, the old versioned leader will just read
> the
> > > head
> > > >> >> part of the data.
> > > >> >>
> > > >> >> Thanks for the reminder!
> > > >> >>
> > > >> >> Luke
> > > >> >>
> > > >> >> On Thu, Dec 9, 2021 at 4:00 AM Guozhang Wang  >
> > > >> wrote:
> > > >> >>
> > > >> >> > Hi Luke,
> > > >> >> >
> > > >> >> > Thanks for the KIP.
> > > >> >> >
> > > >> >> > One thing I'd like to double check is that, since the
> > > >> >> > ConsumerProtocolSubscription is not auto generated from the
> json
> > > >> file, we
> > > >> >> > need to make sure the old-versioned leader would be able to
> ignore
> > > >> the
> > > >> >> new
> > > >> >> > field during an upgrade e.g. without crashing. Other than
> that, the
> > > >> KIP
> > > >> >> > lgtm.
> > > >> >> >
> > > >> >> > Guozhang
> > > >> >> >
> > > >> >> > On Tue, Dec 7, 2021 at 6:16 PM Luke Chen 
> > > wrote:
> > > >> >> >
> > > >> >> > > Hi Colin,
> > > >> >> > >
> > > >> >> > > I'm not quite sure if I understand your thoughts correctly.
> > > >> >> > > If I was wrong, please let me know.
> > > >> >> > >
> > > >> >> > > Also, I'm not quite sure how I could lock this feature to a
> new
> > > IBP
> > > >> >> > > version.
> > > >> >> > > I saw "KIP-584: Versioning scheme for features" is still
> under
> > > >> >> > development.
> > > >> >> > > Not sure if I need to lock the IBP version, how should I do?
> > > >> >> > >
> > > >> >> > > Thank you.
> > > >> >> > > Luke
> > > >> >> > >
> > > >> >> > > On Tue, Dec 7, 2021 at 9:41 PM Luke Chen 
> > > >> wrote:
> > > >> >> > >
> > > >> >> > > > Hi Colin,
> > > >> >> > > >
> > > >> >> > > > Thanks for your comments. I've updated the KIP to mention
> about
> > > >> the
> > > >> >> KIP
> > > >> >> > > > won't affect current broker side behavior.
> > > >> >> > > >
> > > >> >> > > > > One scenario that we need to consider is what happens
> during
> > > a
> > > >> >> > rolling
> > > >> >> > > > upgrade. If the coordinator moves back and forth between
> > > brokers
> > > >> with
> > > >> >> > > > different IBPs, it seems that the same epoch numbers could
> be
> > > >> reused
> > > >> >> > for
> > > >> >> > > a
> > > >> >> > > > group, if things are done in the obvious manner (old IBP =
> > > don't
> > > >> read
> > > >> >> > or
> > > >> >> > > > write epoch, new IBP = do)
> > > >> >> > > >
> > > >> >> > > > I think this KIP doesn't care about the group epoch number
> at
> > > >> all.
> > > >> >> The
> > > >> >> > > > subscription metadata is passed from each member to group
> > > >> >> coordinator,
> > > >> >> > > and
> > > >> >> > > > then the group coordinator pass all of them back to the
> > > consumer
> > > >> >> lead.
> > > >> >> > So
> > > >> >> > > > even if the epoch number is reused in a group, it should be
> > > >> fine. On
> > > >> >> > the
> > > >> >> > > > other hand, the group coordinator will have no idea if the
> join
> > > >> group
> > > >> >> > > > request sent from consumer containing the new subscription
> > > >> >> "generation"
> > > >> >> > > > field or not, because group coordinator won't deserialize
> the
> > > >> >> metadata.
> > > >> >> > > >
> > > >> >> > > > I've added also added them into the KIP.
> > > >> >> > > >
> > > 

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[David Jacot]

+1 (binding). Thanks for the KIP!

On Fri, Jan 28, 2022 at 6:13 PM Tom Bentley  wrote:
>
> Hi Luke,
>
> Thanks for the KIP, +1 (binding).
>
> Kind regards,
>
> Tom
>
> On Wed, 19 Jan 2022 at 13:16, Luke Chen  wrote:
>
> > Hi all,
> >
> > Bump this thread to see if there are other comments to this KIP.
> > So far, I have one +1 vote (binding), and need more votes.
> >
> > Thank you.
> > Luke
> >
> > On Tue, Dec 21, 2021 at 10:33 AM Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > Bump this thread to see if there are other comments to this KIP.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, Dec 17, 2021 at 1:27 AM Colin McCabe  wrote:
> > >
> > >> Thanks for the explanation, Luke. That makes sense.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >> On Thu, Dec 9, 2021, at 13:31, Guozhang Wang wrote:
> > >> > Thanks Luke, in that case I'm +1 on this KIP.
> > >> >
> > >> > On Thu, Dec 9, 2021 at 1:46 AM Luke Chen  wrote:
> > >> >
> > >> >> Hi Guozhang,
> > >> >>
> > >> >> Thanks for your comment.
> > >> >>
> > >> >> > we need to make sure the old-versioned leader would be able to
> > >> ignore the
> > >> >> new
> > >> >> field during an upgrade e.g. without crashing.
> > >> >>
> > >> >> Yes, I understand. I'll be careful to make sure it won't crash the
> > old
> > >> >> versioned leader.
> > >> >> But basically, it won't, because we appended the new field into the
> > >> last of
> > >> >> the ConsumerProtocolSubscription, which means, when read/deserialize
> > >> the
> > >> >> Subscription metadata, the old versioned leader will just read the
> > head
> > >> >> part of the data.
> > >> >>
> > >> >> Thanks for the reminder!
> > >> >>
> > >> >> Luke
> > >> >>
> > >> >> On Thu, Dec 9, 2021 at 4:00 AM Guozhang Wang 
> > >> wrote:
> > >> >>
> > >> >> > Hi Luke,
> > >> >> >
> > >> >> > Thanks for the KIP.
> > >> >> >
> > >> >> > One thing I'd like to double check is that, since the
> > >> >> > ConsumerProtocolSubscription is not auto generated from the json
> > >> file, we
> > >> >> > need to make sure the old-versioned leader would be able to ignore
> > >> the
> > >> >> new
> > >> >> > field during an upgrade e.g. without crashing. Other than that, the
> > >> KIP
> > >> >> > lgtm.
> > >> >> >
> > >> >> > Guozhang
> > >> >> >
> > >> >> > On Tue, Dec 7, 2021 at 6:16 PM Luke Chen 
> > wrote:
> > >> >> >
> > >> >> > > Hi Colin,
> > >> >> > >
> > >> >> > > I'm not quite sure if I understand your thoughts correctly.
> > >> >> > > If I was wrong, please let me know.
> > >> >> > >
> > >> >> > > Also, I'm not quite sure how I could lock this feature to a new
> > IBP
> > >> >> > > version.
> > >> >> > > I saw "KIP-584: Versioning scheme for features" is still under
> > >> >> > development.
> > >> >> > > Not sure if I need to lock the IBP version, how should I do?
> > >> >> > >
> > >> >> > > Thank you.
> > >> >> > > Luke
> > >> >> > >
> > >> >> > > On Tue, Dec 7, 2021 at 9:41 PM Luke Chen 
> > >> wrote:
> > >> >> > >
> > >> >> > > > Hi Colin,
> > >> >> > > >
> > >> >> > > > Thanks for your comments. I've updated the KIP to mention about
> > >> the
> > >> >> KIP
> > >> >> > > > won't affect current broker side behavior.
> > >> >> > > >
> > >> >> > > > > One scenario that we need to consider is what happens during
> > a
> > >> >> > rolling
> > >> >> > > > upgrade. If the coordinator moves back and forth between
> > brokers
> > >> with
> > >> >> > > > different IBPs, it seems that the same epoch numbers could be
> > >> reused
> > >> >> > for
> > >> >> > > a
> > >> >> > > > group, if things are done in the obvious manner (old IBP =
> > don't
> > >> read
> > >> >> > or
> > >> >> > > > write epoch, new IBP = do)
> > >> >> > > >
> > >> >> > > > I think this KIP doesn't care about the group epoch number at
> > >> all.
> > >> >> The
> > >> >> > > > subscription metadata is passed from each member to group
> > >> >> coordinator,
> > >> >> > > and
> > >> >> > > > then the group coordinator pass all of them back to the
> > consumer
> > >> >> lead.
> > >> >> > So
> > >> >> > > > even if the epoch number is reused in a group, it should be
> > >> fine. On
> > >> >> > the
> > >> >> > > > other hand, the group coordinator will have no idea if the join
> > >> group
> > >> >> > > > request sent from consumer containing the new subscription
> > >> >> "generation"
> > >> >> > > > field or not, because group coordinator won't deserialize the
> > >> >> metadata.
> > >> >> > > >
> > >> >> > > > I've added also added them into the KIP.
> > >> >> > > >
> > >> >> > > > Thank you.
> > >> >> > > > Luke
> > >> >> > > >
> > >> >> > > > On Mon, Dec 6, 2021 at 10:39 AM Colin McCabe <
> > cmcc...@apache.org
> > >> >
> > >> >> > wrote:
> > >> >> > > >
> > >> >> > > >> Hi Luke,
> > >> >> > > >>
> > >> >> > > >> Thanks for the explanation.
> > >> >> > > >>
> > >> >> > > >> I don't see any description of how the broker decides to use
> > >> the new
> > >> >> > > >> version of ConsumerProtocolSubscription or not. This probably
> > >> needs
> > >> >> to
> > >>