Re: [DISCUSS] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Ryanne Dolan]

Gotcha, thanks.

On Fri, Jun 4, 2021, 8:20 PM Ismael Juma  wrote:

> Documentation, yes. Including the downloads page.
>
> Ismael
>
> On Fri, Jun 4, 2021, 4:44 PM Ryanne Dolan  wrote:
>
> > Thanks Ismael, this will be great for 4.0, but I don't understand what
> > exactly will change in 3.0? Just documentation?
> >
> > I guess I don't know what it means to deprecate a build target.
> >
> > Thanks!
> >
> > Ryanne
> >
> > On Fri, Jun 4, 2021, 9:06 AM Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > Please take a look at the KIP and provide your feedback:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> > >
> > > Ismael
> > >
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #198

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 472278 lines...]
[2021-06-04T22:08:23.513Z] [INFO] 

[2021-06-04T22:08:23.513Z] [INFO] Reactor Build Order:
[2021-06-04T22:08:23.513Z] [INFO] 
[2021-06-04T22:08:23.513Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2021-06-04T22:08:23.513Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2021-06-04T22:08:23.513Z] [INFO] 
[2021-06-04T22:08:23.513Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2021-06-04T22:08:23.513Z] [INFO] Building Kafka Streams :: Quickstart 
3.0.0-SNAPSHOT[1/2]
[2021-06-04T22:08:23.513Z] [INFO] [ pom 
]-
[2021-06-04T22:08:23.513Z] [INFO] 
[2021-06-04T22:08:23.513Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart ---
[2021-06-04T22:08:23.513Z] [INFO] 
[2021-06-04T22:08:23.513Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart ---
[2021-06-04T22:08:24.443Z] [INFO] 
[2021-06-04T22:08:24.443Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart ---
[2021-06-04T22:08:25.374Z] [INFO] 
[2021-06-04T22:08:25.374Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart ---
[2021-06-04T22:08:25.374Z] [INFO] 
[2021-06-04T22:08:25.374Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart ---
[2021-06-04T22:08:25.374Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.0.0-SNAPSHOT/streams-quickstart-3.0.0-SNAPSHOT.pom
[2021-06-04T22:08:25.374Z] [INFO] 
[2021-06-04T22:08:25.374Z] [INFO] --< 
org.apache.kafka:streams-quickstart-java >--
[2021-06-04T22:08:25.374Z] [INFO] Building streams-quickstart-java 
3.0.0-SNAPSHOT[2/2]
[2021-06-04T22:08:25.374Z] [INFO] --[ maven-archetype 
]---
[2021-06-04T22:08:25.374Z] [INFO] 
[2021-06-04T22:08:25.374Z] [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ streams-quickstart-java ---
[2021-06-04T22:08:25.374Z] [INFO] 
[2021-06-04T22:08:25.374Z] [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ streams-quickstart-java ---
[2021-06-04T22:08:25.374Z] [INFO] 
[2021-06-04T22:08:25.375Z] [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ streams-quickstart-java ---
[2021-06-04T22:08:26.305Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-06-04T22:08:26.305Z] [INFO] Copying 6 resources
[2021-06-04T22:08:26.305Z] [INFO] Copying 3 resources
[2021-06-04T22:08:26.305Z] [INFO] 
[2021-06-04T22:08:26.305Z] [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ streams-quickstart-java ---
[2021-06-04T22:08:26.305Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-06-04T22:08:26.305Z] [INFO] Copying 2 resources
[2021-06-04T22:08:26.305Z] [INFO] Copying 3 resources
[2021-06-04T22:08:26.305Z] [INFO] 
[2021-06-04T22:08:26.305Z] [INFO] --- maven-archetype-plugin:2.2:jar 
(default-jar) @ streams-quickstart-java ---
[2021-06-04T22:08:26.814Z] [INFO] Building archetype jar: 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.0.0-SNAPSHOT
[2021-06-04T22:08:26.814Z] [INFO] 
[2021-06-04T22:08:26.814Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart-java ---
[2021-06-04T22:08:26.814Z] [INFO] 
[2021-06-04T22:08:26.814Z] [INFO] --- 
maven-archetype-plugin:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[2021-06-04T22:08:26.814Z] [INFO] 
[2021-06-04T22:08:26.814Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart-java ---
[2021-06-04T22:08:26.814Z] [INFO] 
[2021-06-04T22:08:26.814Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart-java ---
[2021-06-04T22:08:26.814Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.0.0-SNAPSHOT.jar
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.0.0-SNAPSHOT/streams-quickstart-java-3.0.0-SNAPSHOT.jar
[2021-06-04T22:08:26.814Z] [INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.0.0-SNAPSHOT/streams-quickstart-java-3.0.0-SNAPSHOT.pom
[2021-06-04T22:08:26.814Z] [INFO] 
[2021-06-04T22:08:26.814Z] [INFO] --- 
maven-archetype-plugin:2.2:update-local-catalog 

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Konstantine Karantasis]

Hi Cyrus.

The proposal looks good and I like the API spec definition the way it's
presented.

Having said that, a few examples that would list the request type and body,
the returned status and the json response would be nice too, following the
tradition of other KIPs.

See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
for a recent example.

I also see there's no mention regarding the future of the current
`/connector-plugins` endpoint or any deprecation plans.

I think we should make our intentions clear in the KIP itself
which introduces the new endpoint as a superset of the old one, beyond the
discussion in this thread, for future readers.
Also, I think it's useful to keep in mind that deprecation doesn't
necessarily mean imminent removal. The next opportunity to remove this end
point would be the next major release at the earliest.

Regards,
Konstantine


On Mon, Apr 19, 2021 at 10:13 AM Cyrus Vafadari  wrote:

> Thanks! Anyone else from the community with final thoughts before going to
> vote?
>
> On Mon, Apr 19, 2021 at 4:16 AM Tom Bentley  wrote:
>
> > Hi Cyrus,
> >
> > That seems reasonable to me.
> >
> > On Thu, Apr 15, 2021 at 6:44 PM Cyrus Vafadari 
> > wrote:
> >
> > > What do you think of "type" field remaining in, and returning
> > > "transformation", "converter" or whatever type it is. This way the
> schema
> > > remains consistent, and you can programmatically understand what
> plugins
> > > are returned on the holistic "GET /plugins" endpoint? It will be
> slightly
> > > redundant in the case where you specify the plugin-type as a path
> > > parameter.
> > >
> > > On Thu, Apr 15, 2021 at 1:13 PM Tom Bentley 
> wrote:
> > >
> > > > Hi Cyrus,
> > > >
> > > > Re 2: A very minor thing but while type=source|sink for a connector,
> it
> > > > doesn't makes sense for the other plugin types, but so the json for
> > those
> > > > plugins should omit that property rather than have type=null.
> > > >
> > > > Apart from that it seems reasonable to me. Thanks again,
> > > >
> > > > Tom
> > > >
> > > > On Thu, Apr 15, 2021 at 3:49 PM Cyrus Vafadari 
> > > > wrote:
> > > >
> > > > > Hi Tom,
> > > > >
> > > > > Thanks for taking the time to respond with thoughtful questions:
> > > > >
> > > > > 1. I propose HTTP-400, will update the KIP to reflect this proposal
> > > > > 2. I think the schema should be of the same format. It is quite
> > > minimal,
> > > > so
> > > > > there is room to add fields in the future without breaking
> > > compatibility.
> > > > > I will update the KIP to specify.
> > > > > 3. Yes, that's right.
> > > > > 4. I personally don't see the value in deprecating since it's a
> > simple
> > > > > alias (interface and impl will both be simple changes). I would be
> > > > > comfortable kicking that can down the road if/when there is a need
> > for
> > > an
> > > > > actual breaking change. That way we can keep the scope and diff
> here
> > > > tight.
> > > > > But iff the community feels strongly that deprecating is the right
> > > thing
> > > > to
> > > > > do, I'm happy to update the KIP to propose deprecating.
> > > > >
> > > > > Thanks again!
> > > > >
> > > > > On Wed, Apr 14, 2021 at 9:38 AM Tom Bentley 
> > > wrote:
> > > > >
> > > > > > Hi Cyrus,
> > > > > >
> > > > > > Thanks for the KIP. A few questions:
> > > > > >
> > > > > > 1. What status code does the /plugins/{plugin-type} endpoint
> return
> > > > when
> > > > > an
> > > > > > unknown type is given?
> > > > > > 2. The result of /connector-plugins is a list of objects with
> > > 'class',
> > > > > > 'type' and 'version' properties. Presumably /plugins/connector is
> > the
> > > > > same,
> > > > > > but what is the schema for the other plugin types?
> > > > > > 3. You're not proposing an equivalent of the
> > > > > > /connector-plugins/{connectorType}/config/validate endpoint for
> > > > > > non-connector types, which I think makes sense, because you would
> > > > > validate
> > > > > > an SMT's config via its used by a connector, so the existing
> > endpoint
> > > > > > suffices, right?
> > > > > > 4. Would /connector-plugins eventually be deprecated in favour of
> > > > > > /plugins/connector, or do we expect it to remain in the API
> > > > indefinitely
> > > > > > and accept that /connector-plugins and /plugins/connector provide
> > > > > identical
> > > > > > responses? If we had the intention to deprecate in the future
> maybe
> > > we
> > > > > > should add a /plugins/connector/config/validate endpoint now?
> > > > > >
> > > > > > Many thanks,
> > > > > >
> > > > > > Tom
> > > > > >
> > > > > > On Tue, Apr 13, 2021 at 6:46 PM Cyrus Vafadari <
> > cvafad...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > As the number of connect plugins, SMT's, etc have grown, I
> wanted
> > > to
> > > > > bump
> > > > > > > this thread to see if there is more interest in adding a
> Connect
> > > REST
> > > > > > > endpoint to 

Re: [VOTE] KIP-738: Removal of Connect's internal converter properties

[Konstantine Karantasis]

+1 (binding)  but also with the note that we don't tend to require a KIP in
order to remove configurations that have been deprecated in previous KIPs.

A jira ticket with the right release as a target should suffice. In future
KIPs adding a note that says that a feature is deprecated and may be
removed in future major releases can help make this clear in the future.

Konstantine.


On Thu, Jun 3, 2021 at 10:47 AM Tom Bentley  wrote:

> +1 (binding).
>
> Thanks Chris!
>
> On Wed, Jun 2, 2021 at 11:00 PM Randall Hauch  wrote:
>
> > Thanks for putting this together, Chris.
> >
> > Technically, we don't need a new KIP to explicitly remove an API, config,
> > etc. that was previously deprecated under an earlier approved KIP. But
> > KIP-174 could have been a bit more explicit (note taken for future KIPs)
> > that deprecation means "deprecation and future removal", and KIP-738 does
> > add a nice migration path for anyone stuck using a non-default internal
> > converter.
> >
> > So I'm +1 (binding) to just wrap this up and remove these configs.
> >
> > I'd also point out there is a wiki page [1] and dev discussion thread [2]
> > highlighting all Connect-related KIPs, including all deprecations. I'll
> > update that page to include KIP-736, but it'd be great to have more
> > discussion there.
> >
> > Randall
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177047362
> > [2]
> >
> >
> https://lists.apache.org/thread.html/r01e9fa52998a337e435e5d0effca02e74b0552bdec271c1eeca39cd2%40%3Cdev.kafka.apache.org%3E
> >
> > On Tue, May 18, 2021 at 11:26 AM Ryanne Dolan 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > Thanks!
> > >
> > > Ryanne
> > >
> > > On Tue, May 18, 2021, 6:38 AM Chris Egerton
>  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to call for a vote on KIP-738:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-738%3A+Removal+of+Connect%27s+internal+converter+properties
> > > >
> > > > The discussion thread (which was originally titled with "KIP-736")
> can
> > be
> > > > found here:
> > > >
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202105.mbox/%3CCAMdOrUV0bqqs-ry7Q6FkNNn21ZhODTrg2d61zE5WZJw1MpQvSQ%40mail.gmail.com%3E
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Ismael Juma]

Documentation, yes. Including the downloads page.

Ismael

On Fri, Jun 4, 2021, 4:44 PM Ryanne Dolan  wrote:

> Thanks Ismael, this will be great for 4.0, but I don't understand what
> exactly will change in 3.0? Just documentation?
>
> I guess I don't know what it means to deprecate a build target.
>
> Thanks!
>
> Ryanne
>
> On Fri, Jun 4, 2021, 9:06 AM Ismael Juma  wrote:
>
> > Hi all,
> >
> > Please take a look at the KIP and provide your feedback:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> >
> > Ismael
> >
>

Re: [VOTE] KIP-722: Enable connector client overrides by default

[Konstantine Karantasis]

Good to flip the switch on this useful feature with the opportunity of the
major release here as well.
I don't see any disadvantages in doing so.

+1 (binding)

Konstantine

On Thu, May 6, 2021 at 6:07 AM Ryanne Dolan  wrote:

> +1 (non-binding) Thanks!
>
> Ryanne
>
> On Wed, May 5, 2021, 4:04 PM Randall Hauch  wrote:
>
> > I'd like to start a vote on KIP-722:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-722%3A+Enable+connector+client+overrides+by+default
> >
> > +1 (binding) from myself.
> >
> > Thanks, and best regards!
> >
> > Randall
> >
>

Re: [VOTE] KIP-721: Enable connector log contexts in Connect Log4j configuration

[Konstantine Karantasis]

I agree 3.0.0 is a good opportunity to enable this useful feature by
default.

+1 (binding)

Konstantine

On Fri, May 7, 2021 at 6:33 PM Dongjin Lee  wrote:

> +1 (Non-binding)
>
> Thanks,
> Dongjin
>
> On Thu, May 6, 2021 at 3:19 AM Randall Hauch  wrote:
>
> > I'd like to start a vote on KIP-721:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-721%3A+Enable+connector+log+contexts+in+Connect+Log4j+configuration
> >
> > +1 (binding) from myself.
> >
> > Thanks, and best regards!
> >
> > Randall
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [DISCUSS] KIP-745: Connect API to restart connector and tasks

[Konstantine Karantasis]

Thanks for the KIP Randall!
Really happy to see this feature coming along finally. Will indeed resolve
an unintuitive aspect of Connect's REST API.

I'm in favor of the current approach overall. Just a few comments below
that could use a brief discussion:

1) Is there a specific reason why you choose the config topic to persist
the restart requests?
I know it doesn't really matter which internal topic is used for backwards
compatibility, because the worker ignores records that does not understand.

But are there any implications upon worker restart? I read you specifically
mention that a worker will read restart requests upon its own restart. But
is this really desirable?
The config topic offers a way to persist configurations of the latest
running connectors and upon cluster restart to bring these connectors back
up.
However, I'm not sure we want to persist recent restart attempts, the same
way we persist configurations. Additionally configurations might get
additional backups in order to allow for the config topic to be recovered
for disaster recovery reasons. To that respect, I'm not sure that the
restart requests require the same guarantees.

Given the above, would the status topic offer a better alternative in order
to broadcast the restart requests across the workers? Haven't looked
closely what would be the implications of writing these new records to the
status topic, but this topic tends to be more transient.
Or is it that you've preferred the config topic for the workers to be aware
of the interleaving between restart requests and requests for
reconfigurations? If that's the case an explicit call out might be
worthwhile in the KIP itself.

2) You mention in the KIP the status STOPPED. But this is not currently
part of the task or connector states that get persisted to the status
topic, as these are defined in AbstractStatus#State enum. Should we remove
any reference to a STOPPED state to avoid confusion? The only new state
seems to be the RESTARTING state.

And a few minor comments.
a) The sentence "We need to keep the existing behavior of the method to
just restart the Connector object must be retained for backward
compatibility," doesn't read well for me.
b) Subsection title: "Use REST API for Worker-to-Worker Communication and"
is possibly an unfinished sentence.
c) In the paragraph right below, the last sentence "especially when the
number of workers " might also be missing something.

Best,
Konstantine

On Thu, Jun 3, 2021 at 8:42 AM Kalpesh Patel 
wrote:

> Agreed, this would be a great enhancement.
>
> On Thu, Jun 3, 2021 at 9:26 AM Nikita Kretov  wrote:
>
> > Hello! It's really not intuitive that you need to restart connect and
> > tasks separately! I'd like to see this KIP landed in 3.0.0 release!
> >
> > On 6/2/21 7:40 PM, Randall Hauch wrote:
> > > Hello all,
> > >
> > > Many users struggle with the connector restart REST API only restarting
> > the
> > > Connector instance rather than everything they associated with a
> "named"
> > > connector. I've created a KIP that attempts to solve this problem via
> new
> > > query parameters on an existing REST API method, though by default it
> > > remains backward compatible with older behavior:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-745%3A+Connect+API+to+restart+connector+and+tasks
> > >
> > > Please take a look and respond here with questions. I'd love to get
> this
> > > into AK 3.0.0, and the KIP freeze is coming up soon.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> >
>

Re: [DISCUSS] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Ryanne Dolan]

Thanks Ismael, this will be great for 4.0, but I don't understand what
exactly will change in 3.0? Just documentation?

I guess I don't know what it means to deprecate a build target.

Thanks!

Ryanne

On Fri, Jun 4, 2021, 9:06 AM Ismael Juma  wrote:

> Hi all,
>
> Please take a look at the KIP and provide your feedback:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
>
> Ismael
>

Re: [VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Josep Prat]

Hi Guozhang,
So if I understand correctly, it's only a couple of small changes that need
to be made to this KIP to be aligned with KAFKA-12370, right?

I'm guessing that StreamsMetadata would not only moved to o.a.k.streams but
also be split with Interface + internal implementation, am I right?


If that's the case, I could, most probably, update the KIP by Saturday
afternoon CEST.

Let me know if I understood you correctly.

Thanks for the comments!

———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Sat, Jun 5, 2021, 00:11 Guozhang Wang  wrote:

> Hello Josep,
>
> Thanks for the proposal! The write-up looks good to me in general. I'm just
> wondering if you feel comfortable to align this with another JIRA/KIP
> further down the road:
>
> https://issues.apache.org/jira/browse/KAFKA-12370
>
> Which tries to clean up the metadata hierarchy and the callers as
> StreamsMetadata -> ThreadMetadata -> TaskMetadata, and most Streams APIs
> return the top-level StreamsMetadata.
>
> It just have slight differences with the current proposal: 1) instead of
> returning a ThreadMetadata, "localThreadsMetadata" returns
> a StreamsMetadata, and 2) the `StreamsMetadata` would also be moved from
> o.a.k.streams.state to o.a.k.streams.
>
> What do you think about this? It's totally okay if you are not comfortable
> changing or expanding the scope of this KIP, that's totally fine with me as
> well, and we can just change again in the future if necessary --- just
> trying to see if we can align the direction on the first shot here :)
>
>
> Guozhang
>
> On Fri, Jun 4, 2021 at 1:51 AM Bruno Cadonna  wrote:
>
> > Thanks, Josep!
> >
> > +1 (binding)
> >
> > Bruno
> >
> > On 04.06.21 10:27, Josep Prat wrote:
> > > Hi all,
> > > I'd like to call for a vote on KIP-744: Migrate TaskMetadata and
> > > ThreadMetadata to an interface with internal implementation
> > > KIP page can be found here:
> https://cwiki.apache.org/confluence/x/XIrOCg
> > > Discussion thread can be found here:
> > >
> >
> https://lists.apache.org/x/thread.html/r1d20fb6dbd6b01bb84cbb17e992f4d08308980dfc5f2e0a68d674413@%3Cdev.kafka.apache.org%3E
> > >
> > > As it was pointed out, hopefully this KIP can be approved before the
> 3.0
> > > deadline, as we can clean up some non naming compliant methods recently
> > > introduced.
> > >
> > >
> > > Please note that the scope of the KIP increased during the discussion
> to
> > > also include ThreadMetadata.
> > >
> > > Thank you,
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Konstantine Karantasis]

Hi all,

Just a quick reminder that KIP Freeze is next Wednesday, June 9th.
A vote thread needs to be open for at least 72 hours, so to everyone that
is working hard on proposals targeting 3.0.0, please make sure that your
[VOTE] threads are started on time.

Best,
Konstantine


On Wed, May 26, 2021 at 8:10 PM Israel Ekpo  wrote:

> +1 on the new schedule.
>
> On Wed, May 26, 2021 at 8:14 PM Sophie Blee-Goldman
>  wrote:
>
> > Ah ok, thanks Konstantine. I won't bug you about every new KIP that comes
> > in between now and KIP Freeze :P
> >
> > +1 on the scheduling changes as well
> >
> > On Wed, May 26, 2021 at 4:00 PM David Arthur  wrote:
> >
> > > The new schedule looks good to me, +1
> > >
> > > On Wed, May 26, 2021 at 6:29 PM Ismael Juma  wrote:
> > >
> > > > Thanks Konstantine, +1 from me.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> > > >  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Please find below the updated release plan for the Apache Kafka
> 3.0.0
> > > > > release.
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > > >
> > > > > New suggested dates for the release are as follows:
> > > > >
> > > > > KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > > > > Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > > > > Code Freeze is 14 July 2021 (new date, extended by two weeks)
> > > > >
> > > > > At least two weeks of stabilization will follow Code Freeze.
> > > > >
> > > > > The release plan is up to date and currently includes all the
> > approved
> > > > KIPs
> > > > > that are targeting 3.0.0.
> > > > >
> > > > > Please let me know if you have any objections with the recent
> > extension
> > > > of
> > > > > Feature Freeze and Code Freeze or any other concerns.
> > > > >
> > > > > Regards,
> > > > > Konstantine
> > > > >
> > > >
> > > --
> > > David Arthur
> > >
> >
>

Re: [VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Guozhang Wang]

Hello Josep,

Thanks for the proposal! The write-up looks good to me in general. I'm just
wondering if you feel comfortable to align this with another JIRA/KIP
further down the road:

https://issues.apache.org/jira/browse/KAFKA-12370

Which tries to clean up the metadata hierarchy and the callers as
StreamsMetadata -> ThreadMetadata -> TaskMetadata, and most Streams APIs
return the top-level StreamsMetadata.

It just have slight differences with the current proposal: 1) instead of
returning a ThreadMetadata, "localThreadsMetadata" returns
a StreamsMetadata, and 2) the `StreamsMetadata` would also be moved from
o.a.k.streams.state to o.a.k.streams.

What do you think about this? It's totally okay if you are not comfortable
changing or expanding the scope of this KIP, that's totally fine with me as
well, and we can just change again in the future if necessary --- just
trying to see if we can align the direction on the first shot here :)


Guozhang

On Fri, Jun 4, 2021 at 1:51 AM Bruno Cadonna  wrote:

> Thanks, Josep!
>
> +1 (binding)
>
> Bruno
>
> On 04.06.21 10:27, Josep Prat wrote:
> > Hi all,
> > I'd like to call for a vote on KIP-744: Migrate TaskMetadata and
> > ThreadMetadata to an interface with internal implementation
> > KIP page can be found here: https://cwiki.apache.org/confluence/x/XIrOCg
> > Discussion thread can be found here:
> >
> https://lists.apache.org/x/thread.html/r1d20fb6dbd6b01bb84cbb17e992f4d08308980dfc5f2e0a68d674413@%3Cdev.kafka.apache.org%3E
> >
> > As it was pointed out, hopefully this KIP can be approved before the 3.0
> > deadline, as we can clean up some non naming compliant methods recently
> > introduced.
> >
> >
> > Please note that the scope of the KIP increased during the discussion to
> > also include ThreadMetadata.
> >
> > Thank you,
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-724: Drop support for message formats v0 and v1

[Jason Gustafson]

Hi Ismael,

I agree it would be awesome to drop support for the old formats! A few
comments below:

1. Suppose that a partition with 3 replicas begins at v0. One broker is
upgraded to v2 with the new "force upgrade" config as part of a rolling
restart, which leaves two replicas on v0 and one on v2. The KIP documents
that replicas will take the format from the leader without any conversion,
so I'm a little concerned that leader changes in this state could result in
the version flipping from v2 to v0. Could that happen? Say, for example, if
the third replica was catching up while the rolling restart was in
progress. Since it is probably an unlikely case in practice, would it make
sense to allow up-conversion during replication when "force upgrade" has
been enabled? Not sure I see any great alternatives to this.

2. A related question concerns batching when upgrading from the non-batched
formats (i.e. v0 and v1 when no compression is in use). I suspect the only
reasonable way to do this is to treat each individual record as a batch of
1 during the conversion process. Otherwise, I am not sure we can come up
with a deterministic approach to re-batching which can be applied
consistently on all replicas (especially taking into account compaction).
The reason this is important is that replication is aligned by batches, so
if the batches are not consistent, the replication won't work. The only
problem with using single-record batches is that the v2 message format has
some additional overhead for the batch itself when compared with the v0 and
v1 individual record format. So the up-conversion process would likely
increase storage. I don't think this is a dealbreaker, but probably worth
calling out somewhere.

3. In 4.0, we are dropping support for Fetch versions 0 through 3. I might
have missed it in the KIP, but it sounds like we need a similar restriction
on Produce versions since up-conversion will also not be supported? Would
it be reasonable to drop ListOffsets v0 at the same time? It is not related
to the message format, but the Fetch restriction will already effectively
kill any client that might still be using this wonky version. Not sure if
there are any similar opportunities, but might be worth checking.

Thanks,
Jason



On Fri, Jun 4, 2021 at 6:14 AM Ismael Juma  wrote:

> If there are no comments or objections, I will start a vote on this soon.
>
> On Tue, Jun 1, 2021 at 7:59 AM Ismael Juma  wrote:
>
> > Hi all,
> >
> > It's time to start the process of sunsetting message formats v0 and v1 in
> > order to establish a new baseline in terms of supported client/broker
> > behavior and to improve maintainability & supportability of Kafka. Please
> > take a look at the proposal:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
> >
> > Ismael
> >
>

Enforcing reviews on pull requests

[David Arthur]

Occasionally a Pull Request will get merged to trunk with an approval from
a community member, but not a committer. We should consider enabling the
GitHub branch protection feature that requires an approval from a committer.

We discussed this some time ago (December 2017)
https://lists.apache.org/thread.html/3ef19c439e978993d1e40f0f30cc0126e79761b7b8910400fb7f18bb%40%3Cdev.kafka.apache.org%3E

While trunk is a protected branch (as far as I know), we have not enabled
the additional review requirement.

Also, since that discussion, GitHub has added Code Owners which allows more
fine grained approval requirements for different areas of code. For a
project as large as Kafka, it might make sense to leverage this as well.

Thoughts?
-- 
David Arthur

Re: [DISCUSS] KIP-731: Record Rate Limiting for Kafka Connect

[Ryanne Dolan]

Hey Tom, thanks for taking a look.

> It's a bit weird that there's a separate start(Time) method

Good call, I think we can use a second constructor instead.

> No metrics for batch rates?

Good call. TBH I assumed there would already be put/poll rates, but looking
again I don't see them. Will add to the KIP.

> I think it might be nicer to have a consistent configuration mechanism

I had previously implemented this as you propose (same as SMTs), but found
it to be a little heavy for the common use-cases. I didn't like how users
needed to specify the classnames in order to use the built-in rate limiters.

But thinking again about this, if we include default values for
rate.limiters, rate.limiter.record.type, and rate.limiter.batch.type, we'd
get the same effect. Namely, most users would just need to
specify rate.limiter.record.limit or rate.limiter.batch.limit.

So I think you're right -- the common use-cases don't necessarily suffer,
and custom rate limiters would definitely benefit. I'll fix.

> hard.rate.limiters [..vs..] rate.limiters

I think the difference may be immaterial. As implemented currently,
RecordRateLimiter and RecordBatchRateLimiter are very "soft" in that they
don't define a window of time in which a max number of records or batches
can be processed. Instead, they just tap the breaks when the instantaneous
rate is observed to be too high. But a "hard" rate limiter could be
implemented with the same interface, e.g. by sleeping until the end of the
current window.

Ryanne

On Fri, May 21, 2021 at 7:10 AM Tom Bentley  wrote:

> Hi Ryanne,
>
> Thanks for the KIP. I can see this would be useful.
>
> 1. Can you elaborate on the life cycle of the RateLimiter interface (in the
> Javadoc)? In particular it's not clear to me how calls to accumulate() and
> throttleTime() can be interleaved (I assume arbitrarily).
>
> 2. It's a bit weird that there's a separate start(Time) method in addition
> to the configure() inherited from Configurable. Perhaps passing the Time to
> accumulate() would be simpler than needing a two stage configuration step,
> even if it would be the same instance on every call. If start() really is
> needed you should document that it's called after configure().
>
> 3. Maybe including the unit in the method name, i.e. throttleTimeMs(), to
> avoid any ambiguity about how the result is interpreted?
>
> 4. The metrics: Are they windowed over some time period, if so, what?
>
> 5. No metrics for batch rates?
>
> 6. It doesn't seem to be stated, but I assume the throttle time used is the
> maximum of the throttleTime() returned by all the limiters.
>
> 7. The configuration uses a different mechanism than for SMTs and also
> requires to add three common configs (with a risk of collision with any
> connector which already defines configs with these names). I think it might
> be nicer to have a consistent configuration mechanism, so for example
>   rate.limiters=record,batch
>   rate.limiter.record.type=RecordRateLimiter
>   rate.limiter.record.limit=123
>   rate.limiter.batch.type=RecordBatchRateLimiter
>   rate.limiter.batch.limit=456
> This means there's only a single new common config, as the others depend on
> the aliases used, so further collisions can be avoided.
>
> 8. A cluster where every connector has a quota could end up being
> underutilised, yet a subset of connectors could be running at their limit.
> While this makes sense for the firehose problem it seems to be problematic
> for the noisy neighbour case, where the spare capacity could be shared
> between all the throttled tasks on the worker. While I'm not suggesting you
> need to implement this as part of the KIP, maybe the API could accommodate
> it being added later. Perhaps this could be as simple as using
> hard.rate.limiters rather than just rate.limiters, so that
> soft.rate.limiters could be added later, though maybe there are use cases
> where a single limiter needs to supply both soft and hard limits.
>
> Thanks again,
>
> Tom
>
> On Fri, May 14, 2021 at 6:26 PM Ryanne Dolan 
> wrote:
>
> > Hey y'all, I've expanded the scope of this KIP slightly to include a
> > pluggable interface, RateLimiter.
> >
> > After implementing this a few different ways, it's clear that the
> > configuration story is actually simpler with a pluggable model.
> > Out-of-the-box, we have just two configuration properties to tweak:
> > record.rate.limit and record.batch.rate.limit (subj to change ofc). These
> > are provided by built-in RecordRateLimiter and RecordBatchRateLimiter
> > impls.
> >
> > From there, additional custom RateLimiters can be enabled with whatever
> > configuration they need. This is essentially the same pattern taken with
> > MetricsReporters and others.
> >
> > I had originally envisioned that the set of built-in limits would expand
> > over time, eg individual put/poll/commit/flush limits. However, these can
> > all be throttled adequately with the proposed API by limiting overall
> > record and batch thruput.

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #197

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 473512 lines...]
[2021-06-04T17:47:10.510Z] > Task :storage:api:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:api:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:api:classes UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :streams:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :streams:classes UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :storage:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:json:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:json:classes UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:json:javadoc SKIPPED
[2021-06-04T17:47:10.510Z] > Task :raft:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:json:javadocJar
[2021-06-04T17:47:10.510Z] > Task :clients:compileTestJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :clients:testClasses UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :metadata:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :core:compileJava NO-SOURCE
[2021-06-04T17:47:10.510Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:json:testClasses UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task :connect:json:testJar
[2021-06-04T17:47:10.510Z] > Task :connect:json:testSrcJar
[2021-06-04T17:47:10.510Z] > Task :streams:copyDependantLibs
[2021-06-04T17:47:10.510Z] > Task :streams:jar UP-TO-DATE
[2021-06-04T17:47:10.510Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-06-04T17:47:11.518Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-06-04T17:47:15.294Z] > Task :connect:api:javadoc
[2021-06-04T17:47:15.294Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-06-04T17:47:15.294Z] > Task :connect:api:jar UP-TO-DATE
[2021-06-04T17:47:15.294Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-06-04T17:47:15.294Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-06-04T17:47:15.294Z] > Task :connect:json:jar UP-TO-DATE
[2021-06-04T17:47:15.294Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-06-04T17:47:15.294Z] > Task :connect:api:javadocJar
[2021-06-04T17:47:15.294Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-06-04T17:47:15.294Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-06-04T17:47:15.294Z] > Task :connect:api:testJar
[2021-06-04T17:47:15.294Z] > Task :connect:api:testSrcJar
[2021-06-04T17:47:15.294Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-06-04T17:47:15.294Z] > Task :connect:json:publishToMavenLocal
[2021-06-04T17:47:15.294Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-06-04T17:47:15.294Z] > Task :connect:api:publishToMavenLocal
[2021-06-04T17:47:17.311Z] > Task :streams:javadoc
[2021-06-04T17:47:17.311Z] > Task :streams:javadocJar
[2021-06-04T17:47:20.395Z] > Task :clients:javadoc
[2021-06-04T17:47:20.395Z] > Task :clients:javadocJar
[2021-06-04T17:47:21.403Z] > Task :clients:testJar
[2021-06-04T17:47:21.403Z] > Task :clients:testSrcJar
[2021-06-04T17:47:21.403Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2021-06-04T17:47:21.403Z] > Task :clients:publishToMavenLocal
[2021-06-04T17:47:41.162Z] > Task :core:compileScala
[2021-06-04T17:48:53.168Z] > Task :core:classes
[2021-06-04T17:48:53.168Z] > Task :core:compileTestJava NO-SOURCE
[2021-06-04T17:49:15.192Z] > Task :core:compileTestScala
[2021-06-04T17:50:07.385Z] > Task :core:testClasses
[2021-06-04T17:50:32.522Z] > Task :streams:compileTestJava
[2021-06-04T17:51:35.193Z] > Task :streams:testClasses
[2021-06-04T17:51:35.194Z] > Task :streams:testJar
[2021-06-04T17:51:35.194Z] > Task :streams:testSrcJar
[2021-06-04T17:51:35.194Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-06-04T17:51:35.194Z] > Task :streams:publishToMavenLocal
[2021-06-04T17:51:35.194Z] 
[2021-06-04T17:51:35.194Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-06-04T17:51:35.194Z] Use '--warning-mode all' to show the individual 
deprecation warnings.
[2021-06-04T17:51:35.194Z] See 
https://docs.gradle.org/7.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2021-06-04T17:51:35.194Z] 
[2021-06-04T17:51:35.194Z] Execution optimizations have been disabled for 2 
invalid unit(s) of work during this build to ensure correctness.
[2021-06-04T17:51:35.194Z] Please consult deprecation warnings for more details.
[2021-06-04T17:51:35.194Z] 
[2021-06-04T17:51:35.194Z] BUILD SUCCESSFUL in 4m 40s
[2021-06-04T17:51:35.194Z] 71 actionable tasks: 37 executed, 34 up-to-date
[Pipeline] sh
[2021-06-04T17:51:38.467Z] + grep ^version= gradle.properties
[2021-06-04T17:51:38.467Z] + cut -d= -f 2
[Pipeline] dir
[2021-06-04T17:51:39.311Z] Running in 

Re: [DISCUSS] KIP-747 Add support for basic aggregation APIs

[Alexandre Brasil]

Hi Mohan,

I like the idea of adding those methods to the API, but I'd like to make a
suggestion:

Although the most used scenario for min() / max() might possibly be for
numeric values, I think they could also be
useful on other objects like Dates, LocalDates or Strings. Why limit the
API to Numbers only?

Extending on the above, couldn't we change the API to provide a
Comparator instead of the Function
for those methods, and make them return a KTable instead? Not only
would this approach not limit the
usage of those methods to Numbers, but they'd also preserve the origin from
the min/max value [1]. The extraction of
a single (numeric?) value could be achieved by a subsequent .mapValues()
operator, and this strategy would also
allow us to reuse the stream's current value serde on min / max, making the
Materialized an optional parameter.

One extra complication of this approach is that now we'd have to handle
repeated min/max values from different
origins (two semantically different objects for which the comparator
returns 0), but we could solve that by adding
a parameter to specify whether to use the older or newer value (or assuming
one of these options as default for a
simpler API?).


I know it's an implementation issue, but I'm curious on how you'd solve
handling the  on
the sum(). Since the multiple implementations of this interface don't have
a common constructor nor an interface
method to add two Numbers, would it be possible to implement sum() and
retain the original VR type on the
returned KTable?

[1]: An example scenario for this would be to find the min / max Bidding
for a product where, at the end of the
auction, I need not only the min / max value of said Bidding, but also the
bidder's contact information.

Best,
Alexandre

On Wed, Jun 2, 2021 at 8:54 PM Mohan Parthasarathy 
wrote:

> Hi,
>
> I have created a proposal for adding some additional aggregation APIs like
> count.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-747+Add+support+for+basic+aggregation+APIs
>
> I have noted down some of the issues that need discussion. Thanks to
> Matthias for helping me with the scope of the proposal.
>
> Thanks
> Mohan
>

[jira] [Created] (KAFKA-12898) Owned partitions in the subscription must be sorted

[David Jacot (Jira)]

David Jacot created KAFKA-12898:
---

 Summary: Owned partitions in the subscription must be sorted
 Key: KAFKA-12898
 URL: https://issues.apache.org/jira/browse/KAFKA-12898
 Project: Kafka
  Issue Type: Bug
Reporter: David Jacot
Assignee: David Jacot


While investigating https://issues.apache.org/jira/browse/KAFKA-12896, I have 
noticed that the leader was always sending the same subscribed partitions but 
not always in the order. The group coordinator compares the provided 
subscription with the store subscription based on their bytes representation. 
So if the subscribed partitions are not in the same order, the group 
coordinator would consider that they are different and rebalance the group.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12897) KRaft Controller cannot create topic with more partitions than racks

[Ron Dagostino (Jira)]

Ron Dagostino created KAFKA-12897:
-

 Summary: KRaft Controller cannot create topic with more partitions 
than racks
 Key: KAFKA-12897
 URL: https://issues.apache.org/jira/browse/KAFKA-12897
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 3.0.0
Reporter: Ron Dagostino
Assignee: Ron Dagostino
 Fix For: 3.0.0


https://github.com/apache/kafka/pull/10494 introduced a bug in the KRaft 
controller where the controller will loop forever in `StripedReplicaPlacer` 
trying to identify the racks on which to place partitions if the number of 
requested partitions in a CREATE_TOPICS request exceeds the number of effective 
racks ("effective" meaning a single rack if none are specified).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-746: Revise KRaft Metadata Records

[Jun Rao]

Hi, Colin,

1. Sounds good.

2. Yes, adding the listener fields will make it clear how
BrokerRegistrationChangeRecord
will be used.

Thanks,

Jun

On Thu, Jun 3, 2021 at 4:34 PM Colin McCabe  wrote:

> On Thu, Jun 3, 2021, at 16:29, Jun Rao wrote:
> > Hi, Colin,
> >
> > Thanks for the KIP. Just a couple of minor comments.
> >
>
> Hi Jun,
>
> Thanks for taking a look. Sorry I just started the vote thread before I
> saw this! :)
>
> > 1. Fields like RemovingReplicas are added as tagged fields in
> > PartitionChangeRecord, but as non-tagged fields in PartitionRecord.
> Should
> > we make them consistent?
> >
>
> I think it makes sense to make them normal fields in PartitionRecord,
> since they will always be present there. In PartitionChangeRecord, these
> fields will only be present if they are changing, so it makes sense to make
> them tagged fields.
>
> > 2. Should we add BrokerRegistrationChangeRecord later when it has more
> > fields than what's already covered in FenceBrokerRecord and
> > UnfenceBrokerRecord?
> >
>
> Hmm... eventually we want to have the ability to change the broker
> endpoints dynamically (just like we can do in the ZK-enabled broker). That
> will certainly belong in BrokerRegistrationChangeRecord. If I add this to
> the record, does that give enough motivation to add it now rather than
> later? I like the consistency of having a single change record.
>
> best,
> Colin
>
>
> > Jun
> >
> >
> > On Wed, Jun 2, 2021 at 11:02 AM Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > I have posted a KIP about updating the KRaft metadata records for 3.0.
> > >
> > > Check it out at : https://cwiki.apache.org/confluence/x/34zOCg
> > >
> > > best,
> > > Colin
> > >
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #196

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 476058 lines...]
[2021-06-04T10:01:01.057Z] SaslSslAdminIntegrationTest > 
testCreateTopicsResponseMetadataAndConfig() STARTED
[2021-06-04T10:02:12.258Z] 
[2021-06-04T10:02:12.258Z] SaslSslAdminIntegrationTest > 
testCreateTopicsResponseMetadataAndConfig() PASSED
[2021-06-04T10:02:12.258Z] 
[2021-06-04T10:02:12.258Z] SaslSslAdminIntegrationTest > 
testAttemptToCreateInvalidAcls() STARTED
[2021-06-04T10:03:32.970Z] 
[2021-06-04T10:03:32.970Z] SaslSslAdminIntegrationTest > 
testAttemptToCreateInvalidAcls() PASSED
[2021-06-04T10:03:32.970Z] 
[2021-06-04T10:03:32.970Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() STARTED
[2021-06-04T10:04:53.913Z] 
[2021-06-04T10:04:53.913Z] SaslSslAdminIntegrationTest > 
testAclAuthorizationDenied() PASSED
[2021-06-04T10:04:53.913Z] 
[2021-06-04T10:04:53.913Z] SaslSslAdminIntegrationTest > testAclOperations() 
STARTED
[2021-06-04T10:06:16.531Z] 
[2021-06-04T10:06:16.531Z] SaslSslAdminIntegrationTest > testAclOperations() 
PASSED
[2021-06-04T10:06:16.531Z] 
[2021-06-04T10:06:16.531Z] SaslSslAdminIntegrationTest > testAclOperations2() 
STARTED
[2021-06-04T10:07:38.086Z] 
[2021-06-04T10:07:38.086Z] SaslSslAdminIntegrationTest > testAclOperations2() 
PASSED
[2021-06-04T10:07:38.086Z] 
[2021-06-04T10:07:38.086Z] SaslSslAdminIntegrationTest > testAclDelete() STARTED
[2021-06-04T10:09:14.130Z] 
[2021-06-04T10:09:14.130Z] SaslSslAdminIntegrationTest > testAclDelete() PASSED
[2021-06-04T10:09:14.130Z] 
[2021-06-04T10:09:14.130Z] TransactionsTest > testBumpTransactionalEpoch() 
STARTED
[2021-06-04T10:09:35.227Z] 
[2021-06-04T10:09:35.227Z] TransactionsTest > testBumpTransactionalEpoch() 
PASSED
[2021-06-04T10:09:35.227Z] 
[2021-06-04T10:09:35.227Z] TransactionsTest > 
testSendOffsetsWithGroupMetadata() STARTED
[2021-06-04T10:10:01.423Z] 
[2021-06-04T10:10:01.423Z] TransactionsTest > 
testSendOffsetsWithGroupMetadata() PASSED
[2021-06-04T10:10:01.423Z] 
[2021-06-04T10:10:01.423Z] TransactionsTest > testBasicTransactions() STARTED
[2021-06-04T10:10:17.121Z] 
[2021-06-04T10:10:17.121Z] TransactionsTest > testBasicTransactions() PASSED
[2021-06-04T10:10:17.121Z] 
[2021-06-04T10:10:17.121Z] TransactionsTest > testSendOffsetsWithGroupId() 
STARTED
[2021-06-04T10:10:41.822Z] 
[2021-06-04T10:10:41.822Z] TransactionsTest > testSendOffsetsWithGroupId() 
PASSED
[2021-06-04T10:10:41.822Z] 
[2021-06-04T10:10:41.822Z] TransactionsTest > testFencingOnSendOffsets() STARTED
[2021-06-04T10:10:59.592Z] 
[2021-06-04T10:10:59.592Z] TransactionsTest > testFencingOnSendOffsets() PASSED
[2021-06-04T10:10:59.592Z] 
[2021-06-04T10:10:59.592Z] TransactionsTest > testFencingOnAddPartitions() 
STARTED
[2021-06-04T10:11:20.560Z] 
[2021-06-04T10:11:20.560Z] TransactionsTest > testFencingOnAddPartitions() 
PASSED
[2021-06-04T10:11:20.560Z] 
[2021-06-04T10:11:20.560Z] TransactionsTest > 
testFencingOnTransactionExpiration() STARTED
[2021-06-04T10:11:32.863Z] 
[2021-06-04T10:11:32.863Z] TransactionsTest > 
testFencingOnTransactionExpiration() PASSED
[2021-06-04T10:11:32.863Z] 
[2021-06-04T10:11:32.863Z] TransactionsTest > 
testDelayedFetchIncludesAbortedTransaction() STARTED
[2021-06-04T10:11:52.995Z] 
[2021-06-04T10:11:52.995Z] TransactionsTest > 
testDelayedFetchIncludesAbortedTransaction() PASSED
[2021-06-04T10:11:52.995Z] 
[2021-06-04T10:11:52.995Z] TransactionsTest > 
testOffsetMetadataInSendOffsetsToTransaction() STARTED
[2021-06-04T10:12:04.182Z] 
[2021-06-04T10:12:04.182Z] TransactionsTest > 
testOffsetMetadataInSendOffsetsToTransaction() PASSED
[2021-06-04T10:12:04.182Z] 
[2021-06-04T10:12:04.182Z] TransactionsTest > testInitTransactionsTimeout() 
STARTED
[2021-06-04T10:12:22.918Z] 
[2021-06-04T10:12:22.918Z] TransactionsTest > testInitTransactionsTimeout() 
PASSED
[2021-06-04T10:12:22.918Z] 
[2021-06-04T10:12:22.918Z] TransactionsTest > 
testConsecutivelyRunInitTransactions() STARTED
[2021-06-04T10:12:32.247Z] 
[2021-06-04T10:12:32.247Z] TransactionsTest > 
testConsecutivelyRunInitTransactions() PASSED
[2021-06-04T10:12:32.247Z] 
[2021-06-04T10:12:32.247Z] TransactionsTest > 
testReadCommittedConsumerShouldNotSeeUndecidedData() STARTED
[2021-06-04T10:12:47.742Z] 
[2021-06-04T10:12:47.742Z] TransactionsTest > 
testReadCommittedConsumerShouldNotSeeUndecidedData() PASSED
[2021-06-04T10:12:47.742Z] 
[2021-06-04T10:12:47.742Z] TransactionsTest > 
testSendOffsetsToTransactionTimeout() STARTED
[2021-06-04T10:13:12.945Z] 
[2021-06-04T10:13:12.945Z] TransactionsTest > 
testSendOffsetsToTransactionTimeout() PASSED
[2021-06-04T10:13:12.945Z] 
[2021-06-04T10:13:12.945Z] TransactionsTest > testFailureToFenceEpoch() STARTED
[2021-06-04T10:14:23.142Z] 
[2021-06-04T10:14:23.142Z] TransactionsTest > testFailureToFenceEpoch() PASSED
[2021-06-04T10:14:23.142Z] 
[2021-06-04T10:14:23.142Z] TransactionsTest > testFencingOnSend() STARTED

Re: [DISCUSS] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Josep Prat]

Hi Ismael,

I see no major problem with the KIP.
The vast majority of Scala projects offer support for 2 major versions of
Scala as well (note Scala uses epoch.major.minor scheme), so the community
is more or less used to it. Also migrating from major versions has
significantly improved since older times.

Best,

———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Fri, Jun 4, 2021, 16:07 Ismael Juma  wrote:

> Hi all,
>
> Please take a look at the KIP and provide your feedback:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
>
> Ismael
>

[jira] [Created] (KAFKA-12896) Group rebalance loop caused by repeated group leader JoinGroups

[Lucas Bradstreet (Jira)]

Lucas Bradstreet created KAFKA-12896:


 Summary: Group rebalance loop caused by repeated group leader 
JoinGroups
 Key: KAFKA-12896
 URL: https://issues.apache.org/jira/browse/KAFKA-12896
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.6.0
Reporter: Lucas Bradstreet


We encountered a strange case of a rebalance loop with the "cooperative-sticky" 
assignor. The logs show the following for several hours:

 

{{Apr 7, 2021 @ 03:58:36.040[GroupCoordinator 7]: Stabilized group mygroup 
generation 19830137 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.992[GroupCoordinator 7]: Preparing to rebalance 
group mygroup in state PreparingRebalance with old generation 19830136 
(__consumer_offsets-7) (reason: Updating metadata for member 
mygroup-1-7ad27e07-3784-4588-97e1-d796a74d4ecc during CompletingRebalance)}}

{{Apr 7, 2021 @ 03:58:35.988[GroupCoordinator 7]: Stabilized group mygroup 
generation 19830136 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.972[GroupCoordinator 7]: Preparing to rebalance 
group mygroup in state PreparingRebalance with old generation 19830135 
(__consumer_offsets-7) (reason: Updating metadata for member mygroup during 
CompletingRebalance)}}

{{Apr 7, 2021 @ 03:58:35.965[GroupCoordinator 7]: Stabilized group mygroup 
generation 19830135 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.953[GroupCoordinator 7]: Preparing to rebalance 
group mygroup in state PreparingRebalance with old generation 19830134 
(__consumer_offsets-7) (reason: Updating metadata for member 
mygroup-7ad27e07-3784-4588-97e1-d796a74d4ecc during CompletingRebalance)}}

{{Apr 7, 2021 @ 03:58:35.941[GroupCoordinator 7]: Stabilized group mygroup 
generation 19830134 (__consumer_offsets-7)}}

{{Apr 7, 2021 @ 03:58:35.926[GroupCoordinator 7]: Preparing to rebalance 
group mygroup in state PreparingRebalance with old generation 19830133 
(__consumer_offsets-7) (reason: Updating metadata for member mygroup during 
CompletingRebalance)}}

Every single time, it was the same member that triggered the JoinGroup and it 
was always the leader of the group.{{}}

The leader has the privilege of being able to trigger a rebalance by sending 
`JoinGroup` even if its subscription metadata has not changed. But why would it 
do so?

It is possible that this is due to the same issue or a similar bug to 
https://issues.apache.org/jira/browse/KAFKA-12890.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Ismael Juma]

Hi all,

Please take a look at the KIP and provide your feedback:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218

Ismael

[DISCUSS] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Ismael Juma]

Hi all,

Please take a look at the KIP and provide your feedback:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223

Ismael

[jira] [Created] (KAFKA-12895) KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-12895:
---

 Summary: KIP-751: Drop support for Scala 2.12 in Kafka 4.0 
(deprecate in 3.0)
 Key: KAFKA-12895
 URL: https://issues.apache.org/jira/browse/KAFKA-12895
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
Assignee: Ismael Juma


We propose to deprecate Scala 2.12 support n Apache Kafka 3.0 and to drop it in 
Apache Kafka 4.0.

 

KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12894) KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-12894:
---

 Summary: KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate 
in 3.0)
 Key: KAFKA-12894
 URL: https://issues.apache.org/jira/browse/KAFKA-12894
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
Assignee: Ismael Juma


We propose deprecating Java 8 support in Apache Kafka 3.0 and dropping support 
in Apache Kafka 4.0.

KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12893) MM2 fails to replicate if starting two+ nodes same time

[Tommi Vainikainen (Jira)]

Tommi Vainikainen created KAFKA-12893:
-

 Summary: MM2 fails to replicate if starting two+ nodes same time
 Key: KAFKA-12893
 URL: https://issues.apache.org/jira/browse/KAFKA-12893
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.8.0
Reporter: Tommi Vainikainen


I've observed a situation where starting more than one MM2 node in parallel, 
MM2 fails to start replication ie. replication flow seems to be stuck without 
action. I used exactly same mm2.properties file to start only one at a time, 
and the replication flow was proceeding smoothly.

In my setup dc1 has topic "mytopic1" and there is a producer with approx 1 
msg/sec, and I'm trying to repilcate this to dc2. What I observed is that 
dc1.mytopic1 is created when initially launching two paraller MM2 instances, 
but no messages gets written into the topic as I would expect. If I kill MM2 
instances, and only start one MM2 node, then MM2 starts replicating the 
messages in mytopic1.

My mm2.properties:

clusters=dc2, dc1
dc1->dc2.emit.heartbeats.enabled=true
dc1->dc2.enabled=true
dc1->dc2.sync.group.offsets.enabled=false
dc1->dc2.sync.group.offsets.interval.seconds=45
dc1->dc2.topics=mytopic1
dc1->dc2.topics.exclude=
dc1.bootstrap.servers=tvainika-dc1-dev-sandbox.aivencloud.com:12693
dc1.security.protocol=SSL
dc1.ssl.keystore.type=PKCS12
dc1.ssl.keystore.location=dc1/client.keystore.p12
dc1.ssl.keystore.password=secret
dc1.ssl.key.password=secret
dc1.ssl.truststore.location=dc1/client.truststore.jks
dc1.ssl.truststore.password=secret
dc2.bootstrap.servers=tvainika-dc2-dev-sandbox.aivencloud.com:12693
dc2.security.protocol=SSL
dc2.ssl.keystore.type=PKCS12
dc2.ssl.keystore.location=dc2/client.keystore.p12
dc2.ssl.keystore.password=secret
dc2.ssl.key.password=secret
dc2.ssl.truststore.location=dc2/client.truststore.jks
dc2.ssl.truststore.password=secret
tasks.max=3






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12892) InvalidACLException thrown in tests caused jenkins build unstable

[Luke Chen (Jira)]

Luke Chen created KAFKA-12892:
-

 Summary: InvalidACLException thrown in tests caused jenkins build 
unstable
 Key: KAFKA-12892
 URL: https://issues.apache.org/jira/browse/KAFKA-12892
 Project: Kafka
  Issue Type: Bug
Reporter: Luke Chen
 Attachments: image-2021-06-04-21-05-57-222.png

In KAFKA-12866, we fixed the issue that Kafka required ZK root access even when 
using a chroot. But after the PR merged (build #183), trunk build keeps failing 
at least one test group (mostly, JDK 15 and Scala 2.13). The build result will 
said nothing useful:
{code:java}
> Task :core:integrationTest FAILED
[2021-06-04T03:19:18.974Z] 
[2021-06-04T03:19:18.974Z] FAILURE: Build failed with an exception.
[2021-06-04T03:19:18.974Z] 
[2021-06-04T03:19:18.974Z] * What went wrong:
[2021-06-04T03:19:18.974Z] Execution failed for task ':core:integrationTest'.
[2021-06-04T03:19:18.974Z] > Process 'Gradle Test Executor 128' finished with 
non-zero exit value 1
[2021-06-04T03:19:18.974Z]   This problem might be caused by incorrect test 
process configuration.
[2021-06-04T03:19:18.974Z]   Please refer to the test execution section in the 
User Manual at 
https://docs.gradle.org/7.0.2/userguide/java_testing.html#sec:test_execution
{code}
 

After investigation, I found the failed tests is because there are many 
`InvalidACLException` thrown during the tests, ex:

 
{code:java}
GssapiAuthenticationTest > testServerNotFoundInKerberosDatabase() FAILED
[2021-06-04T02:25:45.419Z] 
org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = 
InvalidACL for /config/topics/__consumer_offsets
[2021-06-04T02:25:45.419Z] at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
[2021-06-04T02:25:45.419Z] at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
[2021-06-04T02:25:45.419Z] at 
kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:583)
[2021-06-04T02:25:45.419Z] at 
kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
[2021-06-04T02:25:45.419Z] at 
kafka.zk.KafkaZkClient.createOrSet$1(KafkaZkClient.scala:366)
[2021-06-04T02:25:45.419Z] at 
kafka.zk.KafkaZkClient.setOrCreateEntityConfigs(KafkaZkClient.scala:376)
[2021-06-04T02:25:45.419Z] at 
kafka.zk.AdminZkClient.createTopicWithAssignment(AdminZkClient.scala:109)
[2021-06-04T02:25:45.419Z] at 
kafka.zk.AdminZkClient.createTopic(AdminZkClient.scala:60)
[2021-06-04T02:25:45.419Z] at 
kafka.utils.TestUtils$.$anonfun$createTopic$1(TestUtils.scala:357)
[2021-06-04T02:25:45.419Z] at 
kafka.utils.TestUtils$.createTopic(TestUtils.scala:848)
[2021-06-04T02:25:45.419Z] at 
kafka.utils.TestUtils$.createOffsetsTopic(TestUtils.scala:428)
[2021-06-04T02:25:45.419Z] at 
kafka.api.IntegrationTestHarness.doSetup(IntegrationTestHarness.scala:109)
[2021-06-04T02:25:45.419Z] at 
kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:84)
[2021-06-04T02:25:45.419Z] at 
kafka.server.GssapiAuthenticationTest.setUp(GssapiAuthenticationTest.scala:68)
{code}
 

Log can be found 
[here|[https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka/branches/trunk/runs/195/nodes/14/steps/145/log/?start=0]|https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka/branches/trunk/runs/195/nodes/14/steps/145/log/?start=0].]

After tracing back, I found it could because we add a test in the KAFKA-12866 
to lock root access in zookeeper, but somehow it didn't unlock after the test 
in testChrootExistsAndRootIsLocked. Also, while all the InvalidACLException 
failed tests happened right after testChrootExistsAndRootIsLocked not long. Ex: 
below testChrootExistsAndRootIsLocked completed at 02:24:30, and the above 
failed test is at 02:25:45 (and following more than 10 tests with the same 
InvalidACLException. 
{code:java}
[2021-06-04T02:24:29.370Z] ZkClientAclTest > testChrootExistsAndRootIsLocked() 
STARTED
[2021-06-04T02:24:30.321Z] 
[2021-06-04T02:24:30.321Z] ZkClientAclTest > testChrootExistsAndRootIsLocked() 
PASSED{code}
 

!image-2021-06-04-21-05-57-222.png|width=489,height=!

We should have further investigation to see how to improve the test to avoid 
breaking the build. Before that, we can disable the test first. Thanks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-724: Drop support for message formats v0 and v1

[Ismael Juma]

If there are no comments or objections, I will start a vote on this soon.

On Tue, Jun 1, 2021 at 7:59 AM Ismael Juma  wrote:

> Hi all,
>
> It's time to start the process of sunsetting message formats v0 and v1 in
> order to establish a new baseline in terms of supported client/broker
> behavior and to improve maintainability & supportability of Kafka. Please
> take a look at the proposal:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1
>
> Ismael
>

Adding @NotNull annotation to public APIs, KIP needed or not?

[Matthew de Detrich]

Hello everyone,

I was thinking of doing a PR which involved adding @NotNull annotations to
various Kafka API's. Afaik the @NotNull annotation doesn't break binary
compatibility however it can break source compatibility.

The point is that even though using @NotNull can break source
compatibility, if it does (assuming that the @NotNull is added
correctly) then you would have gotten a runtime NotNullException or null
related error and hence your code would have been broken anyways.

So ultimately I guess the question is do I need to create a KIP to work on
such a task? I think technically speaking you do need to create one however
because of what I just said it may not be needed?

Regards
-- 

Matthew de Detrich

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491603708037

*w:* aiven.io *e:* matthew.dedetr...@aiven.io

[DISCUSS] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[wenbing shen]

Hi all

I'd like to discuss the following kip, any suggestions are welcome.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-749%3A+Add+--files+and+--file-separator+options+to+the+ConsoleProducer

Many thanks,

Wenbing

[jira] [Created] (KAFKA-12891) Add --files and --file-separator options to the ConsoleProducer

[Wenbing Shen (Jira)]

Wenbing Shen created KAFKA-12891:


 Summary: Add --files and --file-separator options to the 
ConsoleProducer
 Key: KAFKA-12891
 URL: https://issues.apache.org/jira/browse/KAFKA-12891
 Project: Kafka
  Issue Type: New Feature
  Components: tools
Reporter: Wenbing Shen
Assignee: Wenbing Shen


Introduce *--files* to the producer command line tool to support reading data 
from a given *multi-file*,
Multiple files are separated by *--files-separator*, the default *comma* is the 
separator.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Bruno Cadonna]

Thanks, Josep!

+1 (binding)

Bruno

On 04.06.21 10:27, Josep Prat wrote:

Hi all,
I'd like to call for a vote on KIP-744: Migrate TaskMetadata and
ThreadMetadata to an interface with internal implementation
KIP page can be found here: https://cwiki.apache.org/confluence/x/XIrOCg
Discussion thread can be found here:
https://lists.apache.org/x/thread.html/r1d20fb6dbd6b01bb84cbb17e992f4d08308980dfc5f2e0a68d674413@%3Cdev.kafka.apache.org%3E

As it was pointed out, hopefully this KIP can be approved before the 3.0
deadline, as we can clean up some non naming compliant methods recently
introduced.


Please note that the scope of the KIP increased during the discussion to
also include ThreadMetadata.

Thank you,

[VOTE] KIP-744: Migrate TaskMetadata and ThreadMetadata to an interface with internal implementation

[Josep Prat]

Hi all,
I'd like to call for a vote on KIP-744: Migrate TaskMetadata and
ThreadMetadata to an interface with internal implementation
KIP page can be found here: https://cwiki.apache.org/confluence/x/XIrOCg
Discussion thread can be found here:
https://lists.apache.org/x/thread.html/r1d20fb6dbd6b01bb84cbb17e992f4d08308980dfc5f2e0a68d674413@%3Cdev.kafka.apache.org%3E

As it was pointed out, hopefully this KIP can be approved before the 3.0
deadline, as we can clean up some non naming compliant methods recently
introduced.


Please note that the scope of the KIP increased during the discussion to
also include ThreadMetadata.

Thank you,

-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491715557497

*w:* aiven.io

*e:* josep.p...@aiven.io

Re: [DISCUSS] KIP-748: Add Broker Count Metrics

[David Jacot]

Hi Ryan,

Thanks for the KIP.

+1 for adding RegisteredBrokerCount to the ZK controller as well. This
would be really helpful.

Best,
David

On Fri, Jun 4, 2021 at 12:44 AM Colin McCabe  wrote:

> Hi Ryan,
>
> Thanks for the KIP. I think it would be good to provide the
> RegisteredBrokerCount metric for the ZK controller as well as for the
> Quorum controller. Looks good aside from that!
>
> best,
> Colin
>
> On Thu, Jun 3, 2021, at 14:09, Ryan Dielhenn wrote:
> > Hey kafka-dev,
> >
> > I created KIP-748 as a proposal to add broker count metrics to the
> > Quorum Controller.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-748%3A+Add+Broker+Count+Metrics#KIP748:AddBrokerCountMetrics
> >
> > Best,
> > Ryan Dielhenn
> >
>

[jira] [Created] (KAFKA-12890) Consumer group stuck in `CompletingRebalance`

[David Jacot (Jira)]

David Jacot created KAFKA-12890:
---

 Summary: Consumer group stuck in `CompletingRebalance`
 Key: KAFKA-12890
 URL: https://issues.apache.org/jira/browse/KAFKA-12890
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.6.2, 2.7.1, 2.8.0, 2.6.1, 2.7.0
Reporter: David Jacot
Assignee: David Jacot


We have seen recently multiple consumer groups stuck in `CompletingRebalance`. 
It appears that those group never receives the assignment from the leader of 
the group and remains stuck in this state forever.

When a group transitions to the `CompletingRebalance` state, the group 
coordinator sets up `DelayedHeartbeat` for each member of the group. It does so 
to ensure that the member sends a sync request within the session timeout. If 
it does not, the group coordinator rebalances the group. Note that here, 
`DelayedHeartbeat` is used here for this purpose. `DelayedHeartbeat` are also 
completed when member heartbeats.

The issue is that https://github.com/apache/kafka/pull/8834 has changed the 
heartbeat logic to allow members to heartbeat while the group is in the 
`CompletingRebalance` state. This was not allowed before. Now, if a member 
starts to heartbeat while the group is in the `CompletingRebalance`, the 
heartbeat request will basically complete the pending `DelayedHeartbeat` that 
was setup previously for catching not receiving the sync request. Therefore, if 
the sync request never comes, the group coordinator does not notice anymore.

We need to bring that behavior back somehow.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12889) log clean group consider empty log segment to avoid empty log left

[qiang Liu (Jira)]

qiang Liu created KAFKA-12889:
-

 Summary: log clean group consider empty log segment to avoid empty 
log left
 Key: KAFKA-12889
 URL: https://issues.apache.org/jira/browse/KAFKA-12889
 Project: Kafka
  Issue Type: Improvement
  Components: log cleaner
Affects Versions: 0.10.1.1
Reporter: qiang Liu


to avoid log index 4 byte relative offset overflow, log cleaner group check log 
segments offset to make sure group offset range not exceed Int.MaxValue.

this offset check currentlly not cosider next is next log segment is empty, so 
there will left empty log files every about 2^31 messages.

the left empty logs will be reprocessed every clean cycle, which will rewrite 
it with same empty content, witch cause little no need io.

for __consumer_offsets topic, normally we can set cleanup.policy to 
compact,delete to get rid of this.

my cluster is 0.10.1.1, but after aylize trunk code, it should has same problem 
too.

 

some of my left empty logs,(run ls -l)

-rw-r- 1 u g 0 Dec 16 2017 .index
-rw-r- 1 u g 0 Dec 16 2017 .log
-rw-r- 1 u g 0 Dec 16 2017 .timeindex
-rw-r- 1 u g 0 Jan  15 2018 002148249632.index
-rw-r- 1 u g 0 Jan  15 2018 002148249632.log
-rw-r- 1 u g 0 Jan  15 2018 002148249632.timeindex
-rw-r- 1 u g 0 Jan  27 2018 004295766494.index
-rw-r- 1 u g 0 Jan  27 2018 004295766494.log
-rw-r- 1 u g 0 Jan  27 2018 004295766494.timeindex

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)