[RESULTS] [VOTE] Release Kafka version 2.8.0

[John Roesler]

This vote passes with 6 +1 votes (4 binding) and no 0 or -1
votes.

+1 votes
PMC Members:
* Bill Bejeck
* Randall Hauch
* Colin McCabe
* John Roesler

Community:
* Israel Ekpo
* Satish Duggana

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://lists.apache.org/thread.html/rd7888d15a6db647b5c6a96ec1afd6cd47f11237a8f41340ae8af6b15%40%3Cdev.kafka.apache.org%3E

I'll continue with the release process and the release
announcement will follow in the next few days.

Thanks,
John Roesler

On Wed, 2021-04-14 at 15:04 -0500, John Roesler wrote:
> Hello Kafka users, developers and client-developers,
> 
> This is the third candidate for release of Apache Kafka
> 2.8.0.This is a major release that includes many new
> features, including:
> 
>  * Early-access release of replacing Zookeeper with a
>    self-managed quorum
>  * Add Describe Cluster API
>  * Support mutual TLS authentication on SASL_SSL listeners
>  * Ergonomic improvements to Streams TopologyTestDriver
>  * Logger API improvement to respect the hierarchy
>  * Request/response trace logs are now JSON-formatted
>  * New API to add and remove Streams threads while running
>  * New REST API to expose Connect task configurations
>  * Fixed the TimeWindowDeserializer to be able to
>    deserialize
>    keys outside of Streams (such as in the console consumer)
>  * Streams resilient improvement: new uncaught exception
>    handler
>  * Streams resilience improvement: automatically recover
>    from transient timeout exceptions
> 
> Release notes for the 2.8.0 release:
> https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/RELEASE_NOTES.html
> 
> *** Please download, test and vote by 19 April 2021 ***
> 
> Kafka's KEYS file containing PGP keys we use to sign the
> release:
> https://kafka.apache.org/KEYS
> 
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/
> 
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> 
> * Javadoc:
> https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/javadoc/
> 
> * Tag to be voted upon (off 2.8 branch) is the 2.8.0 tag:
> https://github.com/apache/kafka/releases/tag/2.8.0-rc2
> 
> * Documentation:
> https://kafka.apache.org/28/documentation.html
> 
> * Protocol:
> https://kafka.apache.org/28/protocol.html
> 
> * Successful Jenkins builds for the 2.8 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/
> (still flaky)
> 
> System tests:
>  
> 
> https://jenkins.confluent.io/job/system-test-kafka/job/2.8/6
> 0/ 
>  
> 
> http://confluent-kafka-2-8-system-test-results.s3-us-west-2.amazonaws.com/2021-04-14--001.1618407001--confluentinc--2.8--1b61272d45/report.html
> 
> /**
> 
> Thanks,
> John
> 
> 

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Ismael Juma]

Is the following accurate? Do we have data that not many users would be
affected?

"We also believe that 2.3.0 broker a sufficiently old such that not too
many user may be affected"

I wonder if we should fallback to an actual value if the brokers are 2.3.x
or older.

Ismael

[jira] [Created] (KAFKA-12684) The valid partition list is incorrectly replaced by the successfully elected partition list

[Wenbing Shen (Jira)]

Wenbing Shen created KAFKA-12684:


 Summary: The valid partition list is incorrectly replaced by the 
successfully elected partition list
 Key: KAFKA-12684
 URL: https://issues.apache.org/jira/browse/KAFKA-12684
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 2.7.0, 2.6.0
Reporter: Wenbing Shen
 Fix For: 3.0.0
 Attachments: election-preferred-leader.png, non-preferred-leader.png

When using the kafka-election-tool for preferred replica election, if there are 
partitions in the elected list that are in the preferred replica, the list of 
partitions already in the preferred replica will be replaced by the 
successfully elected partition list.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Luke Chen]

Hi Matthias,
+1 (non-binding)
Thanks for the KIP.

Luke


On Fri, Apr 16, 2021 at 3:32 PM Bruno Cadonna  wrote:

> Thanks Matthias,
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 16.04.21 01:03, Jorge Esteban Quilcate Otoya wrote:
> > +1
> >
> > Thanks Matthias!
> >
> > On Thu, 15 Apr 2021, 20:48 Israel Ekpo,  wrote:
> >
> >> Makes perfect sense to me
> >>
> >> +1 as well.
> >>
> >> Thanks Matthias.
> >>
> >>
> >> On Thu, Apr 15, 2021 at 2:41 PM Guozhang Wang 
> wrote:
> >>
> >>> +1 as well. Thanks!
> >>>
> >>> On Wed, Apr 14, 2021 at 4:30 PM Bill Bejeck  wrote:
> >>>
>  Thanks for the KIP Matthias.
> 
>  +1 (binding)
> 
>  -Bill
> 
>  On Wed, Apr 14, 2021 at 7:06 PM Sophie Blee-Goldman
>   wrote:
> 
> > Thanks Matthias. I'm +1 (binding)
> >
> > -Sophie
> >
> > On Wed, Apr 14, 2021 at 3:36 PM Matthias J. Sax 
>  wrote:
> >
> >> Hi,
> >>
> >> Because this KIP is rather small, I would like to skip a dedicated
> >> discussion thread and call for a vote right way. If there are any
> >> concerns, we can just discuss on this vote thread:
> >>
> >>
> >>
> >
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-733%3A+change+Kafka+Streams+default+replication+factor+config
> >>
> >> Note, that we actually backed this change via
> >>
> >
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722
> >> already.
> >>
> >> However, I felt it might be worth do make this change more explicit
> >>> as
> >> KIP-464 is rather old now.
> >>
> >> Quote from KIP-464:
> >>
> >>> To exploit this new feature in KafkaStreams, we update the
> >> default
> > value
> >> of Streams configuration parameter `replication.factor` from `1` to
>  `-1`.
> >>
> >>
> >>
> >>
> >> -Matthias
> >>
> >
> 
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #49

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #48

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12683) Remove deprecated "UsePreviousTimeOnInvalidTimeStamp"

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-12683:
-

 Summary: Remove deprecated "UsePreviousTimeOnInvalidTimeStamp"
 Key: KAFKA-12683
 URL: https://issues.apache.org/jira/browse/KAFKA-12683
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12633) Remove deprecated "TopologyTestDriver#pipeInput / readOutput"

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-12633.
---
Fix Version/s: 3.0.0
   Resolution: Fixed

> Remove deprecated "TopologyTestDriver#pipeInput / readOutput"
> -
>
> Key: KAFKA-12633
> URL: https://issues.apache.org/jira/browse/KAFKA-12633
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Major
> Fix For: 3.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-12682) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

[jacky (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jacky reopened KAFKA-12682:
---

> Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order 
> -
>
> Key: KAFKA-12682
> URL: https://issues.apache.org/jira/browse/KAFKA-12682
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.8.0
>Reporter: jacky
>Priority: Minor
>
> In version 2.8, MetadataPartitionsBuilder has the field _localChanged and 
> _localRemoved which record the change and delete partition, but we always 
> process _localChanged partitions, and then _localRemoved in the 
> kafka.server.RaftReplicaManager#handleMetadataRecords, not respect the 
> original order, for example, 
> 1. migrate the partition p1 from b0 to b1;
> 2. change the leader of p1 
> 3.migrate p1 from b1 to b0
> and the _localRemoved will delete the p1 at last.
> and I think MetadataPartition should include topic uuid, and the topic name 
> is optional, the topic name only can find the partition by the 
> for example,
> create topic t1, delete topic t1, create topic t1, change leader of p1
> and then compact the records 
> delete topic t1, change t1, p1
> but currently, implementation will be
> 1. process change t1, p1
> 2. process delete topic t1
> but the MetadataPartition doesn't include topic uuid, it only include topic 
> name, when to stop, it can't find the origin topic id, and find the latest 
> the topic id, but it's not right.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12682) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

[jacky (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jacky resolved KAFKA-12682.
---
Resolution: Not A Problem

> Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order 
> -
>
> Key: KAFKA-12682
> URL: https://issues.apache.org/jira/browse/KAFKA-12682
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.8.0
>Reporter: jacky
>Priority: Minor
>
> In version 2.8, MetadataPartitionsBuilder has the field _localChanged and 
> _localRemoved which record the change and delete partition, but we always 
> process _localChanged partitions, and then _localRemoved in the 
> kafka.server.RaftReplicaManager#handleMetadataRecords, not respect the 
> original order, for example, 
> 1. migrate the partition p1 from b0 to b1;
> 2. change the leader of p1 
> 3.migrate p1 from b1 to b0
> and the _localRemoved will delete the p1 at last.
> and I think MetadataPartition should include topic uuid, and the topic name 
> is optional, the topic name only can find the partition by the 
> for example,
> create topic t1, delete topic t1, create topic t1, change leader of p1
> and then compact the records 
> delete topic t1, change t1, p1
> but currently, implementation will be
> 1. process change t1, p1
> 2. process delete topic t1
> but the MetadataPartition doesn't include topic uuid, it only include topic 
> name, when to stop, it can't find the origin topic id, and find the latest 
> the topic id, but it's not right.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-12682) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

[jacky (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jacky reopened KAFKA-12682:
---

> Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order 
> -
>
> Key: KAFKA-12682
> URL: https://issues.apache.org/jira/browse/KAFKA-12682
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.8.0
>Reporter: jacky
>Priority: Minor
>
> In version 2.8, MetadataPartitionsBuilder has the field _localChanged and 
> _localRemoved which record the change and delete partition, but we always 
> process _localChanged partitions, and then _localRemoved in the 
> kafka.server.RaftReplicaManager#handleMetadataRecords, not respect the 
> original order, for example, 
> 1. migrate the partition p1 from b0 to b1;
> 2. change the leader of p1 
> 3.migrate p1 from b1 to b0
> and the _localRemoved will delete the p1 at last.
> and I think MetadataPartition should include topic uuid, and the topic name 
> is optional, the topic name only can find the partition by the 
> for example,
> create topic t1, delete topic t1, create topic t1, change leader of p1
> and then compact the records 
> delete topic t1, change t1, p1
> but currently, implementation will be
> 1. process change t1, p1
> 2. process delete topic t1
> but the MetadataPartition doesn't include topic uuid, it only include topic 
> name, when to stop, it can't find the origin topic id, and find the latest 
> the topic id, but it's not right.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-12682) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

[jacky (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jacky resolved KAFKA-12682.
---
Resolution: Not A Problem

> Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order 
> -
>
> Key: KAFKA-12682
> URL: https://issues.apache.org/jira/browse/KAFKA-12682
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.8.0
>Reporter: jacky
>Priority: Minor
>
> In version 2.8, MetadataPartitionsBuilder has the field _localChanged and 
> _localRemoved which record the change and delete partition, but we always 
> process _localChanged partitions, and then _localRemoved in the 
> kafka.server.RaftReplicaManager#handleMetadataRecords, not respect the 
> original order, for example, 
> 1. migrate the partition p1 from b0 to b1;
> 2. change the leader of p1 
> 3.migrate p1 from b1 to b0
> and the _localRemoved will delete the p1 at last.
> and I think MetadataPartition should include topic uuid, and the topic name 
> is optional, the topic name only can find the partition by the 
> for example,
> create topic t1, delete topic t1, create topic t1, change leader of p1
> and then compact the records 
> delete topic t1, change t1, p1
> but currently, implementation will be
> 1. process change t1, p1
> 2. process delete topic t1
> but the MetadataPartition doesn't include topic uuid, it only include topic 
> name, when to stop, it can't find the origin topic id, and find the latest 
> the topic id, but it's not right.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

[Marco Aurélio Lotz]

Hi Matthias,


Originally I had written a really long answer against deprecations of the
method. When I finished writing, I realised that I actually agree with all
your points.


As you said, if a user is worried about how a table store is persisted, she
will almost always also be worried about how the subscription store is
persisted. Thus we should deprecate the method with only table store
materialization.


I will change the KIP to reflect that. I will also add that we need to
provide information to the users-docs about subscription stores, since I
couldn’t find anything on Kafka documentation about it.


One small question - with us deprecating the API call, should we still have
a bug-fix to piggyback on the materialisation method provided to the
table-store? Not sure if you plan to have it removed at 3.0.0 or would like
to have a longer deprecation time.


Cheers,

Marco Lotz

On Wed, Apr 14, 2021 at 11:29 PM Matthias J. Sax  wrote:

> If you argue this way, would the consequence not be that we would need
> to add even more overloads that allow to only use Materialized for the
> subscription store?
>
> -> I only want to switch to in-memory for the subscription store: Why do
> I need to set Materialized for the main store?
>
> Btw: the above is not something I would really argue for.
>
>
> The surface area of our public API is quite big and tend to overwhelm
> users, and thus I would like to keep it small is possible.
>
>
> > Must the user know about the subscription-stores beforehand? If yes, then
> > indeed we should deprecate.
>
> I tend to argue "yes".
>
>
> Keeping the old methods seems to be a rare use case IMHO. It seems most
> likely that user want to overwrite all stores. And thus, it seems to be
> error prone: I want to overwrite the stores, but I forget to overwrite
> the subscription store.
>
> And even if one wants to keep RocksDB for the subscription stores only,
> they can still pass in both Materialized object using the new methods.
>
>
>
> -Matthias
>
>
> On 4/14/21 1:23 PM, Marco Aurélio Lotz wrote:
> > Hi Matthias,
> >
> > Thanks for reviewing it. That's a nice question. I can provide you the
> > answer as a user of the API - but of course it very much depends on what
> > the original intention towards the user is.
> > The only reason I found out it was opinionated was when my tests using
> > in-memory materialization started failing due to the state-store clean-up
> > bug
> > <
> https://stackoverflow.com/questions/50602512/failed-to-delete-the-state-directory-in-ide-for-kafka-stream-application
> >
> > in Windows systems.
> >
> > To be honest, I think I wouldn't have investigated how subscription
> stores
> > are used without seeing that my system couldn't handle the expected data.
> > Must the user know about the subscription-stores beforehand? If yes, then
> > indeed we should deprecate.
> >
> > Having said that, it seems to me that deprecating would not be consistent
> > with other parts of the API that have parameterized methods hiding
> > complexity.
> > E.g. from the top of my head, join() will make a rocksDB materialization
> if
> > no-materialization method is provided.
> >
> > public  KTable join(final KTable other, final
> > ValueJoiner joiner0
> >
> >
> > I see this as something equivalent. If the user didn't provide all the
> > methods, shouldn't it be fine to fallback to a default materialization
> > method as the other sections of the API already does?
> > In that scenario, there's no need to deprecate and we just need to make a
> > reasonable guess towards it. Using the same method as the table-store
> > sounds like a reasonable guess in this scenario.
> >
> > In my perspective as a user, I would prefer to have simpler methods
> knowing
> > I would be losing some fine-grain configuration. Does it sound reasonable
> > to you?
> >
> > Cheers,
> > Marco Lotz
> >
> > On Wed, Apr 14, 2021 at 8:47 PM Matthias J. Sax 
> wrote:
> >
> >> I just re-read the KIP, and I am wondering why we would *only add* new
> >> methods.
> >>
> >> Is it an expected use case to only change the main stores but not the
> >> subscription stores?
> >>
> >> Wondering if we should deprecate the existing methods that only accept a
> >> single `Materialized` parameter?
> >>
> >>
> >> -Matthias
> >>
> >> On 4/9/21 1:32 PM, Marco Aurélio Lotz wrote:
> >>> Hi everyone,
> >>>
> >>> I am fine with sticking with Materialized and adding the info to the
> >>> javadoc then - so we keep the footprint smaller.
> >>> I will then update the KIP to match what we discussed here and send it
> >> for
> >>> a vote next week.
> >>>
> >>> I will raise a new bug-fix ticket and change KAFKA-10383
> >>>  to become a
> feature
> >> -
> >>> if that's ok.
> >>>
> >>> Cheers,
> >>> Marco
> >>>
> >>> On Wed, Apr 7, 2021 at 4:15 AM Matthias J. Sax 
> wrote:
> >>>
>  Just catching up here.
> 
>  I agree that we have two issue, and the first (align subscription
> store

[jira] [Resolved] (KAFKA-12681) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

[jacky (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jacky resolved KAFKA-12681.
---
Resolution: Duplicate

> Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order 
> -
>
> Key: KAFKA-12681
> URL: https://issues.apache.org/jira/browse/KAFKA-12681
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.8.0
>Reporter: jacky
>Priority: Minor
>
> In version 2.8, MetadataPartitionsBuilder has the field _localChanged and 
> _localRemoved which record the change and delete partition, but we always 
> process _localChanged partitions, and then _localRemoved in the 
> kafka.server.RaftReplicaManager#handleMetadataRecords, not respect the 
> original order, for example, 
> 1. migrate the partition p1 from b0 to b1;
> 2. change the leader of p1 
> 3.migrate p1 from b1 to b0
> and the _localRemoved will delete the p1 at last.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12681) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

[jacky (Jira)]

jacky created KAFKA-12681:
-

 Summary: Kraft MetadataPartitionsBuilder _localChanged and 
_localRemoved out of order 
 Key: KAFKA-12681
 URL: https://issues.apache.org/jira/browse/KAFKA-12681
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: jacky


In version 2.8, MetadataPartitionsBuilder has the field _localChanged and 
_localRemoved which record the change and delete partition, but we always 
process _localChanged partitions, and then _localRemoved in the 
kafka.server.RaftReplicaManager#handleMetadataRecords, not respect the original 
order, for example, 
1. migrate the partition p1 from b0 to b1;
2. change the leader of p1 
3.migrate p1 from b1 to b0
and the _localRemoved will delete the p1 at last.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12682) Kraft MetadataPartitionsBuilder _localChanged and _localRemoved out of order

[jacky (Jira)]

jacky created KAFKA-12682:
-

 Summary: Kraft MetadataPartitionsBuilder _localChanged and 
_localRemoved out of order 
 Key: KAFKA-12682
 URL: https://issues.apache.org/jira/browse/KAFKA-12682
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: jacky


In version 2.8, MetadataPartitionsBuilder has the field _localChanged and 
_localRemoved which record the change and delete partition, but we always 
process _localChanged partitions, and then _localRemoved in the 
kafka.server.RaftReplicaManager#handleMetadataRecords, not respect the original 
order, for example, 
1. migrate the partition p1 from b0 to b1;
2. change the leader of p1 
3.migrate p1 from b1 to b0
and the _localRemoved will delete the p1 at last.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)