Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

[John Roesler]

Hi Will,

This proposal looks good to me overall. Thanks for the contribution!

Just a couple of minor notes:

The system time method would return a cached timestamp that Streams looks up 
once when it starts processing a record. This may be confusing, so it might be 
good to state it in the javadoc.

I thought the javadoc for the stream time might be a bit confusing. We normally 
talk about “Tasks” not “partition groups” in the public api. Maybe just saying 
that it’s “the maximum timestamp of any record yet processed by the task” would 
be both high level and accurate. 

Thanks again!
-John

On Mon, Jun 22, 2020, at 02:10, William Bottrell wrote:
> Thanks, Bruno. I updated the KIP, so hopefully it makes more sense. Thanks
> to Matthias J. Sax and Piotr Smolinski for helping with details.
> 
> I welcome more feedback. Let me know if something doesn't make sense or I
> need to provide more detail. Also, feel free to enlighten me. Thanks!
> 
> On Thu, Jun 11, 2020 at 1:11 PM Bruno Cadonna  wrote:
> 
> > Hi Will,
> >
> > Thank you for the KIP.
> >
> > 1. Could you elaborate a bit more on the motivation in the KIP? An
> > example would make the motivation clearer.
> >
> > 2. In section "Proposed Changes" you do not need to show the
> > implementation and describe internals. A description of the expected
> > behavior of the newly added methods should suffice.
> >
> > 3. In "Compatibility, Deprecation, and Migration Plan" you should
> > state that the change is backward compatible because the two methods
> > will be added and no other method will be changed or removed.
> >
> > Best,
> > Bruno
> >
> > On Wed, Jun 10, 2020 at 10:06 AM William Bottrell 
> > wrote:
> > >
> > > Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext
> > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-622%3A+Add+currentSystemTimeMs+and+currentStreamTimeMs+to+ProcessorContext
> > >
> > >
> > > I am extremely new to Kafka, but thank you to John Roesler and Matthias
> > J.
> > > Sax for pointing me in the right direction. I accept any and all
> > feedback.
> > >
> > > Thanks,
> > > Will
> >
>

[jira] [Reopened] (KAFKA-8197) Flaky Test kafka.server.DynamicBrokerConfigTest > testPasswordConfigEncoderSecretChange

[Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sophie Blee-Goldman reopened KAFKA-8197:


Saw this fail again: 
kafka.server.DynamicBrokerConfigTest > testPasswordConfigEncoderSecretChange 
FAILED*16:33:33* org.junit.ComparisonFailure: expected:<[staticLoginModule 
required;]> but was:<[u`??2?e;%h>r/???8e*16:33:33* at 
org.junit.Assert.assertEquals(Assert.java:117)*16:33:33* at 
org.junit.Assert.assertEquals(Assert.java:146)*16:33:33* at 
kafka.server.DynamicBrokerConfigTest.testPasswordConfigEncoderSecretChange(DynamicBrokerConfigTest.scala:309)

> Flaky Test kafka.server.DynamicBrokerConfigTest > 
> testPasswordConfigEncoderSecretChange
> ---
>
> Key: KAFKA-8197
> URL: https://issues.apache.org/jira/browse/KAFKA-8197
> Project: Kafka
>  Issue Type: Improvement
>  Components: core, unit tests
>Affects Versions: 1.1.1
>Reporter: Guozhang Wang
>Priority: Major
>
> {code}
> 09:18:23 kafka.server.DynamicBrokerConfigTest > 
> testPasswordConfigEncoderSecretChange FAILED
> 09:18:23 org.junit.ComparisonFailure: expected:<[staticLoginModule 
> required;]> but was:<[????O?i???A?c'??Ch?|?p]>
> 09:18:23 at org.junit.Assert.assertEquals(Assert.java:115)
> 09:18:23 at org.junit.Assert.assertEquals(Assert.java:144)
> 09:18:23 at 
> kafka.server.DynamicBrokerConfigTest.testPasswordConfigEncoderSecretChange(DynamicBrokerConfigTest.scala:253)
> {code}
> https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/13466/consoleFull



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache Kafka 2.6.0 release

[Boyang Chen]

Hey Randal,

There was another spotted blocker:
https://issues.apache.org/jira/browse/KAFKA-10173
As of current, John is working on a fix.

Boyang

On Wed, Jun 24, 2020 at 4:08 PM Sophie Blee-Goldman 
wrote:

> Hey all,
>
> Just a heads up that we discovered a new blocker. The fix is pretty
> straightforward
> and there's already a PR for it so it should be resolved quickly.
>
> Here's the ticket: https://issues.apache.org/jira/browse/KAFKA-10198
>
> On Sat, May 30, 2020 at 12:52 PM Randall Hauch  wrote:
>
> > Hi, Kowshik,
> >
> > Thanks for the update on KIP-584. This is listed on the "Postponed"
> section
> > of the AK 2.6.0 release plan (
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > ).
> >
> > Best regards,
> >
> > Randall
> >
> > On Fri, May 29, 2020 at 4:51 PM Kowshik Prakasam  >
> > wrote:
> >
> > > Hi Randall,
> > >
> > > We have to remove KIP-584 from the release plan, as this item will not
> be
> > > completed for 2.6 release (although KIP is accepted). We plan to
> include
> > it
> > > in a next release.
> > >
> > >
> > > Cheers,
> > > Kowshik
> > >
> > >
> > > On Fri, May 29, 2020 at 11:43 AM Maulin Vasavada <
> > > maulin.vasav...@gmail.com>
> > > wrote:
> > >
> > > > Hi Randall Hauch
> > > >
> > > > Can we add KIP-519 to 2.6? It was merged to Trunk already in April -
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952
> > > > .
> > > >
> > > > Thanks
> > > > Maulin
> > > >
> > > > On Fri, May 29, 2020 at 11:01 AM Randall Hauch 
> > wrote:
> > > >
> > > > > Here's an update on the AK 2.6.0 release.
> > > > >
> > > > > Code freeze was Wednesday, and the release plan [1] has been
> updated
> > to
> > > > > reflect all of the KIPs that made the release. We've also cut the
> > `2.6`
> > > > > branch that we'll use for the release; see separate email
> announcing
> > > the
> > > > > new branch.
> > > > >
> > > > > The next important date for the 2.6.0 release is CODE FREEZE on
> JUNE
> > > 10,
> > > > > and until that date all bug fixes are still welcome on the release
> > > > branch.
> > > > > But after that, only blocker bugs can be merged to the release
> > branch.
> > > > >
> > > > > If you have any questions or concerns, please contact me or (better
> > > yet)
> > > > > reply to this thread.
> > > > >
> > > > > Thanks, and best regards!
> > > > >
> > > > > Randall
> > > > >
> > > > > [1] AK 2.6.0 Release Plan:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > > >
> > > > >
> > > > > On Wed, May 27, 2020 at 5:53 PM Matthias J. Sax 
> > > > wrote:
> > > > >
> > > > > > Thanks Randall!
> > > > > >
> > > > > > I added missing KIP-594.
> > > > > >
> > > > > >
> > > > > > For the postponed KIP section: I removed KIP-441 and KIP-444 as
> > both
> > > > are
> > > > > > completed.
> > > > > >
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 5/27/20 2:31 PM, Randall Hauch wrote:
> > > > > > > Hey everyone, just a quick update on the 2.6.0 release.
> > > > > > >
> > > > > > > Based on the release plan (
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > > > > ),
> > > > > > > today (May 27) is feature freeze. Any major feature work that
> is
> > > not
> > > > > > > already complete will need to push out to the next release
> > (either
> > > > 2.7
> > > > > or
> > > > > > > 3.0). There are a few PRs for KIPs that are nearing completion,
> > and
> > > > > we're
> > > > > > > having some Jenkins build issues. I will send another email
> later
> > > > today
> > > > > > or
> > > > > > > early tomorrow with an update, and I plan to cut the release
> > branch
> > > > > > shortly
> > > > > > > thereafter.
> > > > > > >
> > > > > > > I have also updated the list of planned KIPs on the release
> plan
> > > > page (
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > > > > ),
> > > > > > > and I've moved to the "Postponed" table any KIP that looks like
> > it
> > > is
> > > > > not
> > > > > > > going to be complete today. If any KIP is in the wrong table,
> > > please
> > > > > let
> > > > > > me
> > > > > > > know.
> > > > > > >
> > > > > > > If you have any questions or concerns, please feel free to
> reply
> > to
> > > > > this
> > > > > > > thread.
> > > > > > >
> > > > > > > Thanks, and best regards!
> > > > > > >
> > > > > > > Randall
> > > > > > >
> > > > > > > On Wed, May 20, 2020 at 2:16 PM Sophie Blee-Goldman <
> > > > > sop...@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hey Randall,
> > > > > > >>
> > > > > > >> Can you also add KIP-613 which was accepted yesterday?
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >> Sophie
> > > > > > >>
> > > > > > >> On Wed, May 20, 2020 at 6:47 AM Randall Hauch <
> rha...@gmail.com
> > >
> > > > > wrote:
> > > > 

Re: [ANNOUNCE] New committer: Xi Hu

[Vahid Hashemian]

Congrats Xi!

--Vahid

On Wed, Jun 24, 2020 at 2:56 PM Mickael Maison 
wrote:

> Congratulations Xi!
>
> On Wed, Jun 24, 2020 at 7:25 PM Matthias J. Sax  wrote:
> >
> > Congrats Xi!
> >
> > On 6/24/20 9:45 AM, Tom Bentley wrote:
> > > Congratulations Xi!
> > >
> > > On Wed, Jun 24, 2020 at 5:34 PM Guozhang Wang 
> wrote:
> > >
> > >> The PMC for Apache Kafka has invited Xi Hu as a committer and we are
> > >> pleased to announce that he has accepted!
> > >>
> > >> Xi Hu has been actively contributing to Kafka since 2016, and is well
> > >> recognized especially for his non-code contributions: he maintains a
> tech
> > >> blog post evangelizing Kafka in the Chinese speaking community (
> > >> https://www.cnblogs.com/huxi2b/), and is one of the most active
> answering
> > >> member in Zhihu (Chinese Reddit / StackOverflow) Kafka topic. He has
> > >> presented in Kafka meetup events in the past and authored a
> > >> book deep-diving on Kafka architecture design and operations as well (
> > >> https://www.amazon.cn/dp/B07JH9G2FL). Code wise, he has contributed
> 75
> > >> patches so far.
> > >>
> > >>
> > >> Thanks for all the contributions Xi. Congratulations!
> > >>
> > >> -- Guozhang, on behalf of the Apache Kafka PMC
> > >>
> > >
> >
>


-- 

Thanks!
--Vahid

Re: [DISCUSS] Apache Kafka 2.6.0 release

[Sophie Blee-Goldman]

Hey all,

Just a heads up that we discovered a new blocker. The fix is pretty
straightforward
and there's already a PR for it so it should be resolved quickly.

Here's the ticket: https://issues.apache.org/jira/browse/KAFKA-10198

On Sat, May 30, 2020 at 12:52 PM Randall Hauch  wrote:

> Hi, Kowshik,
>
> Thanks for the update on KIP-584. This is listed on the "Postponed" section
> of the AK 2.6.0 release plan (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> ).
>
> Best regards,
>
> Randall
>
> On Fri, May 29, 2020 at 4:51 PM Kowshik Prakasam 
> wrote:
>
> > Hi Randall,
> >
> > We have to remove KIP-584 from the release plan, as this item will not be
> > completed for 2.6 release (although KIP is accepted). We plan to include
> it
> > in a next release.
> >
> >
> > Cheers,
> > Kowshik
> >
> >
> > On Fri, May 29, 2020 at 11:43 AM Maulin Vasavada <
> > maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi Randall Hauch
> > >
> > > Can we add KIP-519 to 2.6? It was merged to Trunk already in April -
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=128650952
> > > .
> > >
> > > Thanks
> > > Maulin
> > >
> > > On Fri, May 29, 2020 at 11:01 AM Randall Hauch 
> wrote:
> > >
> > > > Here's an update on the AK 2.6.0 release.
> > > >
> > > > Code freeze was Wednesday, and the release plan [1] has been updated
> to
> > > > reflect all of the KIPs that made the release. We've also cut the
> `2.6`
> > > > branch that we'll use for the release; see separate email announcing
> > the
> > > > new branch.
> > > >
> > > > The next important date for the 2.6.0 release is CODE FREEZE on JUNE
> > 10,
> > > > and until that date all bug fixes are still welcome on the release
> > > branch.
> > > > But after that, only blocker bugs can be merged to the release
> branch.
> > > >
> > > > If you have any questions or concerns, please contact me or (better
> > yet)
> > > > reply to this thread.
> > > >
> > > > Thanks, and best regards!
> > > >
> > > > Randall
> > > >
> > > > [1] AK 2.6.0 Release Plan:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > >
> > > >
> > > > On Wed, May 27, 2020 at 5:53 PM Matthias J. Sax 
> > > wrote:
> > > >
> > > > > Thanks Randall!
> > > > >
> > > > > I added missing KIP-594.
> > > > >
> > > > >
> > > > > For the postponed KIP section: I removed KIP-441 and KIP-444 as
> both
> > > are
> > > > > completed.
> > > > >
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 5/27/20 2:31 PM, Randall Hauch wrote:
> > > > > > Hey everyone, just a quick update on the 2.6.0 release.
> > > > > >
> > > > > > Based on the release plan (
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > > > ),
> > > > > > today (May 27) is feature freeze. Any major feature work that is
> > not
> > > > > > already complete will need to push out to the next release
> (either
> > > 2.7
> > > > or
> > > > > > 3.0). There are a few PRs for KIPs that are nearing completion,
> and
> > > > we're
> > > > > > having some Jenkins build issues. I will send another email later
> > > today
> > > > > or
> > > > > > early tomorrow with an update, and I plan to cut the release
> branch
> > > > > shortly
> > > > > > thereafter.
> > > > > >
> > > > > > I have also updated the list of planned KIPs on the release plan
> > > page (
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=152113430
> > > > > ),
> > > > > > and I've moved to the "Postponed" table any KIP that looks like
> it
> > is
> > > > not
> > > > > > going to be complete today. If any KIP is in the wrong table,
> > please
> > > > let
> > > > > me
> > > > > > know.
> > > > > >
> > > > > > If you have any questions or concerns, please feel free to reply
> to
> > > > this
> > > > > > thread.
> > > > > >
> > > > > > Thanks, and best regards!
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > On Wed, May 20, 2020 at 2:16 PM Sophie Blee-Goldman <
> > > > sop...@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hey Randall,
> > > > > >>
> > > > > >> Can you also add KIP-613 which was accepted yesterday?
> > > > > >>
> > > > > >> Thanks!
> > > > > >> Sophie
> > > > > >>
> > > > > >> On Wed, May 20, 2020 at 6:47 AM Randall Hauch  >
> > > > wrote:
> > > > > >>
> > > > > >>> Hi, Tom. I saw last night that the KIP had enough votes before
> > > > today’s
> > > > > >>> deadline and I will add it to the roadmap today. Thanks for
> > driving
> > > > > this!
> > > > > >>>
> > > > > >>> On Wed, May 20, 2020 at 6:18 AM Tom Bentley <
> tbent...@redhat.com
> > >
> > > > > wrote:
> > > > > >>>
> > > > >  Hi Randall,
> > > > > 
> > > > >  Can we add KIP-585? (I'm not quite sure of the protocol here,
> > but
> > > > > >> thought
> > > > >  it better to ask than to just add it myself).
> > > > > 
> > > > >  Thanks,
> > > > > 
> > > > > 

[jira] [Created] (KAFKA-10198) Dirty tasks may be recycled instead of closed

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-10198:
---

 Summary: Dirty tasks may be recycled instead of closed
 Key: KAFKA-10198
 URL: https://issues.apache.org/jira/browse/KAFKA-10198
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Sophie Blee-Goldman
Assignee: Sophie Blee-Goldman
 Fix For: 2.6.0


We recently added a guard to `Task#closeClean` to make sure we don't 
accidentally clean-close a dirty task, but we forgot to also add this check to 
`Task#closeAndRecycleState`. This meant an otherwise dirty task could be closed 
clean and recycled into a new task when it should have just been closed.

This manifest as an NPE in our test application. Specifically, task 1_0 was 
active on StreamThread-2 but reassigned as a standby. During handleRevocation 
we hit a TaskMigratedException while flushing the tasks and bailed on trying to 
flush and commit the remainder. This left task 1_0 with dirty keys in the 
suppression buffer and the `commitNeeded` flag still set to true.

During handleAssignment, we should have closed all the tasks with pending state 
as dirty (ie any task with commitNeeded = true). Since we don't know about the 
TaskMigratedException we hit during handleRevocation, we rely on the guard in 
Task#closeClean` to throw an exception and force the task to be closed dirty.

Unfortunately, we left this guard out of `closeAndRecycleState`, which meant 
task 1_0 was able to slip through without being closed dirty. Once 
reinitialized as a standby task, we eventually tried to commit it. The 
suppression buffer of course tried to flush its remaining dirty keys from its 
previous life as an active task. But since it's now a standby task, it should 
not be sending anything to the changelog and has a null RecordCollector. We 
tried to access it, and hit the NPE.

 

The fix is simple, we just need to add the guard in closeClean to 
closeAndRecycleState as well



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New committer: Boyang Chen

[Liquan Pei]

Congrats!

On Wed, Jun 24, 2020 at 9:42 AM Raymond Ng  wrote:

> Congrats Boyang! Look forward to more awesome contributions from you in the
> future.
>
> Regards,
> Ray
>
> On Wed, Jun 24, 2020 at 6:07 AM Ismael Juma  wrote:
>
> > Congratulations Boyang!
> >
> > Ismael
> >
> > On Mon, Jun 22, 2020 at 4:26 PM Guozhang Wang 
> wrote:
> >
> > > The PMC for Apache Kafka has invited Boyang Chen as a committer and we
> > are
> > > pleased to announce that he has accepted!
> > >
> > > Boyang has been active in the Kafka community more than two years ago.
> > > Since then he has presented his experience operating with Kafka Streams
> > at
> > > Pinterest as well as several feature development including rebalance
> > > improvements (KIP-345) and exactly-once scalability improvements
> > (KIP-447)
> > > in various Kafka Summit and Kafka Meetups. More recently he's also been
> > > participating in Kafka broker development including post-Zookeeper
> > > controller design (KIP-500). Besides all the code contributions, Boyang
> > has
> > > also helped reviewing even more PRs and KIPs than his own.
> > >
> > > Thanks for all the contributions Boyang! And look forward to more
> > > collaborations with you on Apache Kafka.
> > >
> > >
> > > -- Guozhang, on behalf of the Apache Kafka PMC
> > >
> >
>
-- 
Liquan Pei
Software Engineer, Confluent Inc

Re: [ANNOUNCE] New committer: Xi Hu

[Mickael Maison]

Congratulations Xi!

On Wed, Jun 24, 2020 at 7:25 PM Matthias J. Sax  wrote:
>
> Congrats Xi!
>
> On 6/24/20 9:45 AM, Tom Bentley wrote:
> > Congratulations Xi!
> >
> > On Wed, Jun 24, 2020 at 5:34 PM Guozhang Wang  wrote:
> >
> >> The PMC for Apache Kafka has invited Xi Hu as a committer and we are
> >> pleased to announce that he has accepted!
> >>
> >> Xi Hu has been actively contributing to Kafka since 2016, and is well
> >> recognized especially for his non-code contributions: he maintains a tech
> >> blog post evangelizing Kafka in the Chinese speaking community (
> >> https://www.cnblogs.com/huxi2b/), and is one of the most active answering
> >> member in Zhihu (Chinese Reddit / StackOverflow) Kafka topic. He has
> >> presented in Kafka meetup events in the past and authored a
> >> book deep-diving on Kafka architecture design and operations as well (
> >> https://www.amazon.cn/dp/B07JH9G2FL). Code wise, he has contributed 75
> >> patches so far.
> >>
> >>
> >> Thanks for all the contributions Xi. Congratulations!
> >>
> >> -- Guozhang, on behalf of the Apache Kafka PMC
> >>
> >
>

[VOTE] KIP-623: Add "internal-topics" option to streams application reset tool

[Joel Wee]

Apologies. Changing the subject.

On 24 Jun 2020, at 9:14 PM, Joel Wee 
mailto:joel@outlook.com>> wrote:

Hi all

I would like to start a vote for KIP-623, which adds the option 
--internal-topics to the streams-application-reset-tool: 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158862177.

Please let me know what you think.

Best

Joel

[DISCUSS] KIP-623: Add "internal-topics" option to streams application reset tool

[Joel Wee]

Hi all

I would like to start a vote for KIP-623, which adds the option 
--internal-topics to the streams-application-reset-tool: 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158862177.

Please let me know what you think.

Best

Joel

[jira] [Resolved] (KAFKA-10135) Extract Task#executeAndMaybeSwallow to be a general utility function into TaskManager

[Boyang Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-10135.
-
Resolution: Fixed

> Extract Task#executeAndMaybeSwallow to be a general utility function into 
> TaskManager
> -
>
> Key: KAFKA-10135
> URL: https://issues.apache.org/jira/browse/KAFKA-10135
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Boyang Chen
>Assignee: feyman
>Priority: Major
>
> We have a couple of cases where we need to swallow the exception during 
> operations in both Task class and TaskManager class. This utility method 
> should be generalized at least onto TaskManager level. See discussion comment 
> [here|https://github.com/apache/kafka/pull/8833#discussion_r437697665].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : kafka-2.3-jdk8 #217

[Apache Jenkins Server]

See 

Re: [ANNOUNCE] New committer: Xi Hu

[Matthias J. Sax]

Congrats Xi!

On 6/24/20 9:45 AM, Tom Bentley wrote:
> Congratulations Xi!
> 
> On Wed, Jun 24, 2020 at 5:34 PM Guozhang Wang  wrote:
> 
>> The PMC for Apache Kafka has invited Xi Hu as a committer and we are
>> pleased to announce that he has accepted!
>>
>> Xi Hu has been actively contributing to Kafka since 2016, and is well
>> recognized especially for his non-code contributions: he maintains a tech
>> blog post evangelizing Kafka in the Chinese speaking community (
>> https://www.cnblogs.com/huxi2b/), and is one of the most active answering
>> member in Zhihu (Chinese Reddit / StackOverflow) Kafka topic. He has
>> presented in Kafka meetup events in the past and authored a
>> book deep-diving on Kafka architecture design and operations as well (
>> https://www.amazon.cn/dp/B07JH9G2FL). Code wise, he has contributed 75
>> patches so far.
>>
>>
>> Thanks for all the contributions Xi. Congratulations!
>>
>> -- Guozhang, on behalf of the Apache Kafka PMC
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-629: Use racially neutral terms in our codebase

[Jorge Esteban Quilcate Otoya]

Hi Xavier,

Thank you for this proposal! It's awesome to see the community taking
action on this.

`include` and `exclude` make sense to me.

Probably obvious but is documentation and website considered as well as
part of the KIP?
This would be interesting because it could be also important to make these
guidelines explicit for other components in the ecosystem (e.g. connectors)
to follow.
What do you think?

Cheers,
Jorge.

On Tue, Jun 23, 2020 at 8:38 AM Bruno Cadonna  wrote:

> Hi Xavier,
>
> Thank you very much for starting this initiative!
> Not only for the changes to the code base but also for showing me
> where and how we can use more appropriate terms in general.
>
> Best,
> Bruno
>
> On Tue, Jun 23, 2020 at 4:17 AM John Roesler  wrote:
> >
> > Hi Xavier,
> >
> > I think your approach made a lot of sense; I definitely didn’t mean to
> criticize. Thanks for the update! The new names look good to me.
> >
> > -John
> >
> > On Mon, Jun 22, 2020, at 18:50, Matthias J. Sax wrote:
> > > Great initiative!
> > >
> > > I liked the proposed names, too.
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 6/22/20 4:48 PM, Guozhang Wang wrote:
> > > > Xavier, thanks for the KIP! The proposed names make sense to me.
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Jun 22, 2020 at 4:24 PM Xavier Léauté 
> wrote:
> > > >
> > > >> Please check the list for updated config / argument names.
> > > >>
> > > >> I also added a proposal to replace the term "blackout" with
> "backoff",
> > > >> which is used internally in the replication protocol.
> > > >>
> > > >> On Mon, Jun 22, 2020 at 3:10 PM Xavier Léauté 
> wrote:
> > > >>
> > > >>> I agree we could improve on some of the config names. My thinking
> here is
> > > >>> that unless we had some precedent for a different name, it seemed
> > > >>> relatively straightforward to follow the approach other open source
> > > >>> projects have taken. It also makes migration for users easy if we
> are
> > > >>> consistent in the renaming, so we should find terms we can use
> across the
> > > >>> board.
> > > >>>
> > > >>> A cursory search indicates we already use include/exclude for topic
> > > >>> creation config in Connect, so I think it makes sense to align on
> that.
> > > >>> I'll update the KIP accordingly.
> > > >>>
> > > >>> On Sat, Jun 20, 2020 at 11:37 AM Ryanne Dolan <
> ryannedo...@gmail.com>
> > > >>> wrote:
> > > >>>
> > >  Xavier, I'm dismayed to see some of these instances are my fault.
> Fully
> > >  support your plan.
> > > 
> > >  John, I had the same thought -- "list" is extraneous here. In the
> case
> > > >> of
> > >  "topics.whitelist" we already have precedent to just use "topics".
> > > 
> > >  Ryanne
> > > 
> > >  On Sat, Jun 20, 2020, 12:43 PM John Roesler 
> > > >> wrote:
> > > 
> > > > Thanks Xavier!
> > > >
> > > > I’m +1 on this idea, and I’m glad this is the extent of what
> needs to
> > > >>> be
> > > > changed. I recall when I joined the project being pleased at the
> lack
> > > >>> of
> > > > common offensive terminology. I hadn’t considered
> > > >> whitelist/blacklist,
> > >  but
> > > > I can see the argument.
> > > >
> > > > Allowlist/blocklist are kind of a mouthful, though.
> > > >
> > > > What do you think of just “allow” and “deny” instead? This is
> common
> > > > terminology in ACLs for example, and it doesn’t really seem
> necessary
> > > >>> to
> > > > say “list” in the config name.
> > > >
> > > > Alternatively, looking at the actual configs, it seems like
> > > >> “include”,
> > > > “include-only” (or “only”) and “exclude” might be more
> appropriate in
> > > > context.
> > > >
> > > > I hope this doesn’t kick off a round of bikeshedding. I’m really
> fine
> > > > either way; I doubt it matters much. I just wanted to see if we
> can
> > > >>> name
> > > > these configs without making up new multi-syllable words.
> > > >
> > > > Thanks for bringing it up!
> > > > -John
> > > >
> > > > On Sat, Jun 20, 2020, at 09:31, Ron Dagostino wrote:
> > > >> Yes.  Thank you.
> > > >>
> > > >>> On Jun 20, 2020, at 12:20 AM, Gwen Shapira 
> > >  wrote:
> > > >>>
> > > >>> Thank you so much for this initiative. Small change, but it
> makes
> > > >>> our
> > > >>> community more inclusive.
> > > >>>
> > > >>> Gwen
> > > >>>
> > >  On Fri, Jun 19, 2020, 6:02 PM Xavier Léauté 
> > >  wrote:
> > > 
> > >  Hi Everyone,
> > > 
> > >  There are a number of places in our codebase that use racially
> > >  charged
> > >  terms. I am proposing we update them to use more neutral
> terms.
> > > 
> > >  The KIP lists the ones I have found and proposes alternatives.
> > > >> If
> > >  you
> > > > see
> > >  any I missed or did not consider, please reply and I'll add
> > > >> them.
> > > 
> >

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Gokul Ramanan Subramanian]

Thanks Gwen and Ismael.

On Wed, Jun 24, 2020 at 6:52 PM Gwen Shapira  wrote:

> Kafka follows a "fully compatible model" and new features are always
> optional. We have extensive compatibility testing to make sure rolling
> upgrades are safe and painless.
> This investment in compatibility allows us to reduce the effort needed in
> maintaining old versions (Security CVE is the obvious exception, as well as
> data-loss bugs).
>
> Not every OSS project has the same model, but it has worked for Kafka for
> years now (since the awful upgrade from 0.7 to 0.8 in 2014).
>
> Gwen
>
>
> On Wed, Jun 24, 2020 at 10:40 AM Gokul Ramanan Subramanian <
> gokul24...@gmail.com> wrote:
>
> > I agree that is an option, but is there any reason to not have a 2.3.2
> and
> > 2.4.2 released? If so, it would be nice to know about these reasons.
> > Appreciate your time on this.
> >
> > On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma  wrote:
> >
> > > I think there's some confusion here. You can upgrade to AK 2.5.0 and
> > > completely ignore ZK and TLS. It's completely optional.
> > >
> > > Ismael
> > >
> > > On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian <
> > > gokul24...@gmail.com> wrote:
> > >
> > > > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of
> migrating
> > > > from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it
> > > would
> > > > be nice to have a bug-free version of 2.3 and 2.4.
> > > >
> > > > Is there a technical reason why we hesitate to get these versions
> out?
> > Or
> > > > is it that no one has got around to it?
> > > >
> > > > On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia <
> > > sankalpbhati...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Ismael for the response.
> > > > >
> > > > > For our clusters running 2.3.1 and 2.4.1, we saw some issues which
> > had
> > > > > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it
> > > > > introduces some major changes like support for ZK encryption and a
> > few
> > > > > others, I was wondering if we should choose a smaller upgrade in
> such
> > > > cases
> > > > > as we don't really require the new features in 2.5 and above right
> > now.
> > > > >
> > > > > -
> > > > > Sankalp
> > > > >
> > > > > On Wed, 24 Jun 2020 at 14:23, Ismael Juma 
> wrote:
> > > > >
> > > > > > Hi Sankalp,
> > > > > >
> > > > > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0
> > > instead?
> > > > > We
> > > > > > are working on the 2.5.1 release, which would be the recommended
> > > > release.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia <
> > > > > sankalpbhati...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I would like to know if there are any plans to release a 2.3.2
> > and
> > > > > 2.4.2
> > > > > > > versions for Apache Kafka in the near future. I see there are
> > some
> > > > > issues
> > > > > > > marked as fixed in these two versions
> > > > > > > (https://tinyurl.com/ycdpz5cb).
> > > > > > > However, I could not find a branch/tag corresponding to these
> > > > versions
> > > > > in
> > > > > > > the github repository (https://github.com/apache/kafka).
> > > > > > >
> > > > > > > Also, It would be great if someone can help me understand or
> > share
> > > > any
> > > > > > > documentation around the release processes (specifically on
> when
> > we
> > > > > > decide
> > > > > > > to release a new bug fix version like the ones mentioned
> above.)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sankalp
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Gwen Shapira]

Kafka follows a "fully compatible model" and new features are always
optional. We have extensive compatibility testing to make sure rolling
upgrades are safe and painless.
This investment in compatibility allows us to reduce the effort needed in
maintaining old versions (Security CVE is the obvious exception, as well as
data-loss bugs).

Not every OSS project has the same model, but it has worked for Kafka for
years now (since the awful upgrade from 0.7 to 0.8 in 2014).

Gwen


On Wed, Jun 24, 2020 at 10:40 AM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:

> I agree that is an option, but is there any reason to not have a 2.3.2 and
> 2.4.2 released? If so, it would be nice to know about these reasons.
> Appreciate your time on this.
>
> On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma  wrote:
>
> > I think there's some confusion here. You can upgrade to AK 2.5.0 and
> > completely ignore ZK and TLS. It's completely optional.
> >
> > Ismael
> >
> > On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian <
> > gokul24...@gmail.com> wrote:
> >
> > > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating
> > > from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it
> > would
> > > be nice to have a bug-free version of 2.3 and 2.4.
> > >
> > > Is there a technical reason why we hesitate to get these versions out?
> Or
> > > is it that no one has got around to it?
> > >
> > > On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia <
> > sankalpbhati...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Ismael for the response.
> > > >
> > > > For our clusters running 2.3.1 and 2.4.1, we saw some issues which
> had
> > > > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it
> > > > introduces some major changes like support for ZK encryption and a
> few
> > > > others, I was wondering if we should choose a smaller upgrade in such
> > > cases
> > > > as we don't really require the new features in 2.5 and above right
> now.
> > > >
> > > > -
> > > > Sankalp
> > > >
> > > > On Wed, 24 Jun 2020 at 14:23, Ismael Juma  wrote:
> > > >
> > > > > Hi Sankalp,
> > > > >
> > > > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0
> > instead?
> > > > We
> > > > > are working on the 2.5.1 release, which would be the recommended
> > > release.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia <
> > > > sankalpbhati...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I would like to know if there are any plans to release a 2.3.2
> and
> > > > 2.4.2
> > > > > > versions for Apache Kafka in the near future. I see there are
> some
> > > > issues
> > > > > > marked as fixed in these two versions
> > > > > > (https://tinyurl.com/ycdpz5cb).
> > > > > > However, I could not find a branch/tag corresponding to these
> > > versions
> > > > in
> > > > > > the github repository (https://github.com/apache/kafka).
> > > > > >
> > > > > > Also, It would be great if someone can help me understand or
> share
> > > any
> > > > > > documentation around the release processes (specifically on when
> we
> > > > > decide
> > > > > > to release a new bug fix version like the ones mentioned above.)
> > > > > >
> > > > > > Thanks,
> > > > > > Sankalp
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

[Guozhang Wang]

@Jun Rao 

Regarding your comment about log compaction. After some deep-diving into
this we've decided to propose a new snapshot-based log cleaning mechanism
which would be used to replace the current compaction mechanism for this
meta log. A new KIP will be proposed specifically for this idea.

All,

I've updated the KIP wiki a bit updating one config "election.jitter.max.ms"
to "election.backoff.max.ms" to make it more clear about the usage: the
configured value will be the upper bound of the binary exponential backoff
time after a failed election, before starting a new one.



Guozhang



On Fri, Jun 12, 2020 at 9:26 AM Boyang Chen 
wrote:

> Thanks for the suggestions Guozhang.
>
> On Thu, Jun 11, 2020 at 2:51 PM Guozhang Wang  wrote:
>
> > Hello Boyang,
> >
> > Thanks for the updated information. A few questions here:
> >
> > 1) Should the quorum-file also update to support multi-raft?
> >
> > I'm neutral about this, as we don't know yet how the multi-raft modules
> would behave. If
> we have different threads operating different raft groups, consolidating
> the `checkpoint` files seems
> not reasonable. We could always add `multi-quorum-file` later if possible.
>
> 2) In the previous proposal, there's fields in the FetchQuorumRecords like
> > latestDirtyOffset, is that dropped intentionally?
> >
> > I dropped the latestDirtyOffset since it is associated with the log
> compaction discussion. This is beyond this KIP scope and we could
> potentially get a separate KIP to talk about it.
>
>
> > 3) I think we also need to elaborate a bit more details regarding when to
> > send metadata request and discover-brokers; currently we only discussed
> > during bootstrap how these requests would be sent. I think the following
> > scenarios would also need these requests
> >
> > 3.a) As long as a broker does not know the current quorum (including the
> > leader and the voters), it should continue periodically ask other brokers
> > via "metadata.
> > 3.b) As long as a broker does not know all the current quorum voter's
> > connections, it should continue periodically ask other brokers via
> > "discover-brokers".
> > 3.c) When the leader's fetch timeout elapsed, it should send metadata
> > request.
> >
> > Make sense, will add to the KIP.
>
> >
> > Guozhang
> >
> >
> > On Wed, Jun 10, 2020 at 5:20 PM Boyang Chen 
> > wrote:
> >
> > > Hey all,
> > >
> > > follow-up on the previous email, we made some more updates:
> > >
> > > 1. The Alter/DescribeQuorum RPCs are also re-structured to use
> > multi-raft.
> > >
> > > 2. We add observer status into the DescribeQuorumResponse as we see it
> > is a
> > > low hanging fruit which is very useful for user debugging and
> > reassignment.
> > >
> > > 3. The FindQuorum RPC is replaced with DiscoverBrokers RPC, which is
> > purely
> > > in charge of discovering broker connections in a gossip manner. The
> > quorum
> > > leader discovery is piggy-back on the Metadata RPC for the topic
> > partition
> > > leader, which in our case is the single metadata partition for the
> > version
> > > one.
> > >
> > > Let me know if you have any questions.
> > >
> > > Boyang
> > >
> > > On Tue, Jun 9, 2020 at 11:01 PM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Hey all,
> > > >
> > > > Thanks for the great discussions so far. I'm posting some KIP updates
> > > from
> > > > our working group discussion:
> > > >
> > > > 1. We will be changing the core RPCs from single-raft API to
> > multi-raft.
> > > > This means all protocols will be "batch" in the first version, but
> the
> > > KIP
> > > > itself only illustrates the design for a single metadata topic
> > partition.
> > > > The reason is to "keep the door open" for future extensions of this
> > piece
> > > > of module such as a sharded controller or general quorum based topic
> > > > replication, beyond the current Kafka replication protocol.
> > > >
> > > > 2. We will piggy-back on the current Kafka Fetch API instead of
> > inventing
> > > > a new FetchQuorumRecords RPC. The motivation is about the same as #1
> as
> > > > well as making the integration work easier, instead of letting two
> > > similar
> > > > RPCs diverge.
> > > >
> > > > 3. In the EndQuorumEpoch protocol, instead of only sending the
> request
> > to
> > > > the most caught-up voter, we shall broadcast the information to all
> > > voters,
> > > > with a sorted voter list in descending order of their corresponding
> > > > replicated offset. In this way, the top voter will become a candidate
> > > > immediately, while the other voters shall wait for an exponential
> > > back-off
> > > > to trigger elections, which helps ensure the top voter gets elected,
> > and
> > > > the election eventually happens when the top voter is not responsive.
> > > >
> > > > Please see the updated KIP and post any questions or concerns on the
> > > > mailing thread.
> > > >
> > > > Boyang
> > > >
> > > > On Fri, May 8, 2020 at 5:26 PM Jun Rao  wrote:
> > > >
> > > >> Hi, 

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Gokul Ramanan Subramanian]

I agree that is an option, but is there any reason to not have a 2.3.2 and
2.4.2 released? If so, it would be nice to know about these reasons.
Appreciate your time on this.

On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma  wrote:

> I think there's some confusion here. You can upgrade to AK 2.5.0 and
> completely ignore ZK and TLS. It's completely optional.
>
> Ismael
>
> On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian <
> gokul24...@gmail.com> wrote:
>
> > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating
> > from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it
> would
> > be nice to have a bug-free version of 2.3 and 2.4.
> >
> > Is there a technical reason why we hesitate to get these versions out? Or
> > is it that no one has got around to it?
> >
> > On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia <
> sankalpbhati...@gmail.com>
> > wrote:
> >
> > > Thanks Ismael for the response.
> > >
> > > For our clusters running 2.3.1 and 2.4.1, we saw some issues which had
> > > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it
> > > introduces some major changes like support for ZK encryption and a few
> > > others, I was wondering if we should choose a smaller upgrade in such
> > cases
> > > as we don't really require the new features in 2.5 and above right now.
> > >
> > > -
> > > Sankalp
> > >
> > > On Wed, 24 Jun 2020 at 14:23, Ismael Juma  wrote:
> > >
> > > > Hi Sankalp,
> > > >
> > > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0
> instead?
> > > We
> > > > are working on the 2.5.1 release, which would be the recommended
> > release.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia <
> > > sankalpbhati...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I would like to know if there are any plans to release a 2.3.2 and
> > > 2.4.2
> > > > > versions for Apache Kafka in the near future. I see there are some
> > > issues
> > > > > marked as fixed in these two versions
> > > > > (https://tinyurl.com/ycdpz5cb).
> > > > > However, I could not find a branch/tag corresponding to these
> > versions
> > > in
> > > > > the github repository (https://github.com/apache/kafka).
> > > > >
> > > > > Also, It would be great if someone can help me understand or share
> > any
> > > > > documentation around the release processes (specifically on when we
> > > > decide
> > > > > to release a new bug fix version like the ones mentioned above.)
> > > > >
> > > > > Thanks,
> > > > > Sankalp
> > > > >
> > > >
> > >
> >
>

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Ismael Juma]

I think there's some confusion here. You can upgrade to AK 2.5.0 and
completely ignore ZK and TLS. It's completely optional.

Ismael

On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:

> Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating
> from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it would
> be nice to have a bug-free version of 2.3 and 2.4.
>
> Is there a technical reason why we hesitate to get these versions out? Or
> is it that no one has got around to it?
>
> On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia 
> wrote:
>
> > Thanks Ismael for the response.
> >
> > For our clusters running 2.3.1 and 2.4.1, we saw some issues which had
> > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it
> > introduces some major changes like support for ZK encryption and a few
> > others, I was wondering if we should choose a smaller upgrade in such
> cases
> > as we don't really require the new features in 2.5 and above right now.
> >
> > -
> > Sankalp
> >
> > On Wed, 24 Jun 2020 at 14:23, Ismael Juma  wrote:
> >
> > > Hi Sankalp,
> > >
> > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 instead?
> > We
> > > are working on the 2.5.1 release, which would be the recommended
> release.
> > >
> > > Ismael
> > >
> > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia <
> > sankalpbhati...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I would like to know if there are any plans to release a 2.3.2 and
> > 2.4.2
> > > > versions for Apache Kafka in the near future. I see there are some
> > issues
> > > > marked as fixed in these two versions
> > > > (https://tinyurl.com/ycdpz5cb).
> > > > However, I could not find a branch/tag corresponding to these
> versions
> > in
> > > > the github repository (https://github.com/apache/kafka).
> > > >
> > > > Also, It would be great if someone can help me understand or share
> any
> > > > documentation around the release processes (specifically on when we
> > > decide
> > > > to release a new bug fix version like the ones mentioned above.)
> > > >
> > > > Thanks,
> > > > Sankalp
> > > >
> > >
> >
>

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Gokul Ramanan Subramanian]

Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating
from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it would
be nice to have a bug-free version of 2.3 and 2.4.

Is there a technical reason why we hesitate to get these versions out? Or
is it that no one has got around to it?

On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia 
wrote:

> Thanks Ismael for the response.
>
> For our clusters running 2.3.1 and 2.4.1, we saw some issues which had
> 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it
> introduces some major changes like support for ZK encryption and a few
> others, I was wondering if we should choose a smaller upgrade in such cases
> as we don't really require the new features in 2.5 and above right now.
>
> -
> Sankalp
>
> On Wed, 24 Jun 2020 at 14:23, Ismael Juma  wrote:
>
> > Hi Sankalp,
> >
> > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 instead?
> We
> > are working on the 2.5.1 release, which would be the recommended release.
> >
> > Ismael
> >
> > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia <
> sankalpbhati...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I would like to know if there are any plans to release a 2.3.2 and
> 2.4.2
> > > versions for Apache Kafka in the near future. I see there are some
> issues
> > > marked as fixed in these two versions
> > > (https://tinyurl.com/ycdpz5cb).
> > > However, I could not find a branch/tag corresponding to these versions
> in
> > > the github repository (https://github.com/apache/kafka).
> > >
> > > Also, It would be great if someone can help me understand or share any
> > > documentation around the release processes (specifically on when we
> > decide
> > > to release a new bug fix version like the ones mentioned above.)
> > >
> > > Thanks,
> > > Sankalp
> > >
> >
>

Re: [ANNOUNCE] New committer: Xi Hu

[Bill Bejeck]

Congratulations Xi!

-Bill

On Wed, Jun 24, 2020 at 12:45 PM Tom Bentley  wrote:

> Congratulations Xi!
>
> On Wed, Jun 24, 2020 at 5:34 PM Guozhang Wang  wrote:
>
> > The PMC for Apache Kafka has invited Xi Hu as a committer and we are
> > pleased to announce that he has accepted!
> >
> > Xi Hu has been actively contributing to Kafka since 2016, and is well
> > recognized especially for his non-code contributions: he maintains a tech
> > blog post evangelizing Kafka in the Chinese speaking community (
> > https://www.cnblogs.com/huxi2b/), and is one of the most active
> answering
> > member in Zhihu (Chinese Reddit / StackOverflow) Kafka topic. He has
> > presented in Kafka meetup events in the past and authored a
> > book deep-diving on Kafka architecture design and operations as well (
> > https://www.amazon.cn/dp/B07JH9G2FL). Code wise, he has contributed 75
> > patches so far.
> >
> >
> > Thanks for all the contributions Xi. Congratulations!
> >
> > -- Guozhang, on behalf of the Apache Kafka PMC
> >
>

[jira] [Created] (KAFKA-10197) Elect preferred leader when completing a partition reassignment

[Bob Barrett (Jira)]

Bob Barrett created KAFKA-10197:
---

 Summary: Elect preferred leader when completing a partition 
reassignment
 Key: KAFKA-10197
 URL: https://issues.apache.org/jira/browse/KAFKA-10197
 Project: Kafka
  Issue Type: Improvement
Reporter: Bob Barrett


Currently, when completing a partition reassignment, we elect a leader from the 
new replica assignment without requiring that the leader be the new preferred 
leader. Instead, we just choose any in-sync replica:
{code:java}
private def leaderForReassign(partition: TopicPartition,
  leaderAndIsr: LeaderAndIsr,
  controllerContext: ControllerContext): 
ElectionResult = { 
  val targetReplicas = 
controllerContext.partitionFullReplicaAssignment(partition).targetReplicaAssignment.replicas
  val liveReplicas = targetReplicas.filter(replica => 
controllerContext.isReplicaOnline(replica, partition))
  val isr = leaderAndIsr.isr
  val leaderOpt = 
PartitionLeaderElectionAlgorithms.reassignPartitionLeaderElection(targetReplicas,
 isr, liveReplicas.toSet)
  val newLeaderAndIsrOpt = leaderOpt.map(leader => 
leaderAndIsr.newLeader(leader, isUnclean = false))
  ElectionResult(partition, newLeaderAndIsrOpt, targetReplicas)
}
{code}
If auto preferred leader election is enabled, the preferred leader will 
eventually be elected. However, it would make sense to choose the preferred 
leader after the reassignment without waiting for the automatic trigger.

{{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New committer: Xi Hu

[Tom Bentley]

Congratulations Xi!

On Wed, Jun 24, 2020 at 5:34 PM Guozhang Wang  wrote:

> The PMC for Apache Kafka has invited Xi Hu as a committer and we are
> pleased to announce that he has accepted!
>
> Xi Hu has been actively contributing to Kafka since 2016, and is well
> recognized especially for his non-code contributions: he maintains a tech
> blog post evangelizing Kafka in the Chinese speaking community (
> https://www.cnblogs.com/huxi2b/), and is one of the most active answering
> member in Zhihu (Chinese Reddit / StackOverflow) Kafka topic. He has
> presented in Kafka meetup events in the past and authored a
> book deep-diving on Kafka architecture design and operations as well (
> https://www.amazon.cn/dp/B07JH9G2FL). Code wise, he has contributed 75
> patches so far.
>
>
> Thanks for all the contributions Xi. Congratulations!
>
> -- Guozhang, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New committer: Boyang Chen

[Raymond Ng]

Congrats Boyang! Look forward to more awesome contributions from you in the
future.

Regards,
Ray

On Wed, Jun 24, 2020 at 6:07 AM Ismael Juma  wrote:

> Congratulations Boyang!
>
> Ismael
>
> On Mon, Jun 22, 2020 at 4:26 PM Guozhang Wang  wrote:
>
> > The PMC for Apache Kafka has invited Boyang Chen as a committer and we
> are
> > pleased to announce that he has accepted!
> >
> > Boyang has been active in the Kafka community more than two years ago.
> > Since then he has presented his experience operating with Kafka Streams
> at
> > Pinterest as well as several feature development including rebalance
> > improvements (KIP-345) and exactly-once scalability improvements
> (KIP-447)
> > in various Kafka Summit and Kafka Meetups. More recently he's also been
> > participating in Kafka broker development including post-Zookeeper
> > controller design (KIP-500). Besides all the code contributions, Boyang
> has
> > also helped reviewing even more PRs and KIPs than his own.
> >
> > Thanks for all the contributions Boyang! And look forward to more
> > collaborations with you on Apache Kafka.
> >
> >
> > -- Guozhang, on behalf of the Apache Kafka PMC
> >
>

[ANNOUNCE] New committer: Xi Hu

[Guozhang Wang]

The PMC for Apache Kafka has invited Xi Hu as a committer and we are
pleased to announce that he has accepted!

Xi Hu has been actively contributing to Kafka since 2016, and is well
recognized especially for his non-code contributions: he maintains a tech
blog post evangelizing Kafka in the Chinese speaking community (
https://www.cnblogs.com/huxi2b/), and is one of the most active answering
member in Zhihu (Chinese Reddit / StackOverflow) Kafka topic. He has
presented in Kafka meetup events in the past and authored a
book deep-diving on Kafka architecture design and operations as well (
https://www.amazon.cn/dp/B07JH9G2FL). Code wise, he has contributed 75
patches so far.


Thanks for all the contributions Xi. Congratulations!

-- Guozhang, on behalf of the Apache Kafka PMC

Re: [DISCUSS] KIP-308: GetOffsetShell: new KafkaConsumer API, support for multiple topics, minimize the number of requests to server

[Dániel Urbán]

Hi,

I see that this KIP turned somewhat inactive - I'd like to pick it up and
work on it if it is okay.
Part of the work is done, as switching to the Consumer API is already in
trunk, but some functionality is still missing.

I've seen the current PR and the discussion so far, only have a few things
to add:
- I like the idea of the topic-partition argument, it would be useful to
filter down to specific partitions.
- Instead of a topic list arg, a pattern would be more powerful, and also
fit better with the other tools (e.g. how the kafka-topics tool works).

Regards,
Daniel

Re: contributor request

[Bill Bejeck]

Hi Daniel,

You're all set as a contributor in Jira and can create/edit KIPs as well.

Thanks for your interest in Apache Kafka.

-Bill

On Wed, Jun 24, 2020 at 6:16 AM Dániel Urbán 
wrote:

> Hi,
> I'd like to work on some KIPs. Can you please add me to the contributors?
> My username is durban on both JIRA and Confluence.
> Thanks in advance,
> Daniel
>

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Sankalp Bhatia]

Thanks Ismael for the response.

For our clusters running 2.3.1 and 2.4.1, we saw some issues which had
2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it
introduces some major changes like support for ZK encryption and a few
others, I was wondering if we should choose a smaller upgrade in such cases
as we don't really require the new features in 2.5 and above right now.

-
Sankalp

On Wed, 24 Jun 2020 at 14:23, Ismael Juma  wrote:

> Hi Sankalp,
>
> Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 instead? We
> are working on the 2.5.1 release, which would be the recommended release.
>
> Ismael
>
> On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia 
> wrote:
>
> > Hi All,
> >
> > I would like to know if there are any plans to release a 2.3.2 and 2.4.2
> > versions for Apache Kafka in the near future. I see there are some issues
> > marked as fixed in these two versions
> > (https://tinyurl.com/ycdpz5cb).
> > However, I could not find a branch/tag corresponding to these versions in
> > the github repository (https://github.com/apache/kafka).
> >
> > Also, It would be great if someone can help me understand or share any
> > documentation around the release processes (specifically on when we
> decide
> > to release a new bug fix version like the ones mentioned above.)
> >
> > Thanks,
> > Sankalp
> >
>

Build failed in Jenkins: kafka-2.4-jdk8 #228

[Apache Jenkins Server]

See 


Changes:

[matthias] HOTFIX: use equals() to compare HostInfo in StreamsMetadataState 
(#8919)


--
[...truncated 7.19 MB...]
kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl PASSED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslScramSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.UserQuotaTest > testProducerConsumerOverrideLowerQuota STARTED

kafka.api.UserQuotaTest > testProducerConsumerOverrideLowerQuota PASSED

kafka.api.UserQuotaTest > testProducerConsumerOverrideUnthrottled STARTED

kafka.api.UserQuotaTest > testProducerConsumerOverrideUnthrottled PASSED

kafka.api.UserQuotaTest > testThrottledProducerConsumer STARTED

kafka.api.UserQuotaTest > testThrottledProducerConsumer PASSED

kafka.api.UserQuotaTest > testQuotaOverrideDelete STARTED

kafka.api.UserQuotaTest > testQuotaOverrideDelete PASSED

kafka.api.UserQuotaTest > testThrottledRequest STARTED

kafka.api.UserQuotaTest > testThrottledRequest PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaAssign PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > testNoProduceWithDescribeAcl 
PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl PASSED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslGssapiSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.test.ProducerCompressionTest > testCompression[0 compressionType = 
none] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[0 compressionType = 
none] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[1 compressionType = 
gzip] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[1 compressionType = 
gzip] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[2 compressionType = 
snappy] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[2 compressionType = 
snappy] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[3 compressionType = 
lz4] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[3 compressionType = 
lz4] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[4 compressionType = 
zstd] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[4 compressionType = 
zstd] PASSED

kafka.api.PlaintextEndToEndAuthorizationTest > testListenerName STARTED

kafka.api.PlaintextEndToEndAuthorizationTest > testListenerName PASSED

kafka.api.PlaintextEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.PlaintextEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubsc

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

[Ismael Juma]

Error code names are safe to rename at the moment as they are in an
internal package. The exception class is in a public package though. I was
thinking that PolicyViolationException could be a subclass of the more
generic exception. This approach would mean that older clients would
understand the error code, etc. I didn't think through all the details, but
worth considering.

With regards to the version, OK. I expect trunk to do better than what was
described there on both replication overhead and topic creation/deletion
performance.

Ismael

On Tue, Jun 23, 2020 at 11:10 PM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:

> Ismael,
>
> I am open to using any error code and am not attached to one TBH. Colin had
> suggested creating a new resource code called RESOURCE_LIMIT_EXCEEDED. I am
> happy to reuse the error code corresponding to PolicyViolation. Is it safe
> to rename errors and corresponding exception names? If so, I'd prefer
> reusing the existing code as well.
>
> For performance testing results that I added to this KIP, I used Kafka
> 2.3.1, which was very close to trunk at the time I tested. We have seen
> similar issues with Kafka 2.4.1. Please note that for the tests done in the
> KIP, especially the Produce performance tests, we probably could have
> gotten higher performance, but the focus was on comparing performance
> across a different number of partitions, for a given configuration, rather
> than trying to find out the best performance possible for the "right"
> number of partitions.
>
> Thanks.
>
>
>
>
>
>
>
>
>
> On Wed, Jun 24, 2020 at 4:07 AM Ismael Juma  wrote:
>
> > Thanks for the KIP. A couple of questions:
> >
> > 1. Have we considered reusing the existing PolicyViolation error code and
> > renaming it? This would make it simpler to handle on the client.
> >
> > 2. What version was used for the perf section? I think master should do
> > better than what's described there.
> >
> > Ismael
> >
> > On Wed, Apr 1, 2020, 8:28 AM Gokul Ramanan Subramanian <
> > gokul24...@gmail.com>
> > wrote:
> >
> > > Hi.
> > >
> > > I have opened KIP-578, intended to provide a mechanism to limit the
> > number
> > > of partitions in a Kafka cluster. Kindly provide feedback on the KIP
> > which
> > > you can find at
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions
> > >
> > > I want to specially thank Stanislav Kozlovski who helped in formulating
> > > some aspects of the KIP.
> > >
> > > Many thanks,
> > >
> > > Gokul.
> > >
> >
>

Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Ismael Juma]

Hi Sankalp,

Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 instead? We
are working on the 2.5.1 release, which would be the recommended release.

Ismael

On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia 
wrote:

> Hi All,
>
> I would like to know if there are any plans to release a 2.3.2 and 2.4.2
> versions for Apache Kafka in the near future. I see there are some issues
> marked as fixed in these two versions
> (https://tinyurl.com/ycdpz5cb).
> However, I could not find a branch/tag corresponding to these versions in
> the github repository (https://github.com/apache/kafka).
>
> Also, It would be great if someone can help me understand or share any
> documentation around the release processes (specifically on when we decide
> to release a new bug fix version like the ones mentioned above.)
>
> Thanks,
> Sankalp
>

Question around release plans for Apache Kafka 2.3.2 and 2.4.2

[Sankalp Bhatia]

Hi All,

I would like to know if there are any plans to release a 2.3.2 and 2.4.2
versions for Apache Kafka in the near future. I see there are some issues
marked as fixed in these two versions
(https://tinyurl.com/ycdpz5cb).
However, I could not find a branch/tag corresponding to these versions in
the github repository (https://github.com/apache/kafka).

Also, It would be great if someone can help me understand or share any
documentation around the release processes (specifically on when we decide
to release a new bug fix version like the ones mentioned above.)

Thanks,
Sankalp

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Ismael Juma]

Hi Boyang,

"One thing I'm not fully convinced of is why we need to deprecate the
server side auto topic creation logic,"

The point is to eventually remove auto topic creation logic from
MetadataRequest. It makes no sense for MetadataRequest to cause topics to
be created. It's an unintuitive hack.

Ismael

On Tue, Jun 23, 2020 at 10:12 PM Boyang Chen 
wrote:

> Hey Justin and Jiamei,
>
> I read the KIP and skimmed over the discussion. One thing I'm not fully
> convinced of is why we need to deprecate the server side auto topic
> creation logic, which seems orthogonal towards whether a client wants to
> create the topic or not. Won't it be more natural to assume that only when
> both server and client agree on turning on the switch, will a topic get
> created?
>
> Some clarifications would also be appreciated:
>
> 1. Could we include a link to KIP-464 and explain its relation to KIP-487?
> It's very hard to read through the proposal when readers only have a
> reference number to some KIP that is not briefed.
>
> 2. The KIP suggests, " In the producer, auto-creation of a topic will occur
> through a specific request rather than through a side effect of requesting
> metadata." Could we be specific such as whether we are going to introduce a
> separate RPC, or just send another CreateTopicRequest?
>
> Boyang
>
> On Wed, Jun 17, 2020 at 8:51 AM jiamei xie  wrote:
>
> > Hi all
> > For
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> > ,  It has not been updated for a long time. And I made some update, which
> > has been pushed to https://github.com/apache/kafka/pull/8831
> >
> > MetadataRequest has method Builder(List topics, boolean
> > allowAutoTopicCreation) by which we can set whether to enable
> > allowAutoTopicCreation from producer.
> > By default, allowAutoTopicCreation on Producer is true. And only if when
> > the allowAutoTopicCreation of Broker and Producer are true, the topic can
> > be auto-created.
> >
> > Besides, the test cases are changed:
> > There are 4 cases for brokerAutoTopicCreationEnable and
> > producerAutoCreateTopicsPolicy, Check if the topic is created under these
> > four cases.
> >  If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy
> > are true:  assertTrue(topicCreated)
> >  else : intercept[ExecutionException]
> >
> > Looking forward to your feedback and comments. Thanks.
> >
> > Best wishes
> > Jiamei Xie
> >
> > On 2019/08/12 15:50:22, Harsha Chintalapani  wrote:
> > > On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > A few points:
> > > >
> > > > 1. I think the way backwards compatibility is being used here is not
> > > > correct. Any functionality that is only enabled if set via a config
> is
> > > > backwards compatible. People may disagree with the functionality or
> the
> > > > config, but it's not a backwards compatibility issue.
> > > >
> > >
> > > We are talking about both broker and producer as a  single entity and
> run
> > > by the same team/users. Allowing newer producer to create topics on a
> > older
> > > broker when auto.create.topics.enable set to false, breaks  server side
> > > contract that this config offered from the beginning.  IMO, it clearly
> > > isn't backward compatible. User who set auto.create.topic.enable on
> > broker
> > > will not be the same who will turn it on producer side .
> > >
> > >
> > > > 2. It's an interesting question if auto topic creation via the
> producer
> > > > should be a server driven choice or not. I can see the desire to
> have a
> > > > server-driven default, but it seems like this is often application
> > > > specific. Because the functionality is trivially available via
> > AdminClient
> > > > (released 2 years ago), it's not quite possible to control what
> > > > applications do without the use of ACLs or policies today.
> > > >
> > > >
> > > >
> > > Producers & consumers are the majority of the clients in Kafka
> ecosystem.
> > > Just because AdminClient shipped a while back that doesn't mean all
> users
> > > adopting to it. To this day lot more users are aware of Producer &
> > Consumer
> > > APIs and running them in production compare to AdminClient.
> > >
> > >
> > > > 3. Changing the create topics request in this way is highly
> > unintuitive in
> > > > my opinion and it relies on every client to pass the new field. For
> > > > example, if librdkafka added auto create functionality by relying on
> > their
> > > > AdminClient, it would behave differently than what is proposed here.
> > > > Forcing every client to implement this change when calling auto
> create
> > from
> > > > the producer specifically seems odd
> > > >
> > >
> > > I am not sure why its unintuitive , protocols change. We add or upgrade
> > the
> > > existing protocols all the time.
> > >
> > >
> > > Thanks,
> > > Harsha
> > >
> > > .
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Aug 8, 2019 at 11:51 AM Jun Rao

Re: [ANNOUNCE] New committer: Boyang Chen

[Ismael Juma]

Congratulations Boyang!

Ismael

On Mon, Jun 22, 2020 at 4:26 PM Guozhang Wang  wrote:

> The PMC for Apache Kafka has invited Boyang Chen as a committer and we are
> pleased to announce that he has accepted!
>
> Boyang has been active in the Kafka community more than two years ago.
> Since then he has presented his experience operating with Kafka Streams at
> Pinterest as well as several feature development including rebalance
> improvements (KIP-345) and exactly-once scalability improvements (KIP-447)
> in various Kafka Summit and Kafka Meetups. More recently he's also been
> participating in Kafka broker development including post-Zookeeper
> controller design (KIP-500). Besides all the code contributions, Boyang has
> also helped reviewing even more PRs and KIPs than his own.
>
> Thanks for all the contributions Boyang! And look forward to more
> collaborations with you on Apache Kafka.
>
>
> -- Guozhang, on behalf of the Apache Kafka PMC
>

Re: [ANNOUNCE] New committer: Boyang Chen

[Rajini Sivaram]

Congratulations, Boyang!

Regards,

Rajini

On Wed, Jun 24, 2020 at 2:26 AM Adam Bellemare 
wrote:

> Just adding my congratulations, Boyang! Thank you for all your
> contributions and effort!
>
> On Tue, Jun 23, 2020 at 9:14 PM Kowshik Prakasam 
> wrote:
>
> > Congrats, Boyang! :)
> >
> >
> > Cheers,
> > Kowshik
> >
> > On Tue, Jun 23, 2020 at 8:43 AM Aparnesh Gaurav <
> aparneshgau...@gmail.com>
> > wrote:
> >
> > > Congrats Boyang.
> > >
> > > On Tue, 23 Jun, 2020, 9:07 PM Vahid Hashemian, <
> > vahid.hashem...@gmail.com>
> > > wrote:
> > >
> > > > Congrats Boyang!
> > > >
> > > > --Vahid
> > > >
> > > > On Tue, Jun 23, 2020 at 6:41 AM Wang (Leonard) Ge 
> > > > wrote:
> > > >
> > > > > Congrats Boyang! This is a great achievement.
> > > > >
> > > > > On Tue, Jun 23, 2020 at 10:33 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congrats Boyang! Well deserved
> > > > > >
> > > > > > On Tue, Jun 23, 2020 at 8:20 AM Tom Bentley  >
> > > > wrote:
> > > > > > >
> > > > > > > Congratulations Boyang!
> > > > > > >
> > > > > > > On Tue, Jun 23, 2020 at 8:11 AM Bruno Cadonna <
> > br...@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Congrats, Boyang!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Bruno
> > > > > > > >
> > > > > > > > On Tue, Jun 23, 2020 at 7:50 AM Konstantine Karantasis
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Congrats, Boyang!
> > > > > > > > >
> > > > > > > > > -Konstantine
> > > > > > > > >
> > > > > > > > > On Mon, Jun 22, 2020 at 9:19 PM Navinder Brar
> > > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > > Many Congratulations Boyang. Very well deserved.
> > > > > > > > > >
> > > > > > > > > > Regards,Navinder
> > > > > > > > > >
> > > > > > > > > > On Tuesday, 23 June, 2020, 07:21:23 am IST, Matt
> Wang <
> > > > > > > > wang...@163.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >  Congratulations, Boyang!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Matt Wang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 06/23/2020 07:59，Boyang Chen<
> reluctanthero...@gmail.com
> > >
> > > > > wrote：
> > > > > > > > > > Thanks a lot everyone, I really appreciate the
> recognition,
> > > and
> > > > > > hope to
> > > > > > > > > > make more solid contributions to the community in the
> > future!
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 22, 2020 at 4:50 PM Matthias J. Sax <
> > > > > mj...@apache.org>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Congrats! Well deserved!
> > > > > > > > > >
> > > > > > > > > > -Matthias
> > > > > > > > > >
> > > > > > > > > > On 6/22/20 4:38 PM, Bill Bejeck wrote:
> > > > > > > > > > Congratulations Boyang! Well deserved.
> > > > > > > > > >
> > > > > > > > > > -Bill
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 22, 2020 at 7:35 PM Colin McCabe <
> > > > cmcc...@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Congratulations, Boyang!
> > > > > > > > > >
> > > > > > > > > > cheers,
> > > > > > > > > > Colin
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 22, 2020, at 16:26, Guozhang Wang wrote:
> > > > > > > > > > The PMC for Apache Kafka has invited Boyang Chen as a
> > > committer
> > > > > > and we
> > > > > > > > > > are
> > > > > > > > > > pleased to announce that he has accepted!
> > > > > > > > > >
> > > > > > > > > > Boyang has been active in the Kafka community more than
> two
> > > > years
> > > > > > ago.
> > > > > > > > > > Since then he has presented his experience operating with
> > > Kafka
> > > > > > Streams
> > > > > > > > > > at
> > > > > > > > > > Pinterest as well as several feature development
> including
> > > > > > rebalance
> > > > > > > > > > improvements (KIP-345) and exactly-once scalability
> > > > improvements
> > > > > > > > > > (KIP-447)
> > > > > > > > > > in various Kafka Summit and Kafka Meetups. More recently
> > he's
> > > > > also
> > > > > > been
> > > > > > > > > > participating in Kafka broker development including
> > > > > post-Zookeeper
> > > > > > > > > > controller design (KIP-500). Besides all the code
> > > > contributions,
> > > > > > Boyang
> > > > > > > > > > has
> > > > > > > > > > also helped reviewing even more PRs and KIPs than his
> own.
> > > > > > > > > >
> > > > > > > > > > Thanks for all the contributions Boyang! And look forward
> > to
> > > > more
> > > > > > > > > > collaborations with you on Apache Kafka.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > -- Guozhang, on behalf of the Apache Kafka PMC
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Leonard Ge
> > > > > Software Engineer Intern - Confluent
> > > > >
> > > >
> > > >
> > > > --
> > > >
> >

回复: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[Jiamei Xie]

Hi Boyang, Justin,

" 1. Could we include a link to KIP-464 and explain its relation to KIP-487?
It's very hard to read through the proposal when readers only have a
reference number to some KIP that is not briefed. "
For KIP-464  
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722,  
Its motivation is to make default `num.partitions` and 
`default.replication.factor` available to AdminClient APIs.

For KIP-487  
https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer,
  I think its motivation is to make auto-create-topic configurable from 
producer side.


"2. The KIP suggests, " In the producer, auto-creation of a topic will occur
through a specific request rather than through a side effect of requesting
metadata." Could we be specific such as whether we are going to introduce a
separate RPC, or just send another CreateTopicRequest?"

I think there is no need to introduce a separate RPC.

MetadataRequest already has field allowAutoTopicCreation. For current 
ProducerMetadata, it sets allowAutoTopicCreation to true as 
https://github.com/apache/kafka/blob/448e7d7f0f46db1eae14d4fe7a1d25b7af894b09/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerMetadata.java#L59.

When broker received a metadata request and the topic doesn't exist, it will 
decide whether to create topic according to "allowAutoTopicCreation && 
config.autoCreateTopicsEnable" 
https://github.com/apache/kafka/blob/9a4f00f78bf37041006ae8b6432d194f603ac6cc/core/src/main/scala/kafka/server/KafkaApis.scala#L1107

So the way I implemented it is different from Justine's.
Justine's PR: https://github.com/apache/kafka/pull/7075. It was implemented it 
by CreateTopicsRequest

Jiamei's PR : https://github.com/apache/kafka/pull/8831. I added field " 
allowAutoTopicCreation" to ProducerMetadata, pass it to MetadataRequest. 
builder.

As boyang said, " Won't it be more natural to assume that only when
both server and client agree on turning on the switch, will a topic get
created?"
It was the same as what I thought.

-邮件原件-
发件人: Boyang Chen 
发送时间: 2020年6月24日 13:12
收件人: dev 
主题: Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

Hey Justin and Jiamei,

I read the KIP and skimmed over the discussion. One thing I'm not fully
convinced of is why we need to deprecate the server side auto topic
creation logic, which seems orthogonal towards whether a client wants to
create the topic or not. Won't it be more natural to assume that only when
both server and client agree on turning on the switch, will a topic get
created?

Some clarifications would also be appreciated:

1. Could we include a link to KIP-464 and explain its relation to KIP-487?
It's very hard to read through the proposal when readers only have a
reference number to some KIP that is not briefed.

2. The KIP suggests, " In the producer, auto-creation of a topic will occur
through a specific request rather than through a side effect of requesting
metadata." Could we be specific such as whether we are going to introduce a
separate RPC, or just send another CreateTopicRequest?

Boyang

On Wed, Jun 17, 2020 at 8:51 AM jiamei xie  wrote:

> Hi all
> For
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer
> ,  It has not been updated for a long time. And I made some update, which
> has been pushed to https://github.com/apache/kafka/pull/8831
>
> MetadataRequest has method Builder(List topics, boolean
> allowAutoTopicCreation) by which we can set whether to enable
> allowAutoTopicCreation from producer.
> By default, allowAutoTopicCreation on Producer is true. And only if when
> the allowAutoTopicCreation of Broker and Producer are true, the topic can
> be auto-created.
>
> Besides, the test cases are changed:
> There are 4 cases for brokerAutoTopicCreationEnable and
> producerAutoCreateTopicsPolicy, Check if the topic is created under these
> four cases.
>  If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy
> are true:  assertTrue(topicCreated)
>  else : intercept[ExecutionException]
>
> Looking forward to your feedback and comments. Thanks.
>
> Best wishes
> Jiamei Xie
>
> On 2019/08/12 15:50:22, Harsha Chintalapani  wrote:
> > On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > A few points:
> > >
> > > 1. I think the way backwards compatibility is being used here is not
> > > correct. Any functionality that is only enabled if set via a config is
> > > backwards compatible. People may disagree with the functionality or the
> > > config, but it's not a backwards compatibility issue.
> > >
> >
> > We are talking about both broker and producer as a  single entity and run
> > by the same team/users. Allowing newer producer to create topics on a
> older
> > broker when auto.create.topics.enable set to false, breaks  server side
> > contract that this config offered from the 

contributor request

[Dániel Urbán]

Hi,
I'd like to work on some KIPs. Can you please add me to the contributors?
My username is durban on both JIRA and Confluence.
Thanks in advance,
Daniel

Jenkins build is back to normal : kafka-trunk-jdk8 #4665

[Apache Jenkins Server]

See