[DISCUSS] KIP-730: Producer ID generation in KRaft mode

Hello everyone, I'd like to start the discussion for KIP-730 https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode This KIP proposes a new RPC for generating blocks of IDs for transactional and idempotent producers. Cheers, David Arthur

Permission to Assign Jiras

Hello, May I have permission to assign Jiras to myself? I just got this done https://issues.apache.org/jira/browse/KAFKA-12265 but don't have access to reassign it. Best, Ryan Dielhenn

[jira] [Resolved] (KAFKA-12602) The LICENSE and NOTICE files don't list everything they should

[ https://issues.apache.org/jira/browse/KAFKA-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-12602. -- Resolution: Fixed > The LICENSE and NOTICE files don't list everything they should >

[jira] [Created] (KAFKA-12625) Fix the NOTICE file

John Roesler created KAFKA-12625: Summary: Fix the NOTICE file Key: KAFKA-12625 URL: https://issues.apache.org/jira/browse/KAFKA-12625 Project: Kafka Issue Type: Task Reporter:

[jira] [Created] (KAFKA-12624) Fix LICENSE in 2.6

John Roesler created KAFKA-12624: Summary: Fix LICENSE in 2.6 Key: KAFKA-12624 URL: https://issues.apache.org/jira/browse/KAFKA-12624 Project: Kafka Issue Type: Sub-task

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #10

See

Re: [VOTE] 2.8.0 RC0

Hello again, all, I am closing this vote in favor of the 2.8.0 RC1 thread. Thank you, John On Tue, 2021-03-30 at 17:11 -0500, John Roesler wrote: > Hello again, all, > > I just wanted to mention that I am aware of Justin's > concerns in the 2.6.2 thread: >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #4

See

[jira] [Created] (KAFKA-12620) Producer IDs generated by the controller

David Arthur created KAFKA-12620: Summary: Producer IDs generated by the controller Key: KAFKA-12620 URL: https://issues.apache.org/jira/browse/KAFKA-12620 Project: Kafka Issue Type: New

[jira] [Created] (KAFKA-12622) Automate LICENCSE file validation

John Roesler created KAFKA-12622: Summary: Automate LICENCSE file validation Key: KAFKA-12622 URL: https://issues.apache.org/jira/browse/KAFKA-12622 Project: Kafka Issue Type: Task

[jira] [Created] (KAFKA-12623) Fix LICENSE in 2.7

John Roesler created KAFKA-12623: Summary: Fix LICENSE in 2.7 Key: KAFKA-12623 URL: https://issues.apache.org/jira/browse/KAFKA-12623 Project: Kafka Issue Type: Sub-task

Subject: [VOTE] 2.8.0 RC1

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 2.8.0. This is a major release that includes many new features, including: * Early-access release of replacing Zookeeper with a self- managed quorum * Add Describe Cluster API * Support

[jira] [Created] (KAFKA-12621) Kafka setup with Zookeeper specifying an alternate znode as root fails

Jibitesh Prasad created KAFKA-12621: --- Summary: Kafka setup with Zookeeper specifying an alternate znode as root fails Key: KAFKA-12621 URL: https://issues.apache.org/jira/browse/KAFKA-12621

[GitHub] [kafka-site] cc13ny closed pull request #345: Fix the formatting of example RocksDBConfigSetter

cc13ny closed pull request #345: URL: https://github.com/apache/kafka-site/pull/345 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [kafka-site] ableegoldman commented on pull request #345: Fix the formatting of example RocksDBConfigSetter

ableegoldman commented on pull request #345: URL: https://github.com/apache/kafka-site/pull/345#issuecomment-813896409 > Also how can I know if it will be picked? I'll let you know :) But you can follow. along with the 2.8 release progress by subscribing to the dev mailing

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #7

See

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #673

See Changes: [github] MINOR: update GroupMetadataManager#getMagic docs (#10442) -- [...truncated 3.73 MB...] KafkaZkClientTest > testPropagateLogDir()

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #8

See Changes: -- [...truncated 404224 lines...] [2021-04-06T09:59:48.276Z] [2021-04-06T09:59:48.276Z] SaslPlainSslEndToEndAuthorizationTest >

Re: [DISCUSS] KIP-729 Custom validation of records on the broker prior to log append

Hi Tom, Makes sense. Thanks for the explanation. I get what Colin had meant earlier. Would a different signature for the interface work? Example below, but please feel free to suggest alternatives if there are any possibilities of such. If needed, then deprecating this and introducing a new

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

Thanks for the feedback, Ismael. Renaming the RPC and using start+len instead of start+end sounds fine. And yes, the controller will allocate the IDs in ZK mode for the bridge release. I'll update the KIP to reflect these points. Thanks! On Tue, Apr 6, 2021 at 7:30 PM Ismael Juma wrote: >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #11

See

[jira] [Resolved] (KAFKA-10769) Remove JoinGroupRequest#containsValidPattern as it is duplicate to Topic#containsValidPattern

[ https://issues.apache.org/jira/browse/KAFKA-10769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-10769. Fix Version/s: 3.0.0 Resolution: Fixed > Remove

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

Sorry, one more question: the allocation of ids will be done by the controller even in ZK mode, right? Ismael On Tue, Apr 6, 2021 at 4:26 PM Ismael Juma wrote: > One additional comment: if you return the number of ids instead of the end > range, you can use an int32. > > Ismael > > On Tue, Apr

Re: Permission to Assign Jiras

Hi Ryan. I found your first and last name on jira and added you to the list of contributors for the Apache Kafka project. You should be able to assign tickets to yourself now. Welcome! Konstantine On Tue, Apr 6, 2021 at 2:20 PM Ryan Dielhenn wrote: > Hello, > > May I have permission to assign

Re: [DISCUSS] KIP-633: Drop 24 hour default of grace period in Streams

Thanks for the KIP Sophie. It make total sense to get rid of default grace period of 24h. Some questions/comments: (1) Is there any particular reason why we want to remove `grace(Duration)` method? (2) About `SlidingWindows#withTimeDifferenceAndGrace` -- personally I think it's worth to clean

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

Just catching up here. I agree that we have two issue, and the first (align subscription store to main store) can be done as a bug-fix. For the KIP (that addressed the second), I tend to agree that reusing `Materialized` might be better as it would keep the API surface area smaller. -Matthias

[jira] [Created] (KAFKA-12626) RaftClusterTest and ClusterTestExtensionTest failures

Justine Olshan created KAFKA-12626: -- Summary: RaftClusterTest and ClusterTestExtensionTest failures Key: KAFKA-12626 URL: https://issues.apache.org/jira/browse/KAFKA-12626 Project: Kafka

[jira] [Resolved] (KAFKA-12384) Flaky Test ListOffsetsRequestTest.testResponseIncludesLeaderEpoch

[ https://issues.apache.org/jira/browse/KAFKA-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-12384. Fix Version/s: 3.0.0 Resolution: Fixed > Flaky Test

Re: Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

> > 3. Did we consider the possibility of introducing a new interface which > > extended both CompletionStage and Future? That would make it easier for > > people to update their existing code, since the handling of the result (in > > the case they weren't using the Callback version) would be

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

Thanks for the KIP, David. Any reason not to rename AllocateProducerIdBlockRequest to AllocateProducerIdsRequest? Ismael On Tue, Apr 6, 2021 at 3:51 PM David Arthur wrote: > Hello everyone, > > I'd like to start the discussion for KIP-730 > > >

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

One additional comment: if you return the number of ids instead of the end range, you can use an int32. Ismael On Tue, Apr 6, 2021 at 4:25 PM Ismael Juma wrote: > Thanks for the KIP, David. Any reason not to rename > AllocateProducerIdBlockRequest to AllocateProducerIdsRequest? > > Ismael > >

Re: [DISCUSS] KIP-729 Custom validation of records on the broker prior to log append

Hi Soumyajit, The difficult thing is deciding which fields to share and how to share them. Key and value are probably the minimum we need to make this useful. If we do choose to go with byte buffer, it is not necessary to also pass the size, since ByteBuffer maintains that internally.

[jira] [Resolved] (KAFKA-5146) Kafka Streams: remove compile dependency on connect-json

[ https://issues.apache.org/jira/browse/KAFKA-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-5146. Fix Version/s: 3.0.0 Resolution: Fixed > Kafka Streams: remove compile dependency

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

Great, thanks. Instead of calling it "bridge release", can we say 3.0? Ismael On Tue, Apr 6, 2021 at 7:48 PM David Arthur wrote: > Thanks for the feedback, Ismael. Renaming the RPC and using start+len > instead of start+end sounds fine. > > And yes, the controller will allocate the IDs in ZK

[jira] [Created] (KAFKA-12627) Unify MemoryPool and BufferSupplier

Chia-Ping Tsai created KAFKA-12627: -- Summary: Unify MemoryPool and BufferSupplier Key: KAFKA-12627 URL: https://issues.apache.org/jira/browse/KAFKA-12627 Project: Kafka Issue Type: Task

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #672

See Changes: [github] MINOR: Jenkinsfile's `post` needs `agent` to be set (#10479) -- [...truncated 3.70 MB...]

[GitHub] [kafka-site] cc13ny commented on pull request #345: Fix the formatting of example RocksDBConfigSetter

cc13ny commented on pull request #345: URL: https://github.com/apache/kafka-site/pull/345#issuecomment-813862721 Thanks. I created https://github.com/apache/kafka/pull/10486. I will close it now. But will reopen if it's not picked before the 2.8 release. Also how can I know if it will be

Re: contributor permission

Hi Amuthan, That's now done. Thanks for your interest in Apache Kafka. Kind regards, Tom On Mon, Apr 5, 2021 at 7:52 PM Amuthan wrote: > Hi > > I would like to contribute to > https://issues.apache.org/jira/browse/KAFKA-12559, Could you please give > me > contributor permission, here is my

Re: [DISCUSS] KIP-729 Custom validation of records on the broker prior to log append

Hi Soumyajit, Although that class does indeed have public access at the Java level, it does so only because it needs to be used by internal Kafka code which lives in other packages (there isn't any more restrictive access modifier which would work). What the project considers public Java API is

Re: Re: [DISCUSS] KIP-706: Add method "Producer#produce" to return CompletionStage instead of Future

hi Ismael, In order to minimize the pain (and code changes), I remove all deprecations. The main purpose of this KIP is to introduce new API so I plan to keep using ProduceRecord in internal process. The new interfaces (SendRecord and SendTarget) are used by new API only and they are converted

Re: New Jenkins job for master and release branches

On Sun, Apr 4, 2021 at 2:22 PM Ismael Juma wrote: > There is currently an open PR to > extend the Jenkinsfile with functionality desired for branch builds. Once > that is merged and has been shown to work correctly, I will delete legacy > Jenkins jobs

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

This makes sense to me +1! Walker On Tue, Apr 6, 2021 at 11:08 AM Guozhang Wang wrote: > +1. Thanks! > > On Tue, Apr 6, 2021 at 7:00 AM Leah Thomas > wrote: > > > Thanks for picking this up, Sophie. +1 from me, non-binding. > > > > Leah > > > > On Mon, Apr 5, 2021 at 9:42 PM John Roesler

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

Makes sense to me! +1 Apologies for being brief. This email was sent from my mobile phone. > On 6 Apr 2021, at 18:45, Walker Carlson wrote: > > This makes sense to me +1! > > Walker > >> On Tue, Apr 6, 2021 at 11:08 AM Guozhang Wang wrote: >> >> +1. Thanks! >> >> On Tue, Apr 6, 2021 at

Re: [VOTE] KIP-725: Streamlining configurations for TimeWindowedDeserializer.

Hi Sagar, +1 non-binding. Thanks again for doing this. Leah On Mon, Apr 5, 2021 at 9:40 PM John Roesler wrote: > Thanks, Sagar! > > I’m +1 (binding) > > -John > > On Mon, Apr 5, 2021, at 21:35, Sophie Blee-Goldman wrote: > > Thanks for the KIP! +1 (binding) from me > > > > Cheers, > > Sophie >

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

Thanks for picking this up, Sophie. +1 from me, non-binding. Leah On Mon, Apr 5, 2021 at 9:42 PM John Roesler wrote: > Thanks, Sophie, > > I’m +1 (binding) > > -John > > On Mon, Apr 5, 2021, at 21:34, Sophie Blee-Goldman wrote: > > Hey all, > > > > I'd like to start the voting on KIP-633, to

Jenkins build is unstable: Kafka » Kafka Branch Builder » 2.8 #3

See

help to review patch

hi kafka community: In ProducerPerformance, random payload always same. It has a great impact when use the compression.type option, here is my patch[1], please review when you get a chance, thanks. best, [1] https://github.com/apache/kafka/pull/10469

Re: [VOTE] KIP-707: The future of KafkaFuture

Hi, The vote passes with 4 binding +1s (Ismael, David Chia-Ping and Colin), and 1 non-binding +1 (Ryanne). Many thanks to those who commented and/or voted. Tom On Thu, Apr 1, 2021 at 8:21 PM Colin McCabe wrote: > +1 (binding). Thanks for the KIP. > > Colin > > > On Tue, Mar 30, 2021, at

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

Hi John / Guozhang, If I correctly understood John's message, he agrees on having the two scenarios (piggy-back and api extension). In my view, these two scenarios are separate tasks - the first one is a bug-fix and the second is an improvement on the current API. - bug-fix: On the current API,

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

Hi Marco, Just a quick clarification: I just reviewed the Materialized class. It looks like the only undesirable members are: 1. Retention 2. Key/Value serdes The underlying store type would be “KeyValueStore” , for which case the withRetention javadoc already says it’s ignored. Perhaps we

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

+1. Thanks! On Tue, Apr 6, 2021 at 7:00 AM Leah Thomas wrote: > Thanks for picking this up, Sophie. +1 from me, non-binding. > > Leah > > On Mon, Apr 5, 2021 at 9:42 PM John Roesler wrote: > > > Thanks, Sophie, > > > > I’m +1 (binding) > > > > -John > > > > On Mon, Apr 5, 2021, at 21:34,

Re: [VOTE] KIP-725: Streamlining configurations for TimeWindowedDeserializer.

+1. Thanks! On Tue, Apr 6, 2021 at 7:01 AM Leah Thomas wrote: > Hi Sagar, +1 non-binding. Thanks again for doing this. > > Leah > > On Mon, Apr 5, 2021 at 9:40 PM John Roesler wrote: > > > Thanks, Sagar! > > > > I’m +1 (binding) > > > > -John > > > > On Mon, Apr 5, 2021, at 21:35, Sophie

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #9

See