Re: [VOTE] 2.8.0 RC2
+1 (non-binding) - Ran releaseTarGzAll successfully with no failures. - Ran subset of tests. - Ran through quickstart on builds generated from the tag. - Ran a few internal apps targeting topics on a 5 node cluster. Thanks, Satish. On Sun, 18 Apr 2021 at 04:13, Colin McCabe wrote: > Hi John, > > +1 (binding) from me. > > I performed the following: > > 1. Built the project from the source code tar file, kafka-2.8.0-src.tgz > > 2. Ran all of the junit tests locally via "./gradlew test > -PignoreFailures=true". I had one failure in > SocketServerTest#testIdleConnection, which I was not able to reproduce when > I re-ran the test > > 3. Started a KRaft cluster by following the instructions in > config/kraft/README.md. Verified producing and consuming. > > 4. Ran kafka-metadata-shell.sh, also as described in > config/kraft/README.md, to look at the topics and broker metadata that had > been created. > > cheers, > Colin > > > On Fri, Apr 16, 2021, at 17:08, Randall Hauch wrote: > > Hey, John. > > > > +1 (binding) > > > > I've performed the following: > > 1. Validated the signatures and checksums > > 2. Built the project from the src tgz file, and ran some of the unit > tests > > 3. Built from the tag and ran a subset of the tests > > 4. Went through the quickstart for the broker and the Connect > quickstart > > (plus ran a custom connector) > > 5. Spot reviewed the site and JavaDocs > > 6. Spot checked the issues in the release notes match the Jira reports > > > > Thanks again, John! > > > > Best regards, > > > > Randall > > > > On Thu, Apr 15, 2021 at 11:30 AM Bill Bejeck wrote: > > > > > Hi John, > > > > > > Validation steps taken: > > > > > >1. Validated the signatures and checksums, > > >2. Built the project from the src tgz file, and ran the unit tests > > >3. Went through the quickstart and the Kafka Streams quickstart > > >4. Ran the quickstart for the KRaft module > > > 1. Created a topic > > > 2. Produced and consumed messages > > > 3. Ran the Metadata shell > > > > > > +1 (binding) > > > > > > Thanks for running this release! > > > > > > Bill > > > > > > > > > On Wed, Apr 14, 2021 at 4:03 PM John Roesler > wrote: > > > > > > > Hello Kafka users, developers and client-developers, > > > > > > > > This is the third candidate for release of Apache Kafka > > > > 2.8.0.This is a major release that includes many new > > > > features, including: > > > > > > > > * Early-access release of replacing Zookeeper with a > > > >self-managed quorum > > > > * Add Describe Cluster API > > > > * Support mutual TLS authentication on SASL_SSL listeners > > > > * Ergonomic improvements to Streams TopologyTestDriver > > > > * Logger API improvement to respect the hierarchy > > > > * Request/response trace logs are now JSON-formatted > > > > * New API to add and remove Streams threads while running > > > > * New REST API to expose Connect task configurations > > > > * Fixed the TimeWindowDeserializer to be able to > > > >deserialize > > > >keys outside of Streams (such as in the console consumer) > > > > * Streams resilient improvement: new uncaught exception > > > >handler > > > > * Streams resilience improvement: automatically recover > > > >from transient timeout exceptions > > > > > > > > Release notes for the 2.8.0 release: > > > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/RELEASE_NOTES.html > > > > > > > > *** Please download, test and vote by 19 April 2021 *** > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the > > > > release: > > > > https://kafka.apache.org/KEYS > > > > > > > > * Release artifacts to be voted upon (source and binary): > > > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/ > > > > > > > > * Maven artifacts to be voted upon: > > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > > > * Javadoc: > > > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/javadoc/ > > > > > > > > * Tag to be voted upon (off 2.8 branch) is the 2.8.0 tag: > > > > https://github.com/apache/kafka/releases/tag/2.8.0-rc2 > > > > > > > > * Documentation: > > > > https://kafka.apache.org/28/documentation.html > > > > > > > > * Protocol: > > > > https://kafka.apache.org/28/protocol.html > > > > > > > > * Successful Jenkins builds for the 2.8 branch: > > > > Unit/integration tests: > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/ > > > > (still flaky) > > > > > > > > System tests: > > > > > > > > https://jenkins.confluent.io/job/system-test-kafka/job/2.8/6 > > > > 0/ > > > > > > > > > > > > > > > > http://confluent-kafka-2-8-system-test-results.s3-us-west-2.amazonaws.com/2021-04-14--001.1618407001--confluentinc--2.8--1b61272d45/report.html > > > > > > > > /** > > > > > > > > Thanks, > > > > John > > > > > > > > > > > > > > > > > >
Re: [VOTE] 2.8.0 RC2
Hi John, +1 (binding) from me. I performed the following: 1. Built the project from the source code tar file, kafka-2.8.0-src.tgz 2. Ran all of the junit tests locally via "./gradlew test -PignoreFailures=true". I had one failure in SocketServerTest#testIdleConnection, which I was not able to reproduce when I re-ran the test 3. Started a KRaft cluster by following the instructions in config/kraft/README.md. Verified producing and consuming. 4. Ran kafka-metadata-shell.sh, also as described in config/kraft/README.md, to look at the topics and broker metadata that had been created. cheers, Colin On Fri, Apr 16, 2021, at 17:08, Randall Hauch wrote: > Hey, John. > > +1 (binding) > > I've performed the following: > 1. Validated the signatures and checksums > 2. Built the project from the src tgz file, and ran some of the unit tests > 3. Built from the tag and ran a subset of the tests > 4. Went through the quickstart for the broker and the Connect quickstart > (plus ran a custom connector) > 5. Spot reviewed the site and JavaDocs > 6. Spot checked the issues in the release notes match the Jira reports > > Thanks again, John! > > Best regards, > > Randall > > On Thu, Apr 15, 2021 at 11:30 AM Bill Bejeck wrote: > > > Hi John, > > > > Validation steps taken: > > > >1. Validated the signatures and checksums, > >2. Built the project from the src tgz file, and ran the unit tests > >3. Went through the quickstart and the Kafka Streams quickstart > >4. Ran the quickstart for the KRaft module > > 1. Created a topic > > 2. Produced and consumed messages > > 3. Ran the Metadata shell > > > > +1 (binding) > > > > Thanks for running this release! > > > > Bill > > > > > > On Wed, Apr 14, 2021 at 4:03 PM John Roesler wrote: > > > > > Hello Kafka users, developers and client-developers, > > > > > > This is the third candidate for release of Apache Kafka > > > 2.8.0.This is a major release that includes many new > > > features, including: > > > > > > * Early-access release of replacing Zookeeper with a > > >self-managed quorum > > > * Add Describe Cluster API > > > * Support mutual TLS authentication on SASL_SSL listeners > > > * Ergonomic improvements to Streams TopologyTestDriver > > > * Logger API improvement to respect the hierarchy > > > * Request/response trace logs are now JSON-formatted > > > * New API to add and remove Streams threads while running > > > * New REST API to expose Connect task configurations > > > * Fixed the TimeWindowDeserializer to be able to > > >deserialize > > >keys outside of Streams (such as in the console consumer) > > > * Streams resilient improvement: new uncaught exception > > >handler > > > * Streams resilience improvement: automatically recover > > >from transient timeout exceptions > > > > > > Release notes for the 2.8.0 release: > > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/RELEASE_NOTES.html > > > > > > *** Please download, test and vote by 19 April 2021 *** > > > > > > Kafka's KEYS file containing PGP keys we use to sign the > > > release: > > > https://kafka.apache.org/KEYS > > > > > > * Release artifacts to be voted upon (source and binary): > > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/ > > > > > > * Maven artifacts to be voted upon: > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > * Javadoc: > > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/javadoc/ > > > > > > * Tag to be voted upon (off 2.8 branch) is the 2.8.0 tag: > > > https://github.com/apache/kafka/releases/tag/2.8.0-rc2 > > > > > > * Documentation: > > > https://kafka.apache.org/28/documentation.html > > > > > > * Protocol: > > > https://kafka.apache.org/28/protocol.html > > > > > > * Successful Jenkins builds for the 2.8 branch: > > > Unit/integration tests: > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/ > > > (still flaky) > > > > > > System tests: > > > > > > https://jenkins.confluent.io/job/system-test-kafka/job/2.8/6 > > > 0/ > > > > > > > > > > > http://confluent-kafka-2-8-system-test-results.s3-us-west-2.amazonaws.com/2021-04-14--001.1618407001--confluentinc--2.8--1b61272d45/report.html > > > > > > /** > > > > > > Thanks, > > > John > > > > > > > > > > > >
[jira] [Resolved] (KAFKA-9119) KIP-500: Replace ZooKeeper with a Metadata Quorum
[ https://issues.apache.org/jira/browse/KAFKA-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9119. - Fix Version/s: 2.8.0 Resolution: Done > KIP-500: Replace ZooKeeper with a Metadata Quorum > - > > Key: KAFKA-9119 > URL: https://issues.apache.org/jira/browse/KAFKA-9119 > Project: Kafka > Issue Type: Improvement >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Major > Labels: kip-500 > Fix For: 2.8.0 > > > Umbrella JIRA for tasks related to KIP-500: Replace ZooKeeper with a Metadata > Quorum -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9125) GroupMetadataManager and TransactionStateManager should query the controller instead of zkClient
[ https://issues.apache.org/jira/browse/KAFKA-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9125. - Fix Version/s: 2.8.0 Assignee: Ron Dagostino (was: Viktor Somogyi-Vass) Resolution: Fixed > GroupMetadataManager and TransactionStateManager should query the controller > instead of zkClient > > > Key: KAFKA-9125 > URL: https://issues.apache.org/jira/browse/KAFKA-9125 > Project: Kafka > Issue Type: Sub-task >Reporter: Viktor Somogyi-Vass >Assignee: Ron Dagostino >Priority: Major > Fix For: 2.8.0 > > > Both classes query their respective topic's partition count via the zkClient. > This however could be queried by the broker's local metadata cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congratulations Randall ! > On Apr 17, 2021, at 6:24 PM, Randall Hauch wrote: > > Thanks for the kind responses, everyone! > > Best regards, > > Randall > >> On Sat, Apr 17, 2021 at 4:00 PM Guozhang Wang wrote: >> >> Congratulations Randall ! Well deserved. >> >> Guozhang >> >>> On Fri, Apr 16, 2021 at 4:45 PM Matthias J. Sax wrote: >>> >>> Hi, >>> >>> It's my pleasure to announce that Randall Hauch in now a member of the >>> Kafka PMC. >>> >>> Randall has been a Kafka committer since Feb 2019. He has remained >>> active in the community since becoming a committer. >>> >>> >>> >>> Congratulations Randall! >>> >>> -Matthias, on behalf of Apache Kafka PMC >>> >> >> >> -- >> -- Guozhang >>
Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck
Congratulations Bill!! > On Apr 17, 2021, at 5:20 PM, Kowshik Prakasam > wrote: > > Congrats Bill! > > > Cheers, > Kowshik > >> On Mon, Apr 12, 2021, 11:15 AM Randall Hauch wrote: >> >> Congratulations, Bill! >> >>> On Mon, Apr 12, 2021 at 11:02 AM Guozhang Wang wrote: >>> >>> Congratulations Bill ! >>> >>> Guozhang >>> On Wed, Apr 7, 2021 at 6:16 PM Matthias J. Sax wrote: >>> Hi, It's my pleasure to announce that Bill Bejeck in now a member of the Kafka PMC. Bill has been a Kafka committer since Feb 2019. He has remained active in the community since becoming a committer. Congratulations Bill! -Matthias, on behalf of Apache Kafka PMC >>> >>> >>> -- >>> -- Guozhang >>> >>
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Thanks for the kind responses, everyone! Best regards, Randall On Sat, Apr 17, 2021 at 4:00 PM Guozhang Wang wrote: > Congratulations Randall ! Well deserved. > > Guozhang > > On Fri, Apr 16, 2021 at 4:45 PM Matthias J. Sax wrote: > > > Hi, > > > > It's my pleasure to announce that Randall Hauch in now a member of the > > Kafka PMC. > > > > Randall has been a Kafka committer since Feb 2019. He has remained > > active in the community since becoming a committer. > > > > > > > > Congratulations Randall! > > > > -Matthias, on behalf of Apache Kafka PMC > > > > > -- > -- Guozhang >
[jira] [Resolved] (KAFKA-9166) Implement MetadataFetch API
[ https://issues.apache.org/jira/browse/KAFKA-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-9166. - Fix Version/s: 2.8.0 Resolution: Duplicate Closing as duplicate of KAFKA-10435 > Implement MetadataFetch API > --- > > Key: KAFKA-9166 > URL: https://issues.apache.org/jira/browse/KAFKA-9166 > Project: Kafka > Issue Type: Sub-task > Components: controller >Reporter: Viktor Somogyi-Vass >Assignee: Colin McCabe >Priority: Major > Fix For: 2.8.0 > > > Brief description of the ask is mentioned in KIP-500's > [BrokerMetadataManagement|https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-BrokerMetadataManagement] > Instead of the controller pushing out updates to the other brokers, those > brokers will fetch updates from the active controller via the new > MetadataFetch API. > A MetadataFetch is similar to a fetch request. Just like with a fetch > request, the broker will track the offset of the last updates it fetched, and > only request newer updates from the active controller. > The broker will persist the metadata it fetched to disk. This will allow the > broker to start up very quickly, even if there are hundreds of thousands or > even millions of partitions. (Note that since this persistence is an > optimization, we can leave it out of the first version, if it makes > development easier.) > Most of the time, the broker should only need to fetch the deltas, not the > full state. However, if the broker is too far behind the active controller, > or if the broker has no cached metadata at all, the controller will send a > full metadata image rather than a series of deltas. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9124) KIP-497: ISR changes should be propagated via Kafka protocol
[
https://issues.apache.org/jira/browse/KAFKA-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin McCabe resolved KAFKA-9124.
-
Fix Version/s: 2.7.0
Resolution: Fixed
> KIP-497: ISR changes should be propagated via Kafka protocol
>
>
> Key: KAFKA-9124
> URL: https://issues.apache.org/jira/browse/KAFKA-9124
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Viktor Somogyi-Vass
>Assignee: David Arthur
>Priority: Major
> Fix For: 2.7.0
>
>
> Currently {{Partition.expandIsr}} and {{Partition.shrinkIsr}} updates
> Zookeeper which is listened by the controller and that's how it notices the
> ISR changes and sends out metadata requests.
> Instead of this the brokers should use Kafka protocol messages to send out
> ISR change notifications.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-12345) KIP-500: AlterIsrManager crashes on broker idle-state
[
https://issues.apache.org/jira/browse/KAFKA-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin McCabe resolved KAFKA-12345.
--
Fix Version/s: 2.8.0
Assignee: Boyang Chen
Resolution: Fixed
> KIP-500: AlterIsrManager crashes on broker idle-state
> -
>
> Key: KAFKA-12345
> URL: https://issues.apache.org/jira/browse/KAFKA-12345
> Project: Kafka
> Issue Type: Task
> Components: core
>Affects Versions: 2.8.0
>Reporter: Alok Nikhil
>Assignee: Boyang Chen
>Priority: Minor
> Labels: kip-500
> Fix For: 2.8.0
>
>
> Occasionally, a scheduler thread on a broker crashes with this stack
>
> {code:java}
> [2021-02-19 01:04:24,683] ERROR Uncaught exception in scheduled task
> 'send-alter-isr' (kafka.utils.KafkaScheduler)
> java.lang.NullPointerException
> at kafka.server.AlterIsrManagerImpl.sendRequest(AlterIsrManager.scala:117)
> at
> kafka.server.AlterIsrManagerImpl.propagateIsrChanges(AlterIsrManager.scala:85)
> at
> kafka.server.AlterIsrManagerImpl.$anonfun$start$1(AlterIsrManager.scala:66)
> at kafka.utils.KafkaScheduler.$anonfun$schedule$2(KafkaScheduler.scala:114)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834){code}
>
> After that the broker is unable to fetch any records from any other broker
> (and vice versa)
> {code:java}
> [2021-02-19 01:05:07,000] INFO [ReplicaFetcher replicaId=0, leaderId=4,
> fetcherId=0] Error sending fetch request (sessionId=164432409
> 2, epoch=957) to node 4: (org.apache.kafka.clients.FetchSessionHandler)
> java.io.IOException: Connection to 4 was disconnected before the response
> was read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:110)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:215)
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:313)
> at
> kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:139)
> at
> kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:138)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:121)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-12282) KIP-500: Reconcile configuration variables between trunk and the KIP-500 branch
[ https://issues.apache.org/jira/browse/KAFKA-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-12282. -- Fix Version/s: 2.8.0 Resolution: Fixed > KIP-500: Reconcile configuration variables between trunk and the KIP-500 > branch > --- > > Key: KAFKA-12282 > URL: https://issues.apache.org/jira/browse/KAFKA-12282 > Project: Kafka > Issue Type: Task > Components: core >Affects Versions: 2.8.0 >Reporter: Alok Nikhil >Assignee: Alok Nikhil >Priority: Minor > Labels: kip-500 > Fix For: 2.8.0 > > > Some config changes/additions were made on the KIP-500 branch that need to be > merged back in to trunk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12276) Add KIP-500 controller code
[ https://issues.apache.org/jira/browse/KAFKA-12276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-12276. -- Fix Version/s: 2.8.0 Resolution: Fixed > Add KIP-500 controller code > --- > > Key: KAFKA-12276 > URL: https://issues.apache.org/jira/browse/KAFKA-12276 > Project: Kafka > Issue Type: Improvement > Components: controller >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Major > Labels: kip-500 > Fix For: 2.8.0 > > > Add the KIP-500 controller code -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck
Congrats Bill! Cheers, Kowshik On Mon, Apr 12, 2021, 11:15 AM Randall Hauch wrote: > Congratulations, Bill! > > On Mon, Apr 12, 2021 at 11:02 AM Guozhang Wang wrote: > > > Congratulations Bill ! > > > > Guozhang > > > > On Wed, Apr 7, 2021 at 6:16 PM Matthias J. Sax wrote: > > > > > Hi, > > > > > > It's my pleasure to announce that Bill Bejeck in now a member of the > > > Kafka PMC. > > > > > > Bill has been a Kafka committer since Feb 2019. He has remained > > > active in the community since becoming a committer. > > > > > > > > > > > > Congratulations Bill! > > > > > > -Matthias, on behalf of Apache Kafka PMC > > > > > > > > > -- > > -- Guozhang > > >
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congratulations Randall ! Well deserved. Guozhang On Fri, Apr 16, 2021 at 4:45 PM Matthias J. Sax wrote: > Hi, > > It's my pleasure to announce that Randall Hauch in now a member of the > Kafka PMC. > > Randall has been a Kafka committer since Feb 2019. He has remained > active in the community since becoming a committer. > > > > Congratulations Randall! > > -Matthias, on behalf of Apache Kafka PMC > -- -- Guozhang
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congratulations Randall! On Sat, Apr 17, 2021 at 11:51 AM Kowshik Prakasam wrote: > > Congrats Randall! > > > Cheers, > Kowshik > > On Sat, Apr 17, 2021, 5:28 AM Rankesh Kumar > wrote: > > > Congratulations, Randall! > > Best regards, > > Rankesh Kumar > > Partner Solutions Engineer > > +91 (701)913-0147 > > Follow us: Blog • Slack • Twitter • YouTube > > > > > On 17-Apr-2021, at 1:41 PM, Tom Bentley wrote: > > > > > > Congratulations Randall! > > > > > > > > > > > > On Sat, Apr 17, 2021 at 7:36 AM feyman2009 > > > > > wrote: > > > > > >> Congratulations Randall! > > >> > > >> Haoran > > >> -- > > >> 发件人：Luke Chen > > >> 发送时间：2021年4月17日(星期六) 12:05 > > >> 收件人：Kafka Users > > >> 抄 送：dev > > >> 主 题：Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch > > >> > > >> Congratulations Randall! > > >> > > >> Luke > > >> > > >> Bill Bejeck 於 2021年4月17日 週六 上午11:33 寫道： > > >> > > >>> Congratulations Randall! > > >>> > > >>> -Bill > > >>> > > >>> On Fri, Apr 16, 2021 at 11:10 PM lobo xu > > wrote: > > >>> > > Congrats Randall > > > > >>> > > >> > > >> > > > >
[jira] [Created] (KAFKA-12680) Failed to restart the broker in kraft mode
Wenbing Shen created KAFKA-12680: Summary: Failed to restart the broker in kraft mode Key: KAFKA-12680 URL: https://issues.apache.org/jira/browse/KAFKA-12680 Project: Kafka Issue Type: Bug Reporter: Wenbing Shen I tested kraft mode for the first time today, I deployed a single node kraft mode broker according to the documentation: [https://github.com/apache/kafka/blob/6d1d68617ecd023b787f54aafc24a4232663428d/config/kraft/README.md] first step: ./bin/kafka-storage.sh random-uuid Second step: Use the uuid generated above to execute the following commands: ./bin/kafka-storage.sh format -t -c ./config/kraft/server.properties third step: ./bin/kafka-server-start.sh ./config/kraft/server.properties Then I created two topics with two partitions and a single replica. ./bin/kafka-topics.sh --create --topic test-01 --partitions 2 --replication-factor 1 --bootstrap-server localhost:9092 Verify that there is no problem with production and consumption, but when I call kafka-server-stop.sh, when I call the start command again, the broker starts to report an error. I am not sure if it is a known bug or a problem with my usage [2021-04-18 00:19:37,443] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) java.io.IOException: Invalid argument at java.io.RandomAccessFile.setLength(Native Method) at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:189) at kafka.log.AbstractIndex.resize(AbstractIndex.scala:175) at kafka.log.AbstractIndex.$anonfun$trimToValidSize$1(AbstractIndex.scala:241) at kafka.log.AbstractIndex.trimToValidSize(AbstractIndex.scala:241) at kafka.log.LogSegment.recover(LogSegment.scala:385) at kafka.log.Log.recoverSegment(Log.scala:741) at kafka.log.Log.recoverLog(Log.scala:894) at kafka.log.Log.$anonfun$loadSegments$2(Log.scala:816) at kafka.log.Log$$Lambda$153/391630194.apply$mcJ$sp(Unknown Source) at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17) at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2456) at kafka.log.Log.loadSegments(Log.scala:816) at kafka.log.Log.(Log.scala:326) at kafka.log.Log$.apply(Log.scala:2593) at kafka.raft.KafkaMetadataLog$.apply(KafkaMetadataLog.scala:358) at kafka.raft.KafkaRaftManager.buildMetadataLog(RaftManager.scala:253) at kafka.raft.KafkaRaftManager.(RaftManager.scala:127) at kafka.server.KafkaRaftServer.(KafkaRaftServer.scala:74) at kafka.Kafka$.buildServer(Kafka.scala:79) at kafka.Kafka$.main(Kafka.scala:87) at kafka.Kafka.main(Kafka.scala) -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congrats Randall! Cheers, Kowshik On Sat, Apr 17, 2021, 5:28 AM Rankesh Kumar wrote: > Congratulations, Randall! > Best regards, > Rankesh Kumar > Partner Solutions Engineer > +91 (701)913-0147 > Follow us: Blog • Slack • Twitter • YouTube > > > On 17-Apr-2021, at 1:41 PM, Tom Bentley wrote: > > > > Congratulations Randall! > > > > > > > > On Sat, Apr 17, 2021 at 7:36 AM feyman2009 > > > wrote: > > > >> Congratulations Randall! > >> > >> Haoran > >> -- > >> 发件人：Luke Chen > >> 发送时间：2021年4月17日(星期六) 12:05 > >> 收件人：Kafka Users > >> 抄 送：dev > >> 主 题：Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch > >> > >> Congratulations Randall! > >> > >> Luke > >> > >> Bill Bejeck 於 2021年4月17日 週六 上午11:33 寫道： > >> > >>> Congratulations Randall! > >>> > >>> -Bill > >>> > >>> On Fri, Apr 16, 2021 at 11:10 PM lobo xu > wrote: > >>> > Congrats Randall > > >>> > >> > >> > >
[jira] [Resolved] (KAFKA-10783) Rewrite TopicPartitionStateZNode struct with auto-generated protocol
[ https://issues.apache.org/jira/browse/KAFKA-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dengziming resolved KAFKA-10783. Resolution: Won't Do > Rewrite TopicPartitionStateZNode struct with auto-generated protocol > > > Key: KAFKA-10783 > URL: https://issues.apache.org/jira/browse/KAFKA-10783 > Project: Kafka > Issue Type: Sub-task >Reporter: dengziming >Assignee: dengziming >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
Requesting review of PR 10377
Hi team, Could anyone help to review PR 10377, thanks! KAFKA-12515 ApiVersionManager should create a response based on the request https://github.com/apache/kafka/pull/10377/files Haoran Xuan
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congratulations, Randall! Best regards, Rankesh Kumar Partner Solutions Engineer +91 (701)913-0147 Follow us: Blog • Slack • Twitter • YouTube > On 17-Apr-2021, at 1:41 PM, Tom Bentley wrote: > > Congratulations Randall! > > > > On Sat, Apr 17, 2021 at 7:36 AM feyman2009 > wrote: > >> Congratulations Randall! >> >> Haoran >> -- >> 发件人：Luke Chen >> 发送时间：2021年4月17日(星期六) 12:05 >> 收件人：Kafka Users >> 抄 送：dev >> 主 题：Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch >> >> Congratulations Randall! >> >> Luke >> >> Bill Bejeck 於 2021年4月17日 週六 上午11:33 寫道： >> >>> Congratulations Randall! >>> >>> -Bill >>> >>> On Fri, Apr 16, 2021 at 11:10 PM lobo xu wrote: >>> Congrats Randall >>> >> >>
Re: [DISCUSS] KIP-693: Client-side Circuit Breaker for Partition Write Errors
Thanks a lot for the comments, Jun! Indeed this is a practical solution originated from the field and we really appreciate the guidance to make it more general. Please refer to the embedded response to the specific questions below. On 2021/04/07 17:59:59, Jun Rao wrote: > Hi, George, > > A few more comments on the KIP. > > 1. It would be useful to motivate the problem a bit more. For example, is > the KIP trying to solve a transient broker problem (if so, for how long) or > a permanent broker problem? It would also be useful to list some common > causes that can slow the broker down. [GS] The most common scenario that directly led to this change is disk failure, a permanent problem in the sense that it requires intervention from the admin. However we also saw this effective under situations where the broker can self-heal, such as temporary network connection issue. In the end as long as the short circuit mechanism itself can adapt, it is agnostic to duration of the failure. > > 2. It would be useful to discuss a bit more on the high level approach > (e.g. in the rejected section). This KIP proposes to fix the issue on the > client side by having a pluggable component to redirect the traffic to > other brokers. One potential issue with this is that it requires all > clients to opt in (assuming this is not the default) for the plugin to see > the benefit. In some environments with a large number of clients, > coordinating all those clients may not be easy. Another potential solution > is to fix the issue on the server side. For example, if a broker is slow > because it has noisy neighbors in a virtual environment, we could > proactively bring down the broker and restart it somewhere else. This has > the benefit that it requires less client side coordination. > [GS] Agree with the judgement on the client coordination complexity. We added a section at the end of the KIP to elaborate a bit more. Basically we think client-side circuit breaking and server side broker high availability are complementary instead of conflicting. On one hand it is not likely (or extremely expensive) to implement broker HA in the control plane; on the other hand we have also often seen client side mechanism used to mitigate network problem between client and broker. An analogy is that most RPC frameworks implement filtering for problematic node on both server side and client side. > 3. Regarding how to detect broker slowness in the client. The proposal is > based on the error in the produce response. Typically, if the broker is > just slow, the only type of error the client gets is the timeout exception. > Since the default timeout is 30 seconds, it may not be triggered all the > time and it may be too late to reflect a broker side issue. I am wondering > if there are other better indicators. For example, another potential option > is to use the number of pending batches per partition (or broker) in the > Accumulator. Intuitively, if a broker is slow, all partitions with the > leader on it will gradually accumulate more batches. > [GS] It is indeed naive to rely only on timeout. We iterated this draft twice and the current version does support state context to be passed to custom implementation of the circuit breaker class, including write result, inflight requests and pending batches. > 4. It would be useful to have a solution that works with keyed messages so > that they can still be distributed to the partition based on the hash of > the key. [GS] Agree. We have not found straightforward way to support keyed messages. On the other hand, when we discuss the trade-off between message ordering (within controllable time window) and high availability, almost all our customer prefer the latter. Hence we have rolled out this without support of keyed messages. > > Thanks, > > Jun > > > On Wed, Mar 24, 2021 at 4:05 AM Guoqiang Shu wrote: > > > > > In our current proposal it can be configured via > > producer.circuit.breaker.mute.retry.interval (defaulted to 10 mins), but > > perhaps 'interval' is a confusing name. > > > > On 2021/03/23 00:45:23, Guozhang Wang wrote: > > > Thanks for the updated KIP! Some more comments inlined. > > > > > > > > I'm still not sure if, in your proposal, the muting length is a > > > customizable value (and if yes, through which config) or it is always > > hard > > > coded as 10 minutes? > > > > > > > > > > > Guozhang > > > > >
Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congratulations Randall! On Sat, Apr 17, 2021 at 7:36 AM feyman2009 wrote: > Congratulations Randall! > > Haoran > -- > 发件人：Luke Chen > 发送时间：2021年4月17日(星期六) 12:05 > 收件人：Kafka Users > 抄 送：dev > 主 题：Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch > > Congratulations Randall! > > Luke > > Bill Bejeck 於 2021年4月17日 週六 上午11:33 寫道： > > > Congratulations Randall! > > > > -Bill > > > > On Fri, Apr 16, 2021 at 11:10 PM lobo xu wrote: > > > > > Congrats Randall > > > > > > >
回复：[ANNOUNCE] New Kafka PMC Member: Randall Hauch
Congratulations Randall! Haoran -- 发件人：Luke Chen 发送时间：2021年4月17日(星期六) 12:05 收件人：Kafka Users 抄 送：dev 主 题：Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch Congratulations Randall! Luke Bill Bejeck 於 2021年4月17日 週六 上午11:33 寫道： > Congratulations Randall! > > -Bill > > On Fri, Apr 16, 2021 at 11:10 PM lobo xu wrote: > > > Congrats Randall > > >
