[jira] [Created] (KAFKA-17545) Get rid of fetch queue in SharePartitionManager
Abhinav Dixit created KAFKA-17545: - Summary: Get rid of fetch queue in SharePartitionManager Key: KAFKA-17545 URL: https://issues.apache.org/jira/browse/KAFKA-17545 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit Since we are fetching from the share fetch purgatory, the fetch queue becomes redundant. Hence, we should get rid of it from Share Partition Manager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17401) Add config for purge interval for Share Fetch Purgatory
[ https://issues.apache.org/jira/browse/KAFKA-17401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-17401. --- Fix Version/s: 4.0.0 Resolution: Fixed > Add config for purge interval for Share Fetch Purgatory > --- > > Key: KAFKA-17401 > URL: https://issues.apache.org/jira/browse/KAFKA-17401 > Project: Kafka > Issue Type: Sub-task > Reporter: Abhinav Dixit > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17522) Share partition acquire() need not return a future
Abhinav Dixit created KAFKA-17522: - Summary: Share partition acquire() need not return a future Key: KAFKA-17522 URL: https://issues.apache.org/jira/browse/KAFKA-17522 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit Assignee: Apoorv Mittal As per discussion [https://github.com/apache/kafka/pull/16274#discussion_r1700968453] and [https://github.com/apache/kafka/pull/16969#discussion_r1752362118] , we don't need acquire method to return a future since we are not persisting acquisitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17447) Changed fetch queue processing to reduce the no. of locking and unlocking activity
Abhinav Dixit created KAFKA-17447: - Summary: Changed fetch queue processing to reduce the no. of locking and unlocking activity Key: KAFKA-17447 URL: https://issues.apache.org/jira/browse/KAFKA-17447 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit Assignee: Abhinav Dixit For the share groups fetch request processing, we have an recursive approach of dealing with individual fetch requests. While it works fine with less no. of records (< 1,000,000) and lesser sharing (< 5 share consumers), it seems that some requests are getting stuck when we increase the load and try to increase the throughput. I've replaced this approach by removing the unlocking and locking of fetch queue in between entries. This had reduced the complexity and also removes the reliability issue on increasing the load. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17401) Add config for purge interval for Share Fetch Purgatory
Abhinav Dixit created KAFKA-17401: - Summary: Add config for purge interval for Share Fetch Purgatory Key: KAFKA-17401 URL: https://issues.apache.org/jira/browse/KAFKA-17401 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17400) Intorduce a purgatory to deal with share fetch requests that cannot be completed instantaneously
Abhinav Dixit created KAFKA-17400: - Summary: Intorduce a purgatory to deal with share fetch requests that cannot be completed instantaneously Key: KAFKA-17400 URL: https://issues.apache.org/jira/browse/KAFKA-17400 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit h1. When record lock partition limit is reached, the ShareFetch should wait for up to MaxWaitMs for records to be released -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17210) Broker fixes for smooth concurrent fetches on share partition
[ https://issues.apache.org/jira/browse/KAFKA-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-17210. --- Fix Version/s: 4.0.0 Resolution: Fixed > Broker fixes for smooth concurrent fetches on share partition > - > > Key: KAFKA-17210 > URL: https://issues.apache.org/jira/browse/KAFKA-17210 > Project: Kafka > Issue Type: Sub-task > Reporter: Abhinav Dixit > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0 > > > Identified a couple of reliability issues with broker code for share groups - > # Broker seems to get stuck at times when using multiple share consumers due > to a corner case where the second last fetch request did not contain any > topic partition to fetch, because of which the broker could never complete > the last request. This results in a share fetch request getting stuck. > # Since persister would not perform any business logic around sending state > batches for a share partition, there could be scenarios where it sends state > batches with no AVAILABLE records. This could cause a breach on the limit of > in-flight messages we have configured, and hence broker would never be able > to complete the share fetch requests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17210) Broker fixes for smooth concurrent fetches on share partition
Abhinav Dixit created KAFKA-17210: - Summary: Broker fixes for smooth concurrent fetches on share partition Key: KAFKA-17210 URL: https://issues.apache.org/jira/browse/KAFKA-17210 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit Assignee: Abhinav Dixit Identified a couple of reliability issues with broker code for share groups - # Broker seems to get stuck at times when using multiple share consumers due to a corner case where the second last fetch request did not contain any topic partition to fetch, because of which the broker could never complete the last request. This results in a share fetch request getting stuck. # Since persister would not perform any business logic around sending state batches for a share partition, there could be scenarios where it sends state batches with no AVAILABLE records. This could cause a breach on the limit of in-flight messages we have configured, and hence broker would never be able to complete the share fetch requests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17071) SharePartition - Add more unit tests
[ https://issues.apache.org/jira/browse/KAFKA-17071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-17071. --- Fix Version/s: 4.0.0 3.9.0 Resolution: Fixed > SharePartition - Add more unit tests > > > Key: KAFKA-17071 > URL: https://issues.apache.org/jira/browse/KAFKA-17071 > Project: Kafka > Issue Type: Sub-task > Reporter: Abhinav Dixit > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0, 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17071) SharePartition - Add more unit tests
Abhinav Dixit created KAFKA-17071: - Summary: SharePartition - Add more unit tests Key: KAFKA-17071 URL: https://issues.apache.org/jira/browse/KAFKA-17071 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17026) Implement updateCacheAndOffsets functionality on LSO movement
[ https://issues.apache.org/jira/browse/KAFKA-17026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-17026. --- Fix Version/s: 4.0.0 3.9.0 Resolution: Fixed > Implement updateCacheAndOffsets functionality on LSO movement > - > > Key: KAFKA-17026 > URL: https://issues.apache.org/jira/browse/KAFKA-17026 > Project: Kafka > Issue Type: Sub-task > Reporter: Abhinav Dixit > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0, 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15727) KRaft support in AlterUserScramCredentialsRequestNotAuthorizedTest
[
https://issues.apache.org/jira/browse/KAFKA-15727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhinav Dixit resolved KAFKA-15727.
---
Resolution: Fixed
> KRaft support in AlterUserScramCredentialsRequestNotAuthorizedTest
> --
>
> Key: KAFKA-15727
> URL: https://issues.apache.org/jira/browse/KAFKA-15727
> Project: Kafka
> Issue Type: Task
> Components: core
>Reporter: Sameer Tejani
> Assignee: Abhinav Dixit
>Priority: Minor
> Labels: kraft, kraft-test, newbie
>
> The following tests in AlterUserScramCredentialsRequestNotAuthorizedTest in
> core/src/test/scala/unit/kafka/server/AlterUserScramCredentialsRequestNotAuthorizedTest.scala
> need to be updated to support KRaft
> 47 : def testAlterNothingNotAuthorized(): Unit = {
> 59 : def testAlterSomethingNotAuthorized(): Unit = {
> Scanned 80 lines. Found 0 KRaft tests out of 2 tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16754) Implement release acquired records functionality in SharePartition
[ https://issues.apache.org/jira/browse/KAFKA-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-16754. --- Fix Version/s: 4.0.0 3.9.0 Resolution: Fixed > Implement release acquired records functionality in SharePartition > -- > > Key: KAFKA-16754 > URL: https://issues.apache.org/jira/browse/KAFKA-16754 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0, 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16751) Implement release acquired records in SharePartitionManager
[ https://issues.apache.org/jira/browse/KAFKA-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-16751. --- Fix Version/s: 4.0.0 3.9.0 Resolution: Fixed > Implement release acquired records in SharePartitionManager > --- > > Key: KAFKA-16751 > URL: https://issues.apache.org/jira/browse/KAFKA-16751 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0, 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17003) Implement SharePartitionManager close functionality
[ https://issues.apache.org/jira/browse/KAFKA-17003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-17003. --- Fix Version/s: 4.0.0 3.9.0 Resolution: Fixed > Implement SharePartitionManager close functionality > --- > > Key: KAFKA-17003 > URL: https://issues.apache.org/jira/browse/KAFKA-17003 > Project: Kafka > Issue Type: Sub-task > Reporter: Abhinav Dixit > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0, 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16755) Implement lock timeout functionality in SharePartition
[ https://issues.apache.org/jira/browse/KAFKA-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-16755. --- Fix Version/s: 4.0.0 3.9.0 Resolution: Fixed > Implement lock timeout functionality in SharePartition > -- > > Key: KAFKA-16755 > URL: https://issues.apache.org/jira/browse/KAFKA-16755 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0, 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17004) MINOR: Remove extra synchronized blocks in SharePartitionManager
[ https://issues.apache.org/jira/browse/KAFKA-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-17004. --- Fix Version/s: 4.0.0 3.9.0 Resolution: Fixed > MINOR: Remove extra synchronized blocks in SharePartitionManager > > > Key: KAFKA-17004 > URL: https://issues.apache.org/jira/browse/KAFKA-17004 > Project: Kafka > Issue Type: Sub-task > Reporter: Abhinav Dixit > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0, 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17026) Implement updateCachedAndOffsets functionality on LSO movement
Abhinav Dixit created KAFKA-17026: - Summary: Implement updateCachedAndOffsets functionality on LSO movement Key: KAFKA-17026 URL: https://issues.apache.org/jira/browse/KAFKA-17026 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit Assignee: Abhinav Dixit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17004) MINOR: Remove extra synchronized blocks in SharePartitionManager
Abhinav Dixit created KAFKA-17004: - Summary: MINOR: Remove extra synchronized blocks in SharePartitionManager Key: KAFKA-17004 URL: https://issues.apache.org/jira/browse/KAFKA-17004 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit Assignee: Abhinav Dixit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17003) Implement SharePartitionManager close functionality
Abhinav Dixit created KAFKA-17003: - Summary: Implement SharePartitionManager close functionality Key: KAFKA-17003 URL: https://issues.apache.org/jira/browse/KAFKA-17003 Project: Kafka Issue Type: Sub-task Reporter: Abhinav Dixit Assignee: Abhinav Dixit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16747) Implement share sessions and context
[ https://issues.apache.org/jira/browse/KAFKA-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-16747. --- Resolution: Fixed > Implement share sessions and context > > > Key: KAFKA-16747 > URL: https://issues.apache.org/jira/browse/KAFKA-16747 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal > Assignee: Abhinav Dixit >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16914) Add Share Group dynamic and broker configs
[ https://issues.apache.org/jira/browse/KAFKA-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Dixit resolved KAFKA-16914. --- Resolution: Fixed > Add Share Group dynamic and broker configs > -- > > Key: KAFKA-16914 > URL: https://issues.apache.org/jira/browse/KAFKA-16914 > Project: Kafka > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Apoorv Mittal > Assignee: Abhinav Dixit >Priority: Major > Fix For: 4.0.0 > > > Add the configs required for share group in KafkaConfig or equivalent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Requesting permissions to contribute to Apache Kafka
Thanks Josep. On Mon, Jan 8, 2024 at 7:38 PM Josep Prat wrote: > Hi Abhinav, > > This one I could find :) > > You are now all set! Happy contributing! > > Best, > > On Mon, Jan 8, 2024 at 2:39 PM Abhinav Dixit > wrote: > > > Hi Josep, > > Thanks for the quick response. My WIKI ID is adixit, sorry for the > > confusion. > > Thanks, > > Abhinav > > > > On Mon, Jan 8, 2024 at 6:59 PM Josep Prat > > wrote: > > > > > Hi Abhinav, > > > Thanks for your interest in Apache Kafka. > > > I could set up your Jira account accordingly, but I can't find your > Wiki > > ID > > > in the system. Can you confirm you created it and named it > > > "adixitconfluent"? > > > > > > Best, > > > > > > On Mon, Jan 8, 2024 at 2:10 PM Abhinav Dixit > > > > > > wrote: > > > > > > > Hi Team, > > > > I am writing this email to request permissions to contribute to > Apache > > > > Kafka. Here are my details - > > > > JIRA ID - adixitconfluent > > > > WIKI ID - adixitconfluent > > > > > > > > Please let me know if anything is missing. > > > > Thanks, > > > > Abhinav > > > > > > > > > > > > > -- > > > [image: Aiven] <https://www.aiven.io> > > > > > > *Josep Prat* > > > Open Source Engineering Director, *Aiven* > > > josep.p...@aiven.io | +491715557497 > > > aiven.io <https://www.aiven.io> | < > > https://www.facebook.com/aivencloud > > > > > > > <https://www.linkedin.com/company/aiven/> < > > > https://twitter.com/aiven_io> > > > *Aiven Deutschland GmbH* > > > Alexanderufer 3-7, 10117 Berlin > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud > > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B >
Re: Requesting permissions to contribute to Apache Kafka
Hi Josep, Thanks for the quick response. My WIKI ID is adixit, sorry for the confusion. Thanks, Abhinav On Mon, Jan 8, 2024 at 6:59 PM Josep Prat wrote: > Hi Abhinav, > Thanks for your interest in Apache Kafka. > I could set up your Jira account accordingly, but I can't find your Wiki ID > in the system. Can you confirm you created it and named it > "adixitconfluent"? > > Best, > > On Mon, Jan 8, 2024 at 2:10 PM Abhinav Dixit > wrote: > > > Hi Team, > > I am writing this email to request permissions to contribute to Apache > > Kafka. Here are my details - > > JIRA ID - adixitconfluent > > WIKI ID - adixitconfluent > > > > Please let me know if anything is missing. > > Thanks, > > Abhinav > > > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud > > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B >
Requesting permissions to contribute to Apache Kafka
Hi Team, I am writing this email to request permissions to contribute to Apache Kafka. Here are my details - JIRA ID - adixitconfluent WIKI ID - adixitconfluent Please let me know if anything is missing. Thanks, Abhinav
