[jira] [Created] (KAFKA-19323) RemoteTopicCrudTest to use containsKey for topic config checks

Kamal Chandraprakash created KAFKA-19323: Summary: RemoteTopicCrudTest to use containsKey for topic config checks Key: KAFKA-19323 URL: https://issues.apache.org/jira/browse/KAFKA-19323

Re: [VOTE] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

+1 (binding). Thanks for the KIP! On Wed, May 21, 2025 at 9:04 PM Chia-Ping Tsai wrote: > +1 (binding) > > Kuan-Po Tseng 於 2025年5月21日 週三 下午8:10寫道： > > > Hello everyone, > > > > I would like to start a vote on KIP-1179. > > KIP Link: https://cwiki.apache.org/confluence/x/xAlWFQ > > > > Best rega

Re: [ANNOUNCE] Apache Kafka 3.9.1

Thanks TengYao and Luke for running the release! -- Kamal On Wed, May 21, 2025 at 9:08 PM Chia-Ping Tsai wrote: > Thanks to Teng and Luke for running 3.9.1! > > Justine Olshan 於 2025年5月21日 週三 下午11:34寫道： > > > Yay! Thanks Luke and TengYao for running this release. > >

Re: [DISCUSS] KIP-1179: Introduce remote.log.manager.follower.thread.pool.size config

Hi Kuan-Po, The proposal to add the new remote.log.manager.follower.thread.pool.size config, LGTM. Thanks for making this change! -- Kamal On Mon, May 12, 2025 at 10:19 PM KuanPo Tseng wrote: > Hello everyone, > > I’d like to discuss a KIP regarding introducing a new conf

Re: [VOTE] KIP-1123: Rack-aware partitioning for Kafka Producer

+1 (binding). Thanks for the KIP! On Mon, May 19, 2025, 16:58 Ivan Yurchenko wrote: > Thank you everyone! > > The KIP is accepted with 5 binding and 3 non-binding votes. > I'll make the implementation PR shortly. > > Best, > Ivan > > > On Thu, May 15, 2025, at 09:33, Luke Chen wrote: > > +1 bind

Re: [DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

s, or does it handle the larger response gracefully? Thanks, Kamal On Thu, May 8, 2025 at 7:19 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi Kamal, > Thanks for the KIP. > > While it makes a lot of sense to me to be able to control the fetching > from r

[DISCUSS] KIP-1178: Introduce remote.max.partition.fetch.bytes in Consumer

onsumer Please take a look and suggest your thoughts. Thanks, Kamal

[jira] [Reopened] (KAFKA-15777) Configurable remote fetch bytes per partition from Consumer

[ https://issues.apache.org/jira/browse/KAFKA-15777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reopened KAFKA-15777: -- > Configurable remote fetch bytes per partition from Consu

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

ntain consistency. We need to propose the KIP for it. PR: https://github.com/apache/kafka/pull/16502 Thanks, Kamal On Sat, May 3, 2025 at 2:51 PM Chia-Ping Tsai wrote: > hi Kamal > > Thanks for your response. I have two minor questions. > > 1. Have we considered adding a new config for

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

pool.size/remote.log.manager.expiration.thread.pool.size/remote.log.manager.thread.pool.size> . PR: https://github.com/apache/kafka/pull/18137 -- Kamal On Sat, May 3, 2025 at 11:09 AM Chia-Ping Tsai wrote: > hi Kamal > > Apologies for posting on this older thread. I have a question regarding > the

[jira] [Resolved] (KAFKA-19131) Exception thrown while updating the RemoteLogReader threads

[ https://issues.apache.org/jira/browse/KAFKA-19131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-19131. -- Fix Version/s: 4.1.0 Resolution: Fixed > Exception thrown wh

[jira] [Resolved] (KAFKA-19014) Potential race condition in remote-log-reader and remote-log-index-cleaner thread

[ https://issues.apache.org/jira/browse/KAFKA-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-19014. -- Fix Version/s: 4.1.0 Assignee: Kamal Chandraprakash Resolution

[jira] [Resolved] (KAFKA-19180) Fix the hanging testPendingTaskSize

[ https://issues.apache.org/jira/browse/KAFKA-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-19180. -- Fix Version/s: 4.1.0 Resolution: Fixed > Fix the hang

[jira] [Resolved] (KAFKA-17184) Remote index cache noisy logging

[ https://issues.apache.org/jira/browse/KAFKA-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-17184. -- Fix Version/s: 4.1.0 Resolution: Fixed > Remote index cache no

Re: [ANNOUNCE] New committer: Apoorv Mittal

Congratulations, Apoorv! On Sat, Apr 12, 2025 at 8:23 AM Kuan Po Tseng wrote: > Congrats Apoorv! > > On Sat, Apr 12, 2025 at 9:50 AM Guozhang Wang > wrote: > > > Congratulations Apoorv! > > > > On Fri, Apr 11, 2025 at 3:33 PM Apoorv Mittal > > wrote: > > > > > > Thank you so much everyone! I r

Re: [ANNOUNCE] New committer: TengYao Chi

Congrats, TengYao! Well deserved!! On Sat, Apr 12, 2025 at 7:19 AM Guozhang Wang wrote: > Congratulations TengYao! > > On Fri, Apr 11, 2025 at 4:36 PM Christo Lolov > wrote: > > > > Congratulations! > > > > On Thu, 10 Apr 2025 at 18:22, David Arthur wrote: > > > > > Congrats!! > > > > > > Davi

[jira] [Created] (KAFKA-19131) Exception thrown while updating the RemoteLogReader threads

Kamal Chandraprakash created KAFKA-19131: Summary: Exception thrown while updating the RemoteLogReader threads Key: KAFKA-19131 URL: https://issues.apache.org/jira/browse/KAFKA-19131 Project

Re: [ANNOUNCE] Apache Kafka 4.0.0

Thanks David for running this release!! -- Kamal On Tue, Mar 18, 2025 at 11:33 PM Justine Olshan wrote: > Yay! I'm happy to see this release. > > Thanks David for your hard work as well as everyone else in the community > who rallied to get this over the line! > > Jus

[jira] [Resolved] (KAFKA-18524) Fix flaky RemoteIndexCacheTest#testCorrectnessForCacheAndIndexFilesWhenResizeCache

[ https://issues.apache.org/jira/browse/KAFKA-18524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-18524. -- Fix Version/s: 4.1.0 Resolution: Duplicate > Fix fl

[jira] [Resolved] (KAFKA-18089) RemoteIndexCacheTest fails with caffeine > 3.1.1

[ https://issues.apache.org/jira/browse/KAFKA-18089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-18089. -- Fix Version/s: 4.1.0 Resolution: Fixed > RemoteIndexCacheTest fa

[jira] [Created] (KAFKA-18825) Fix the flaky EligibleLeaderReplicasIntegrationTest

Kamal Chandraprakash created KAFKA-18825: Summary: Fix the flaky EligibleLeaderReplicasIntegrationTest Key: KAFKA-18825 URL: https://issues.apache.org/jira/browse/KAFKA-18825 Project: Kafka

Re: [VOTE] KIP-1126: Serialize changes to Kafka with a build queue

+1 (binding), Thanks for the KIP! -- Kamal On Fri, Feb 14, 2025 at 10:25 AM Matthias J. Sax wrote: > +1 (binding) > > On 2/13/25 8:03 AM, David Arthur wrote: > > Good point, Chris! The vote will remain open for another 2 days. I will > > resend a vote summary if any addi

[jira] [Created] (KAFKA-18787) RemoteIndexCache fails to remove invalid files on init

Kamal Chandraprakash created KAFKA-18787: Summary: RemoteIndexCache fails to remove invalid files on init Key: KAFKA-18787 URL: https://issues.apache.org/jira/browse/KAFKA-18787 Project: Kafka

[jira] [Resolved] (KAFKA-18728) Move ListOffsetsPartitionStatus to server module

[ https://issues.apache.org/jira/browse/KAFKA-18728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-18728. -- Fix Version/s: 4.1.0 Resolution: Fixed > M

Re: Tagged fields constraint to start from tagId: 0

Got it. Thanks for clearing that up! -- Kamal On Tue, Feb 4, 2025 at 4:40 PM David Jacot wrote: > Hi Kamal, > > It is fine to relax the constraint in StructSpec for your internal tags. > For instance, you could allow your internal tags to start from id 1. It > is very unlike

Re: Tagged fields constraint to start from tagId: 0

gt; to start with a very large tag-id like 1 for custom tagged fields to not clash with future tagged fields being introduced upstream? TIA. Thanks, Kamal On Thu, Jan 23, 2025 at 11:47 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > In KIP-482 > &

[jira] [Created] (KAFKA-18722) Remove the unreferenced methods from TopicBasedRemoteLogMetadataManager

Kamal Chandraprakash created KAFKA-18722: Summary: Remove the unreferenced methods from TopicBasedRemoteLogMetadataManager Key: KAFKA-18722 URL: https://issues.apache.org/jira/browse/KAFKA-18722

Tagged fields constraint to start from tagId: 0

unique tagged-ids for their user-defined fields. I’d appreciate it if you could take a look and share your thoughts or feedback. Thanks, Kamal

Re: [DISCUSS] KIP-1114 Introducing Chunk in Partition

And, this strategy will only work for topics with cleanup policy set to "delete". -- Kamal On Thu, Jan 2, 2025 at 10:23 PM David Arthur wrote: > Hey De Gao, thanks for the KIP! > > As you’re probably aware, a Partition is a logical construct in Kafka. A > broker hosts a partiti

Re: [VOTE] KIP-1118: Add Deadlock Protection on Producer Network Thread

+1 (binding). Thanks for the KIP! On Mon, Dec 23, 2024, 08:28 TengYao Chi wrote: > Hi everyone, > > As the vote has been pending for a week, I would like to bump it manually. > Thank you for your attention. > > Sincerely, > TengYao > > Andrew Schofield 於 2024年12月16日 週一 > 下午10:25寫道： > > > +1 (bi

[jira] [Resolved] (KAFKA-17928) Make remote log manager thread-pool configs dynamic

[ https://issues.apache.org/jira/browse/KAFKA-17928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-17928. -- Resolution: Fixed > Make remote log manager thread-pool configs dyna

Re: [ANNOUNCE] New committer: Andrew Schofield

Congrats Andrew! On Mon, Dec 2, 2024 at 7:57 PM Abhinav Dixit wrote: > Congratulations Andrew. Very happy for you :) > > On Mon, Dec 2, 2024 at 7:52 PM TengYao Chi wrote: > > > Congratulations Andrew > > Very well deserved. > > > > > > Mayank Shekhar Narula 於 2024年12月2日 週一 > > 下午10:18寫道： > > >

Re: [VOTE] KIP-1030: Change constraints and default values for various configurations

Hi Divij, +1 (Binding). Thanks for the KIP! Kamal On Thu, Nov 28, 2024, 05:37 Luke Chen wrote: > Hi Divij, > > +1 (binding) from me. > Thanks for the KIP! > > Luke > > On Thu, Nov 28, 2024 at 12:21 AM Divij Vaidya > wrote: > > > Corresponding discussion

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

pier/expiration tasks, the follower tasks are light-weight so proposing to reduce that thread-pool size to 2. -- Kamal On Wed, Nov 27, 2024 at 11:48 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Divij, > > I also share the same concern as Greg pointed out

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

art to fail with the new defaults. I'm fine with the proposal provided we plan to fix those tests as part of this KIP. -- Kamal On Wed, Nov 27, 2024 at 9:55 PM Divij Vaidya wrote: > Hello everyone > > Since, I believe I have addressed all the concerns that were raised here, I &

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

ber of cores. > 3. It's now hard for a user to know the actual default value. > 4. It may also be a surprise that the default value changes when a user > adds a listener or a dir. > > Thanks, > > Jun > > On Wed, Nov 20, 2024 at 7:36 AM Kamal Chandraprakash < > ka

Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations

mpatibility, if the user was running with 10 threads in v3.8. In v3.9, the server will start 30 threads. So, I propose to change the default value of copier and expiration thread pools from -1 to 10. This will also ensure smooth validation while making those configs as dynamic. Thanks, Kamal On

Re: [VOTE] KIP-1106: Add duration based offset reset option for consumer clients

Hi Manikumar, Thanks for the KIP! Duration based offset reset is a useful addition when remote storage is enabled. +1 (binding) Thanks, Kamal On Thu, Nov 14, 2024 at 7:51 PM Apoorv Mittal wrote: > Hi Manikumar, > Thanks for the KIP. It will be a valuable addition. > > +1

Re: [VOTE] KIP-1080: Fix the typo: `maxlifeTimeMs` in CreateDelegationTokenOptions

+1 (binding). Thanks for the KIP! On Mon, Nov 11, 2024 at 9:41 AM Luke Chen wrote: > +1 (binding) from me. > > Thanks for the KIP. > > Luke > > On Mon, Nov 11, 2024 at 10:51 AM TengYao Chi wrote: > > > Hi > > Thanks for KIP. > > +1 (non-binding) > > > > Best, > > TengYao > > > > 黃竣陽 於 2024年11月

Re: [VOTE] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi all, The KIP has been passed with 4 binding votes and 2 non-binding votes and no -1 vote. Thanks everyone! Binding: Satish, Christo, Luke, and Chia-Ping. Non-binding: Chu Cheng Li and Federico. Thanks, Kamal On Sun, Nov 10, 2024 at 12:22 PM Chia-Ping Tsai wrote: > +1 (binding) >

[jira] [Created] (KAFKA-17980) Add isReady API to RemoteLogMetadataManager

Kamal Chandraprakash created KAFKA-17980: Summary: Add isReady API to RemoteLogMetadataManager Key: KAFKA-17980 URL: https://issues.apache.org/jira/browse/KAFKA-17980 Project: Kafka

[jira] [Reopened] (KAFKA-17801) RemoteLogManager may compute inaccurate upperBoundOffset for aborted txns

[ https://issues.apache.org/jira/browse/KAFKA-17801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reopened KAFKA-17801: -- > RemoteLogManager may compute inaccurate upperBoundOffset for aborted t

Re: [VOTE] 3.9.0 RC5

Hi all, I took the 3.9 branch from upstream and ran all the unit tests. The StorageToolTest unit test is failing consistently: % ./gradlew core:test --tests StorageToolTest Gradle Test Run :core:test > Gradle Test Executor 20 > StorageToolTest > testFo

Re: [ANNOUNCE] Apache Kafka 3.9.0

Thanks for running the release, Colin! -- Kamal On Fri, Nov 8, 2024 at 8:01 AM Luke Chen wrote: > Hi Colin, > > Thanks for all the hard work for the release! > > Luke > > On Fri, Nov 8, 2024 at 7:12 AM Colin McCabe wrote: > > > The Apache Kafka community is pleas

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

n the source code. Thanks, Kamal On Thu, Nov 7, 2024 at 9:23 PM Satish Duggana wrote: > Thanks Kamal for the KIP. This is useful for dynamically changing the > thread pool configurations, especially in production environments. We > can skip remote.log.manager.thread.pool.size as

[VOTE] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi all, I'd like to call for a vote on KIP-1105 https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic Thanks, Kamal

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi all, If there are no more comments, then I'll start a voting thread as the change is minor. -- Kamal On Thu, Nov 7, 2024 at 9:00 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Federico, > > Updated the KIP by replacing the `isInitialized` to

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi Federico, Updated the KIP by replacing the `isInitialized` to `isReady` in the KIP. On Wed, Nov 6, 2024 at 12:47 PM Federico Valeri wrote: > Thanks Kamal, LGTM, but you should replace all instances of > isInitialized to isReady in the rest of the KIP. > > On Wed, Nov 6, 202

Re: [DISCUSS] Apache Kafka 4.0.0 release

Hi Luke, Please add the KIP-1058 and KIP-1105 to the release plan. KIP-1058 is accepted and PRs are ready. KIP-1105 is a minor one, and will open a voting thread soon. Thanks, Kamal On Thu, Nov 7, 2024 at 7:28 AM Luke Chen wrote: > Hi Alyssa, > > I've added KIP-966 into th

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Hi all, The KIP is passed with 3 (+1) binding votes. Thanks everyone! Binding vote: Divij, Satish, and Luke. -- Kamal On Wed, Nov 6, 2024 at 11:53 AM Luke Chen wrote: > Had another read of the KIP, LGTM! > +1 from me. > Thanks for the KIP! > > Luke > > On Wed, Nov 6, 2

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

Hi Federico, Thanks for the review! 1. Changed the API name to `isReady` 2. Added an example of stacktrace in the KIP. PTAL. Thanks, Kamal On Mon, Nov 4, 2024 at 2:37 PM Federico Valeri wrote: > Hi Kamal, these changes make sense to me. Thanks. > > In this case, I wonder if "

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

use the optimization you authored in the PR) > > Thoughts? > > -- > Divij Vaidya > > > > On Mon, Nov 4, 2024 at 3:01 PM Divij Vaidya > wrote: > > > Thanks for patiently addressing all the comments. > > > > I will add the vote in the other thread.

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

p 16, 2024 at 7:20 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Bumping this thread for vote. PTAL. > > > > On Mon, Sep 9, 2024 at 2:01 PM Kamal Chandraprakash < > > kamal.chandraprak...@gmail.com> wrote: > > > > > H

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Hi all, If the review comments are addressed and KIP looks good, please submit your vote on the voting thread. Thanks, Kamal On Sat, Nov 2, 2024 at 7:52 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > I've opened a PR which adds an integration test > &l

[DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

ndling of remote-log components during server startup. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic Please take a look and suggest your thoughts. Thanks, Kamal

[jira] [Created] (KAFKA-17928) Make remote log manager thread-pool configs dynamic

Kamal Chandraprakash created KAFKA-17928: Summary: Make remote log manager thread-pool configs dynamic Key: KAFKA-17928 URL: https://issues.apache.org/jira/browse/KAFKA-17928 Project: Kafka

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

I've opened a PR which adds an integration test <https://github.com/apache/kafka/pull/17668/files#diff-a60b518846fc0f770164d55b6d7cd31e03a002514377578c80c6e22cc120af40R88> to show the impact of this change/KIP. PTAL. -- Kamal On Fri, Nov 1, 2024 at 3:31 PM Kamal Cha

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

://github.com/apache/kafka/blob/346fdbafc539bc48bb66eedae89a15e240007fd9/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1801 > . Is there a case where the parameter "offset" will not be equal to the > baseOffset of a segment? > > -- > Divij Vaidya > > > &

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

he file null check. Also, updated it in the KIP. Thanks, Kamal On Tue, Oct 29, 2024 at 8:47 PM Divij Vaidya wrote: > A few more points to discuss (please add to the KIP as well) > > 5. How are we determining the value of the TrxIndexEmpty field on segment > rotation? > > One o

[jira] [Created] (KAFKA-17830) Cover unit test for TBRLMM init failure cases

Kamal Chandraprakash created KAFKA-17830: Summary: Cover unit test for TBRLMM init failure cases Key: KAFKA-17830 URL: https://issues.apache.org/jira/browse/KAFKA-17830 Project: Kafka

[jira] [Created] (KAFKA-17797) Use DelayedFuturePurgatory for RemoteListOffsetsPurgatory

Kamal Chandraprakash created KAFKA-17797: Summary: Use DelayedFuturePurgatory for RemoteListOffsetsPurgatory Key: KAFKA-17797 URL: https://issues.apache.org/jira/browse/KAFKA-17797 Project

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bump for review. If the additional proposal looks good, I'll append them to the KIP. PTAL. New API in RLMM#nextRemoteLogSegmentMetadataWithTxnIndex -- Kamal On Sun, Oct 6, 2024 at 7:20 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Christo, > > T

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

/kafka/server/log/remote/metadata/storage/RemoteLogLeaderEpochState.java?L43 [2]: https://sourcegraph.com/github.com/apache/kafka/-/blob/core/src/main/scala/kafka/log/UnifiedLog.scala?L432 Thanks, Kamal On Fri, Oct 4, 2024 at 4:21 PM Christo Lolov wrote: > Heya, > > Apologies for the dela

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

d if they exist. [1]: https://sourcegraph.com/github.com/apache/kafka@trunk/-/blob/core/src/main/java/kafka/log/remote/RemoteLogManager.java?L1732 [2]: https://sourcegraph.com/github.com/apache/kafka@trunk/-/blob/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteIndexCache.java?L383 Thanks, K

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

Thank you all for your warm wishes! -- Kamal On Tue, Oct 1, 2024 at 12:11 AM Viktor Somogyi-Vass wrote: > Congrats Kamal! :) > > On Mon, Sep 30, 2024, 19:21 Matthias J. Sax wrote: > > > Congrats! > > > > On 9/30/24 6:59 AM, Yash Mayya wrote: > > > Congra

[jira] [Created] (KAFKA-17637) Invert the search for LIST_OFFSETS request for remote storage topic

Kamal Chandraprakash created KAFKA-17637: Summary: Invert the search for LIST_OFFSETS request for remote storage topic Key: KAFKA-17637 URL: https://issues.apache.org/jira/browse/KAFKA-17637

[jira] [Resolved] (KAFKA-15266) Static configs set for non primary synonyms are ignored for Log configs

[ https://issues.apache.org/jira/browse/KAFKA-15266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-15266. -- Resolution: Duplicate > Static configs set for non primary synonyms

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

set to -1. Could we clarify it in the KIP? Returning the timestamp will be useful to calculate the time-based consumer lag. The `max-timestamp` spec returns the maximum timestamp in the partition, it may not be the timestamp of the LSO/HW. Thanks, Kamal On Thu, Sep 19, 2024 at 5:39 PM Luke C

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bumping this thread for vote. PTAL. On Mon, Sep 9, 2024 at 2:01 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > I'd like to open voting for KIP-1058. This KIP improves the consumer > reading from remote storage when READ_COMMITTED isolation l

[jira] [Created] (KAFKA-17552) Handle LIST_OFFSETS request for max_timestamp when remote storage is enabled

Kamal Chandraprakash created KAFKA-17552: Summary: Handle LIST_OFFSETS request for max_timestamp when remote storage is enabled Key: KAFKA-17552 URL: https://issues.apache.org/jira/browse/KAFKA-17552

Re: [DISCUSS] Single broker failures causing offline partitions

ht be serving leadership for 1000s of partitions. And, finding the eligible replica manually, will increase the outage mitigation time as the producers/consumers are blocked when there are offline partitions. -- Kamal On Wed, Sep 11, 2024 at 3:57 AM Haruki Okada wrote: > Hi Martin, > > Tha

[VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

llecting+aborted+transactions> Thanks, Kamal

Re: [ANNOUNCE] New committer: Jeff Kim

Congrats, Jeff! On Mon, Sep 9, 2024 at 12:20 PM Chia-Ping Tsai wrote: > Congratulations, Jeff! thanks for you to bring the great new group > coordinator! > > Best, > Chia-Ping > > David Jacot 於 2024年9月9日 週一 下午2:44寫道： > > > Hi all, > > > > The PMC of Apache Kafka is pleased to announce a new Ka

Re: [VOTE] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, The vote is closed. The KIP has been accepted with 4 +1 binding votes (Satish, Luke, Christo, and Chia-Ping). Thank you all for the participation! -- Kamal On Thu, Aug 29, 2024 at 5:14 PM Chia-Ping Tsai wrote: > +1 (binding) > > On 2024/08/19 05:09:27 Kamal Chandraprak

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

dminClient will supply the "default.api.timeout.ms" as timeout in the ListOffsetsRequest and 2. Consumer will supply the "request.timeout.ms" as timeout in the ListOffsetsRequest PTAL. -- Kamal On Tue, Aug 27, 2024 at 2:32 PM Chia-Ping Tsai wrote: > hi Kamal, &

Re: [ANNOUNCE] New committer: Lianet Magrans

Congrats Lianet! On Thu, Aug 29, 2024 at 8:07 AM Luke Chen wrote: > Congratulations Lianet! > > Luke > > On Thu, Aug 29, 2024 at 9:54 AM Lianet M. wrote: > > > Thank you very much everyone! It truly takes the great shared knowledge > you > > all put out there with amazing reviews and discussion

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, This KIP needs one more binding vote to pass. Please review the KIP. Thanks! -- Kamal On Sat, Aug 24, 2024 at 8:11 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Jun, > > Thanks for the review! > > 1. purgatoryName = RemoteListOffsets >

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi Jun, Thanks for the review! 1. purgatoryName = RemoteListOffsets 2. We want to emit the metric at the partition level. Thanks for catching this! Updated the public interfaces section in the KIP! -- Kamal On Sat, Aug 24, 2024 at 3:14 AM Jun Rao wrote: > Hi, Kamal, > > Thanks fo

[VOTE] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi, I would like to open voting for KIP-1075. I've addressed the review comments in the discussion thread. Please vote if the proposal looks good. https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=315494171#content/view/315494171 Thanks, Kamal

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, If there are no more comments, I'll start a vote soon. Thanks, Kamal On Wed, Aug 14, 2024 at 5:05 PM Luke Chen wrote: > Hi Kamal, > > Thanks for the update. > LGTM. > > Luke > > On Wed, Aug 14, 2024 at 7:25 PM Kamal Chandraprakash < > kamal.chandr

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Hi all, > I saw we added some new configs/metrics. I have removed the recent changes to the public interfaces to limit the scope of the KIP to minimum. PTAL. Thanks, Kamal On Wed, Aug 14, 2024 at 9:58 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Luk

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

ode the num threads? > LC6 and LC7 Updated the KIP. Thanks, Kamal On Wed, Aug 14, 2024 at 8:05 AM Luke Chen wrote: > Hi Kamal, > > Thanks for the response. > > I saw we added some new configs/metrics. Comments: > > LC5: Do you think this is a commonly happened iss

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bumping this thread for KIP review! We can go for the simplest solution that is proposed in this KIP and it can be improved in the subsequent iteration. PTAL. Thanks, Kamal On Fri, Aug 2, 2024 at 11:42 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Divij, >

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

ed+remote+list+offsets+purgatory+to+make+LIST_OFFSETS+async> by adding more details. PTAL. Thanks, Kamal On Tue, Aug 13, 2024 at 2:03 PM Luke Chen wrote: > Hi Kamal, > > Thanks for the update. > LC1: I see. Thanks. > LC2: What I still don't u

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

out to the ListOffsetsRequest to the rejected alternatives section. -- Kamal

Re: [VOTE] KIP-950: Tiered Storage Disablement

t; to keep the default value as "false". -- Kamal On Thu, Aug 8, 2024 at 9:23 AM Satish Duggana wrote: > Thanks Kamal, and Luke for improving the earlier solution for KRaft. > > One minor comment I have is to change the config name from > "remote.log.copy.disable" to

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

Bumping this thread. Please take a look. On Fri, Aug 2, 2024 at 12:32 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > I would like to start a discussion thread on KIP-1075 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1075%3A+Int

[DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

he.org/confluence/display/KAFKA/KIP-1075%3A+Introduce+delayed+remote+list+offsets+purgatory+to+make+LIST_OFFSETS+async Draft PR: https://github.com/apache/kafka/pull/16602 Please take a look. Feedbacks and suggestions are welcome. Thanks, Kamal

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

rt compacted topics / historical compacted topics, we can omit this case. If this approach looks good, I can update the KIP with the details. -- Kamal On Tue, Jun 25, 2024 at 4:24 PM Divij Vaidya wrote: > Hi Kamal > > Thanks for the bump. I have been thinking about this passivel

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

wxy, Dung Ha, Edoardo Comar, Eduwer > >> Camacaro, Emanuele Sabellico, Erik van Oosten, Eugene Mitskevich, Fan > >> Yang, Federico Valeri, Fiore Mario Vitale, flashmouse, Florin Akermann, > >> Frederik Rouleau, Gantigmaa Selenge, Gaurav Narula, ghostspiders, > >>

Re: [VOTE] KIP-950: Tiered Storage Disablement

uot;, or "remote.storage.enable=true,remote.copy.disabled=true" for the desired topic, indicating the disablement of tiered storage. 2. Can we clarify in the public interface that the StopReplica v5, tiered_epoch, and tiered_state changes are required only for ZK mode and won't be implemented? Thank

Re: [VOTE] KIP-950: Tiered Storage Disablement

Correction: (2): Wait for all the remote segments to be deleted async due to breach by retention time (or) size, then set the `remote.storage.enable = false` and `remote.log.delete.on.disable = true`. This step is optional. On Thu, Jul 25, 2024 at 11:13 PM Kamal Chandraprakash

Re: [VOTE] KIP-950: Tiered Storage Disablement

. I don't know how the Kraft topic config propagation/validation works. -- Kamal On Thu, Jul 25, 2024 at 7:10 PM Chia-Ping Tsai wrote: > remote.storage.enable=false > remote.log.delete.on.disable=false (default) > If the topic config is set to this, or changed to this, we'll re

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

Hi Harry, Thanks for the updates! Yes, the proposed metric looks good. If the user runs the kafka-reassign-partitions script with throttle set, then the static throttle gets overwritten until the reassignment gets completed. Can you clarify this on the KIP? -- Kamal On Sun, Jul 14, 2024 at

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

ior? This would be separate from the new > MetadataVersion for the RPC/record changes. > > DA4: In the Motivation section, you mention the cluster expansion scenario. > For this scenario, is the expectation that the operator will cordon off the > existing full brokers so placements on

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

]:[log.dir.0],[broker.id.1]:[log.dir.1] This will provide a consistent view of the list of cordoned brokers in the cluster. -- Kamal On Wed, Jul 10, 2024 at 7:53 PM Mickael Maison wrote: > Hi Luke, > > 4. You're right this scenario can happen. In this case I think the > broker s

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

isr and can impact the cluster reliability. If we are going with this proposal, then we may also have to emit a metric to inform the administrator that the leader/follower replication quota is breached. -- Kamal On Thu, Jul 4, 2024 at 8:10 PM Harry Fallows wrote: > Hi everyone, > > Bum

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

Bump. Please review this proposal. On Mon, Jun 17, 2024 at 6:55 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Divij, > > Thanks for the review! Updated the KIP with 1, 2, 3, and 4 review > comments. > > > 4. Potential alternative - Instead of h

Re: [VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

Hi Federico, Thanks for the KIP! +1 from me. On Fri, Jun 21, 2024 at 5:47 PM Luke Chen wrote: > Hi Fede, > > Thanks for the KIP! > +1 from me. > > Luke > > On Fri, Jun 21, 2024 at 6:44 PM Federico Valeri > wrote: > > > Hi all, I'd like to kick off a vote on KIP-1057. > > > > Design doc: > > >

Re: [DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

e dump-logs tool with custom decoder > > - Running kafka-consumer.sh on the topic. > > > > ~Satish. > > > > > > ~Satish. > > > > > > > > On Mon, 17 Jun 2024 at 15:55, Federico Valeri > > wrote: > > > > > > Hi Kamal,

  1   2   3   >