[DISCUSS] KIP-1151: Minimal Movement Replica Balancing Algorithm for Reassignment

Hi Kafka Community, I would like to start a discussion on a new KIP proposal: KIP-1151: Minimal movement replica balancing algorithm for reassignment Motivation Kafka clusters require rebalancing of topic replicas after horizontal scaling to evenly distribute the load. However, the current appro

Re: [DISCUSS] KIP-1151: Minimal Movement Replica Balancing Algorithm for Reassignment

implementation and submitted the PR. The PR also includes a comprehensive set of test cases to help verify the correctness of the algorithm. I would really appreciate it if anyone could take some time to review the KIP and the PR and share your feedback or suggestions. Best regards Jialun Peng 在 2025年3月29日星期

Re: [DISCUSS] KIP-1151: Minimal Movement Replica Balancing Algorithm for Reassignment

Thank you for your reply The proposed load balancer is designed to distribute partitions uniformly across all brokers, assuming homogeneous hardware and similar partition workloads (i.e., CPU, network, and disk I/O demands are consistent). Under these conditions, it guarantees globally balanced pa

[DISCUSS] KIP-1151: Minimal Movement Replica Balancing Algorithm for Reassignment

Hi Kafka Community, I would like to start a discussion on a new KIP proposal: KIP-1151: Minimal movement replica balancing algorithm for reassignment Motivation Kafka clusters require rebalancing of topic replicas after horizontal scaling to evenly distribute the load. However, the current appro

Request for contributor permissions

Hi Kafka community, I'd like to contribute to Apache Kafka. My ASF accounts are: - Wiki ID: pengjialun - Jira ID: pengjialun Please grant me the necessary permissions. Thanks, Jialun Peng

[DISCUSS] KIP-1194: Optimize Replica Assignment for Broker Load Balance in Uneven Rack Configurations

Peng

Re: [DISCUSS] KIP-1201: Add disk threshold strategy to prevent disk full failure

In most use cases, Kafka serves as a messaging middleware where messages that have already been consumed are typically no longer needed and can be safely deleted. Therefore, I propose enhancing the threshold strategy with an automatic deletion feature: When a broker's disk usage reaches 95%, it sh

Fwd: How to fetch offset for all the partitions of one topic through single KafkaClient function call (0.10.x)?

() function? Thanks, Martin -- Forwarded message -- From: Martin Peng Date: 2017-07-06 17:36 GMT-07:00 Subject: How to fetch offset for all the partitions of one topic through single KafkaClient function call (0.10.x)? To: us...@kafka.apache.org Hi, I am using Kafka client 0.10.2

ExpiringCredentialRefreshingLogin causing admin close to block forever

Hi all, I am looking for a bit of guidance on an issue I am seeing in a Structured Streaming (Spark Streaming) query being blocked due to a thread in ExpiringCredentialRefreshingLogin not being able to shutdown. The reason why the streaming query is blocked from making progress is because the kaf

ARM support for Kafka

//twitter.com/ijuma> @gchenke <https://twitter.com/gchenke> @damianguy <https://twitter.com/damianguy> @MatthiasJSax <https://twitter.com/MatthiasJSax> @vahidh <https://twitter.com/vahidh> @rhauch <https://twitter.com/rhauch> @davidjacot <https://twitter.com/davidjacot> What do you think about kafka's support for arm ci? Thanks Lei Peng

Re: ARM support for Kafka

ll/9872]. I don't have permission to configure it to work. Can you help me? Thanks Lei Peng Peng Lei 于2021年1月18日周一 上午10:43写道： > Hi Kafka > I'm pushing kafka to support arm ci. I think this is a very > meaningful work because of it helps kafka get better. If Kafka support

Kafka support Arm CI

://issues.apache.org/jira/browse/KAFKA-10759?jql=project%20%3D%20KAFKA%20AND%20text%20~%20arm Thanks By Lei Peng

Re: Kafka support Arm CI

fka's support for arm ci ? I can give arm ci of Kafka my full support. Now I'm trying to submit a PR to support this work.[https://github.com/apache/kafka/pull/9872] Thanks By Lei Peng Peng Lei 于2020年12月15日周二 上午10:40写道： > Hi Kafka: > > Recently we met a problem that we have

[jira] [Created] (KAFKA-6717) TopicPartition Assined twice to a consumer group for different consumers

Yuancheng PENG created KAFKA-6717: - Summary: TopicPartition Assined twice to a consumer group for different consumers Key: KAFKA-6717 URL: https://issues.apache.org/jira/browse/KAFKA-6717 Project

[jira] [Created] (KAFKA-6103) one broker appear to dead lock after running serval hours with a fresh cluster

Peyton Peng created KAFKA-6103: -- Summary: one broker appear to dead lock after running serval hours with a fresh cluster Key: KAFKA-6103 URL: https://issues.apache.org/jira/browse/KAFKA-6103 Project

[jira] [Created] (KAFKA-4268) huge server.logs during the error frequently happen "Message format version for partition 200 not found"

Peyton Peng created KAFKA-4268: -- Summary: huge server.logs during the error frequently happen "Message format version for partition 200 not found" Key: KAFKA-4268 URL: https://issues.apache.org/jira/br

[jira] [Created] (KAFKA-16102) about DynamicListenerConfig, the dynamic modification of the listener's port or IP does not take effect.

Jialun Peng created KAFKA-16102: --- Summary: about DynamicListenerConfig, the dynamic modification of the listener's port or IP does not take effect. Key: KAFKA-16102 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (KAFKA-19048) Minimal Movement Replica Balancing algorithm

Jialun Peng created KAFKA-19048: --- Summary: Minimal Movement Replica Balancing algorithm Key: KAFKA-19048 URL: https://issues.apache.org/jira/browse/KAFKA-19048 Project: Kafka Issue Type

[jira] [Created] (KAFKA-19507) Optimize Replica Assignment for Broker Load Balance in Uneven Rack Configurations

Jialun Peng created KAFKA-19507: --- Summary: Optimize Replica Assignment for Broker Load Balance in Uneven Rack Configurations Key: KAFKA-19507 URL: https://issues.apache.org/jira/browse/KAFKA-19507