[
https://issues.apache.org/jira/browse/KAFKA-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-12258.
-------------------------------------
Resolution: Fixed
> Change the BatchAccumulator to split records into batches
> ---------------------------------------------------------
>
> Key: KAFKA-12258
> URL: https://issues.apache.org/jira/browse/KAFKA-12258
> Project: Kafka
> Issue Type: Sub-task
> Components: controller, core
> Affects Versions: 2.8.0
> Reporter: Alok Nikhil
> Assignee: Jose Armando Garcia Sancio
> Priority: Major
> Labels: kip-500
>
> Modify the `BatchAccumulator.append contract` to support splitting a batch of
> records whose size is greater than the maximum allowed size (of 1048576
> currently) into batches to avoid RaftClient failures such as
> {code:java}
> leader=0, leaderEpoch=0, partitionEpoch=0) at version 0),
> ApiMessageAndVersion(TopicRecord(name='topic-BEHRW9999',
> topicId=6cRudOGO3yqlsu48RwyPSw) at version 0),
> ApiMessageAndVersion(PartitionRecord(partitionId=0,
> topicId=6cRudOGO3yqlsu48RwyPSw, replicas=[1, 2, 0], isr=[1, 2, 0],
> removingReplicas=null, addingReplicas=null, leader=1, leaderEpoch=0,
> partitionEpoch=0) at version 0)] is 1088890, which exceeds the maximum
> allowed batch size of 1048576 Jan 30 00:13:40 ip-10-0-0-254
> kafka-server-start.sh[633637]: at
> org.apache.kafka.raft.internals.BatchAccumulator.append(BatchAccumulator.java:110)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.raft.KafkaRaftClient.scheduleAppend(KafkaRaftClient.java:1885)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.raft.metadata.MetaLogRaftShim.scheduleWrite(MetaLogRaftShim.java:60)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:406)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.common.utils.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:117)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.common.utils.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:192)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> org.apache.kafka.common.utils.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:165)
> Jan 30 00:13:40 ip-10-0-0-254 kafka-server-start.sh[633637]: at
> java.base/java.lang.Thread.run(Thread.java:834) Jan 30 00:13:40 ip-10-0-0-254
> kafka-server-start.sh[633637]: [2021-01-30 00:13:40,277] INFO [Controller
> 3000] Reverting to snapshot 2232 (org.apache.kafka.timeline.SnapshotRegistry)
> {code}
> *Example use-case*: Creating 10,000 topics in a single API call
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
