[
https://issues.apache.org/jira/browse/KAFKA-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Lin updated KAFKA-6175:
----------------------------
Description:
Currently when we shutdown a broker, we will call AbstractIndex.resize() for
all segments on the broker, regardless of whether the log segment is active or
not. AbstractIndex.resize() incurs raf.setLength(), which is expensive because
it accesses disks. If we do a threaddump during either LogManger.shutdown() or
LogManager.loadLogs(), most threads are in RUNNABLE state at
java.io.RandomAccessFile.setLength().
This patch intends to speed up broker startup and shutdown time by skipping
AbstractIndex.resize() for inactive log segments.
Here is the time of LogManager.shutdown() in various settings. In all these
tests, broker has roughly 6k partitions and 19k segments.
- If broker does not have this patch and KAFKA-6172, LogManager.shutdown()
takes 69 seconds
- If broker has KAFKA-6172 but not this patch, LogManager.shutdown() takes 21
seconds.
- If broker has KAFKA-6172 and this patch, LogManager.shutdown() takes 1.6
seconds.
was:
Currently when we shutdown a broker, we will call AbstractIndex.resize() for
all segments on the broker, regardless of whether the log segment is active or
not. AbstractIndex.resize() incurs raf.setLength(), which is expensive because
it accesses disks. If we do a threaddump during either LogManger.shutdown() or
LogManager.loadLogs(), most threads are in RUNNABLE state at
java.io.RandomAccessFile.setLength().
This patch intends to speed up broker startup and shutdown time by skipping
AbstractIndex.resize() for inactive log segments.
> AbstractIndex should cache index file to avoid unnecessary disk access during
> resize()
> --------------------------------------------------------------------------------------
>
> Key: KAFKA-6175
> URL: https://issues.apache.org/jira/browse/KAFKA-6175
> Project: Kafka
> Issue Type: Improvement
> Reporter: Dong Lin
> Assignee: Dong Lin
> Fix For: 1.0.1
>
>
> Currently when we shutdown a broker, we will call AbstractIndex.resize() for
> all segments on the broker, regardless of whether the log segment is active
> or not. AbstractIndex.resize() incurs raf.setLength(), which is expensive
> because it accesses disks. If we do a threaddump during either
> LogManger.shutdown() or LogManager.loadLogs(), most threads are in RUNNABLE
> state at java.io.RandomAccessFile.setLength().
> This patch intends to speed up broker startup and shutdown time by skipping
> AbstractIndex.resize() for inactive log segments.
> Here is the time of LogManager.shutdown() in various settings. In all these
> tests, broker has roughly 6k partitions and 19k segments.
> - If broker does not have this patch and KAFKA-6172, LogManager.shutdown()
> takes 69 seconds
> - If broker has KAFKA-6172 but not this patch, LogManager.shutdown() takes 21
> seconds.
> - If broker has KAFKA-6172 and this patch, LogManager.shutdown() takes 1.6
> seconds.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
