Yuriy Malygin created KAFKA-21548:
-------------------------------------
Summary: LogManager#loadLog leaves kafka.log:type=Log gauges
registered for stray log directories
Key: KAFKA-21548
URL: https://issues.apache.org/jira/browse/KAFKA-21548
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 4.3.0
Reporter: Yuriy Malygin
h3. Problem
{{UnifiedLog}} registers its {{kafka.log:type=Log}} gauges ({{Size}},
{{NumLogSegments}}, {{LogStartOffset}}, {{LogEndOffset}},
{{RetentionSizeInPercent}}) in its constructor ({{UnifiedLog#newMetrics}}).
{{LogManager#loadLog}} always constructs a {{UnifiedLog}} for every directory
it finds, and then, for stray directories, neither tracks the log nor removes
its metrics:
* a directory whose name ends with {{-stray}} (logs "Loaded stray log");
* a log for which {{isStray}} returns true (the directory is renamed to
{{*-stray}}).
Since the log is not put into {{currentLogs}} / {{futureLogs}} /
{{logsToBeDeleted}}, nothing calls {{UnifiedLog#removeLogMetrics}} for it
later. The runtime path does clean up: {{asyncDelete}} goes through
{{removeLogAndMetrics}}. Only the startup path leaks.
The gauges also keep the {{UnifiedLog}} object reachable for the lifetime of
the JVM.
h3. Impact
The leaked gauges are tagged with the same {{topic}} and {{partition}} as a
live replica would be. If the broker also hosts a live replica of the same
partition (for example a stale directory left after a reassignment, next to the
current replica), both logs register gauges with an identical name. Yammer's
{{newGauge}} returns the gauge that is already registered for that name, so the
series can be bound to the stray log instead of the live one. We saw {{Size}}
reporting the size of the stray directory (about three times the size of the
live replica) on 4.3.0, which makes partition size dashboards and alerts wrong.
h3. Steps to reproduce
Unit test against trunk (b2ab2389a8), added to {{LogManagerTest}}:
{code:java}
@Test
public void testLoadStrayDirDoesNotLeaveLogMetrics() throws IOException {
File logDir = TestUtils.tempDirectory();
logManager.shutdown();
logManager = createLogManager(List.of(logDir));
String testTopic = "test-stray-metrics-topic";
TopicPartition tp = new TopicPartition(testTopic, 0);
logManager.getOrCreateLog(tp, Optional.of(Uuid.randomUuid()));
assertFalse(logMetricsOfTopic(testTopic).isEmpty());
// Mark the replica stray at runtime: renamed to *-stray, metrics removed.
logManager.asyncDelete(tp, false, false, true);
assertTrue(logMetricsOfTopic(testTopic).isEmpty());
// Simulate a broker restart that finds the *-stray directory.
File strayDir = Arrays.stream(logDir.listFiles())
.filter(f -> f.getName().startsWith(testTopic) &&
f.getName().endsWith(UnifiedLog.STRAY_DIR_SUFFIX))
.findFirst().orElseThrow();
logManager.loadLog(strayDir, true, Map.of(), Map.of(), LOG_CONFIG,
Map.of(), new ConcurrentHashMap<>(), l -> false);
assertTrue(logMetricsOfTopic(testTopic).isEmpty(), "leaked: " +
logMetricsOfTopic(testTopic));
}
private Set<MetricName> logMetricsOfTopic(String topic) {
return KafkaYammerMetrics.defaultRegistry().allMetrics().keySet().stream()
.filter(m -> m.getType().equals("Log") &&
m.getMBeanName().contains("topic=" + topic + ","))
.collect(Collectors.toSet());
}
{code}
Result:
{noformat}
AssertionFailedError: leaked: [
kafka.log:type=Log,name=RetentionSizeInPercent,topic=test-stray-metrics-topic,partition=0,
kafka.log:type=Log,name=LogStartOffset,topic=test-stray-metrics-topic,partition=0,
kafka.log:type=Log,name=Size,topic=test-stray-metrics-topic,partition=0,
kafka.log:type=Log,name=NumLogSegments,topic=test-stray-metrics-topic,partition=0,
kafka.log:type=Log,name=LogEndOffset,topic=test-stray-metrics-topic,partition=0
]
==> expected: <true> but was: <false>
{noformat}
h3. Suggested fix
Call {{log.removeLogMetrics()}} in the two stray branches of
{{LogManager#loadLog}} (the {{-stray}} suffix branch and the {{isStray}}
branch). The {{-delete}} branch looks similar: the log is queued with
{{addLogToBeDeleted}} and {{deleteLogs}} calls {{UnifiedLog#delete}}, which
does not remove metrics either. I have not verified that one.
h3. Related
KAFKA-20567 (skip loading {{UnifiedLog}} for stray replicas during startup)
would avoid creating the log, and so its metrics, for replicas detected as
stray from the metadata image. It does not cover directories that already end
with {{-stray}}, and it is a different change from this report.
KAFKA-2233, KAFKA-3572, KAFKA-10282: earlier reports of log metrics outliving
their log, all in the delete path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)