Yuriy Malygin created KAFKA-21548:
-------------------------------------

             Summary: LogManager#loadLog leaves kafka.log:type=Log gauges 
registered for stray log directories
                 Key: KAFKA-21548
                 URL: https://issues.apache.org/jira/browse/KAFKA-21548
             Project: Kafka
          Issue Type: Bug
          Components: core
    Affects Versions: 4.3.0
            Reporter: Yuriy Malygin


h3. Problem

{{UnifiedLog}} registers its {{kafka.log:type=Log}} gauges ({{Size}}, 
{{NumLogSegments}}, {{LogStartOffset}}, {{LogEndOffset}}, 
{{RetentionSizeInPercent}}) in its constructor ({{UnifiedLog#newMetrics}}). 
{{LogManager#loadLog}} always constructs a {{UnifiedLog}} for every directory 
it finds, and then, for stray directories, neither tracks the log nor removes 
its metrics:

* a directory whose name ends with {{-stray}} (logs "Loaded stray log");
* a log for which {{isStray}} returns true (the directory is renamed to 
{{*-stray}}).

Since the log is not put into {{currentLogs}} / {{futureLogs}} / 
{{logsToBeDeleted}}, nothing calls {{UnifiedLog#removeLogMetrics}} for it 
later. The runtime path does clean up: {{asyncDelete}} goes through 
{{removeLogAndMetrics}}. Only the startup path leaks.

The gauges also keep the {{UnifiedLog}} object reachable for the lifetime of 
the JVM.

h3. Impact

The leaked gauges are tagged with the same {{topic}} and {{partition}} as a 
live replica would be. If the broker also hosts a live replica of the same 
partition (for example a stale directory left after a reassignment, next to the 
current replica), both logs register gauges with an identical name. Yammer's 
{{newGauge}} returns the gauge that is already registered for that name, so the 
series can be bound to the stray log instead of the live one. We saw {{Size}} 
reporting the size of the stray directory (about three times the size of the 
live replica) on 4.3.0, which makes partition size dashboards and alerts wrong.

h3. Steps to reproduce

Unit test against trunk (b2ab2389a8), added to {{LogManagerTest}}:

{code:java}
@Test
public void testLoadStrayDirDoesNotLeaveLogMetrics() throws IOException {
    File logDir = TestUtils.tempDirectory();
    logManager.shutdown();
    logManager = createLogManager(List.of(logDir));

    String testTopic = "test-stray-metrics-topic";
    TopicPartition tp = new TopicPartition(testTopic, 0);
    logManager.getOrCreateLog(tp, Optional.of(Uuid.randomUuid()));
    assertFalse(logMetricsOfTopic(testTopic).isEmpty());

    // Mark the replica stray at runtime: renamed to *-stray, metrics removed.
    logManager.asyncDelete(tp, false, false, true);
    assertTrue(logMetricsOfTopic(testTopic).isEmpty());

    // Simulate a broker restart that finds the *-stray directory.
    File strayDir = Arrays.stream(logDir.listFiles())
            .filter(f -> f.getName().startsWith(testTopic) && 
f.getName().endsWith(UnifiedLog.STRAY_DIR_SUFFIX))
            .findFirst().orElseThrow();
    logManager.loadLog(strayDir, true, Map.of(), Map.of(), LOG_CONFIG, 
Map.of(), new ConcurrentHashMap<>(), l -> false);

    assertTrue(logMetricsOfTopic(testTopic).isEmpty(), "leaked: " + 
logMetricsOfTopic(testTopic));
}

private Set<MetricName> logMetricsOfTopic(String topic) {
    return KafkaYammerMetrics.defaultRegistry().allMetrics().keySet().stream()
            .filter(m -> m.getType().equals("Log") && 
m.getMBeanName().contains("topic=" + topic + ","))
            .collect(Collectors.toSet());
}
{code}

Result:

{noformat}
AssertionFailedError: leaked: [
  
kafka.log:type=Log,name=RetentionSizeInPercent,topic=test-stray-metrics-topic,partition=0,
  
kafka.log:type=Log,name=LogStartOffset,topic=test-stray-metrics-topic,partition=0,
  kafka.log:type=Log,name=Size,topic=test-stray-metrics-topic,partition=0,
  
kafka.log:type=Log,name=NumLogSegments,topic=test-stray-metrics-topic,partition=0,
  
kafka.log:type=Log,name=LogEndOffset,topic=test-stray-metrics-topic,partition=0 
]
 ==> expected: <true> but was: <false>
{noformat}

h3. Suggested fix

Call {{log.removeLogMetrics()}} in the two stray branches of 
{{LogManager#loadLog}} (the {{-stray}} suffix branch and the {{isStray}} 
branch). The {{-delete}} branch looks similar: the log is queued with 
{{addLogToBeDeleted}} and {{deleteLogs}} calls {{UnifiedLog#delete}}, which 
does not remove metrics either. I have not verified that one.

h3. Related

KAFKA-20567 (skip loading {{UnifiedLog}} for stray replicas during startup) 
would avoid creating the log, and so its metrics, for replicas detected as 
stray from the metadata image. It does not cover directories that already end 
with {{-stray}}, and it is a different change from this report.

KAFKA-2233, KAFKA-3572, KAFKA-10282: earlier reports of log metrics outliving 
their log, all in the delete path.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to