luohaiyang created KAFKA-21569:
----------------------------------

             Summary: RemoteIndexCache drops eviction cleanup tasks while 
restoring an oversized disk cache
                 Key: KAFKA-21569
                 URL: https://issues.apache.org/jira/browse/KAFKA-21569
             Project: Kafka
          Issue Type: Bug
          Components: Tiered-Storage
         Environment: Kafka trunk c3cbf35565aaf8754f266320f8f8a73a7189abd2; 
Java 17; Caffeine 3.2.4; macOS arm64. Component-level reproduction using local 
cached index files.
            Reporter: luohaiyang
            Assignee: luohaiyang


h3. Problem

When a tiered-storage broker restarts with existing cached indexes exceeding 
remote.log.index.file.cache.total.size.bytes (for example, after reducing that 
setting), RemoteIndexCache can leave evicted index files on disk until the next 
restart.

The constructor [calls init() before 
cleanerScheduler.startup()|https://github.com/apache/kafka/blob/c3cbf35565aaf8754f266320f8f8a73a7189abd2/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteIndexCache.java#L151-L158].
 Restoring entries can trigger Caffeine eviction. The eviction listener 
[renames index files to .deleted and schedules 
cleanup|https://github.com/apache/kafka/blob/c3cbf35565aaf8754f266320f8f8a73a7189abd2/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteIndexCache.java#L248-L259],
 but [KafkaScheduler ignores tasks submitted before 
startup|https://github.com/apache/kafka/blob/c3cbf35565aaf8754f266320f8f8a73a7189abd2/server-common/src/main/java/org/apache/kafka/server/util/KafkaScheduler.java#L144-L165].
 Starting the scheduler later does not replay those tasks.

h3. Reproduction

Verified against trunk c3cbf35565aaf8754f266320f8f8a73a7189abd2 with Java 17 
and Caffeine 3.2.4. This is a component-level reproduction, not a full broker 
integration test.

Run the following class with the storage module's runtime dependencies. It 
creates 200 complete, valid index sets (8-byte offset index, 12-byte time 
index, and empty transaction index), then restores them with a 20-byte cache 
limit. This limit is valid and accommodates one entry. The null 
RemoteStorageManager is not used on this restoration path.

{code:java}
package org.apache.kafka.storage.internals.log;

import org.apache.kafka.common.Uuid;

import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;

public class RemoteIndexCacheInitRepro {
    public static void main(String[] args) throws Exception {
        Path logDir = Files.createTempDirectory("remote-index-init-");
        Path cacheDir = 
Files.createDirectory(logDir.resolve(RemoteIndexCache.DIR_NAME));
        for (int i = 0; i < 200; i++) {
            String prefix = "0_" + Uuid.randomUuid();
            Files.write(cacheDir.resolve(prefix + ".index"),
                ByteBuffer.allocate(8).putInt(1).putInt(0).array());
            Files.write(cacheDir.resolve(prefix + ".timeindex"),
                ByteBuffer.allocate(12).putLong(1L).putInt(1).array());
            Files.write(cacheDir.resolve(prefix + ".txnindex"), new byte[0]);
        }
        try (RemoteIndexCache cache = new RemoteIndexCache(20L, null, 
logDir.toString())) {
            cache.internalCache().cleanUp();
            report("INITIAL", cacheDir, cache);
            Thread.sleep(11_000L);
            report("AFTER 11s", cacheDir, cache);
        }
    }

    private static void report(String phase, Path dir, RemoteIndexCache cache) 
throws Exception {
        try (Stream<Path> paths = Files.list(dir)) {
            long deleted = paths.filter(path -> 
path.toString().endsWith(".deleted")).count();
            System.out.println(phase + ": deleted files=" + deleted
                + ", scheduled=" + cache.expiredIdxPendingForDeletion());
        }
    }
}
{code}

Observed output from one run:
{code}
INITIAL: deleted files=597, scheduled=41
AFTER 11s: deleted files=474, scheduled=0
{code}

The default deletion delay is 10 seconds. Exact counts depend on when Caffeine 
performs maintenance; this does not happen on every startup. Entries evicted 
before scheduler startup have no pending cleanup task.

*Expected:* All evicted index files are removed after the cleanup delay.

*Actual:* Some .deleted files remain even though the cleanup queue is empty. 
They occupy disk outside the active cache accounting until the next 
initialization, which removes leftover .deleted files. The verified impact is 
retained local cache files; this reproduction does not establish message loss 
or a permanent memory leak.

h3. Suggested fix and validation

Make cleanup scheduling available before restoring cache entries, and close the 
scheduler and opened indexes if initialization fails. Add a regression test 
restoring an on-disk cache larger than the configured capacity.

As a minimal comparison, moving cleanerScheduler.startup() before init() 
produced:
{code}
INITIAL: deleted files=597, scheduled=199
AFTER 11s: deleted files=0, scheduled=0
{code}

I plan to contribute a fix and regression test.

h3. Related issues

This differs from KAFKA-18787, which addressed .deleted/.tmp files already 
present when initialization begins. Here, valid indexes are evicted during the 
subsequent restoration pass and their newly submitted cleanup tasks are lost. 
Scheduler-based cleanup was introduced by [PR #19462 / 
KAFKA-17184|https://github.com/apache/kafka/pull/19462].



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to