luohaiyang created KAFKA-21569:
----------------------------------
Summary: RemoteIndexCache drops eviction cleanup tasks while
restoring an oversized disk cache
Key: KAFKA-21569
URL: https://issues.apache.org/jira/browse/KAFKA-21569
Project: Kafka
Issue Type: Bug
Components: Tiered-Storage
Environment: Kafka trunk c3cbf35565aaf8754f266320f8f8a73a7189abd2;
Java 17; Caffeine 3.2.4; macOS arm64. Component-level reproduction using local
cached index files.
Reporter: luohaiyang
Assignee: luohaiyang
h3. Problem
When a tiered-storage broker restarts with existing cached indexes exceeding
remote.log.index.file.cache.total.size.bytes (for example, after reducing that
setting), RemoteIndexCache can leave evicted index files on disk until the next
restart.
The constructor [calls init() before
cleanerScheduler.startup()|https://github.com/apache/kafka/blob/c3cbf35565aaf8754f266320f8f8a73a7189abd2/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteIndexCache.java#L151-L158].
Restoring entries can trigger Caffeine eviction. The eviction listener
[renames index files to .deleted and schedules
cleanup|https://github.com/apache/kafka/blob/c3cbf35565aaf8754f266320f8f8a73a7189abd2/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteIndexCache.java#L248-L259],
but [KafkaScheduler ignores tasks submitted before
startup|https://github.com/apache/kafka/blob/c3cbf35565aaf8754f266320f8f8a73a7189abd2/server-common/src/main/java/org/apache/kafka/server/util/KafkaScheduler.java#L144-L165].
Starting the scheduler later does not replay those tasks.
h3. Reproduction
Verified against trunk c3cbf35565aaf8754f266320f8f8a73a7189abd2 with Java 17
and Caffeine 3.2.4. This is a component-level reproduction, not a full broker
integration test.
Run the following class with the storage module's runtime dependencies. It
creates 200 complete, valid index sets (8-byte offset index, 12-byte time
index, and empty transaction index), then restores them with a 20-byte cache
limit. This limit is valid and accommodates one entry. The null
RemoteStorageManager is not used on this restoration path.
{code:java}
package org.apache.kafka.storage.internals.log;
import org.apache.kafka.common.Uuid;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;
public class RemoteIndexCacheInitRepro {
public static void main(String[] args) throws Exception {
Path logDir = Files.createTempDirectory("remote-index-init-");
Path cacheDir =
Files.createDirectory(logDir.resolve(RemoteIndexCache.DIR_NAME));
for (int i = 0; i < 200; i++) {
String prefix = "0_" + Uuid.randomUuid();
Files.write(cacheDir.resolve(prefix + ".index"),
ByteBuffer.allocate(8).putInt(1).putInt(0).array());
Files.write(cacheDir.resolve(prefix + ".timeindex"),
ByteBuffer.allocate(12).putLong(1L).putInt(1).array());
Files.write(cacheDir.resolve(prefix + ".txnindex"), new byte[0]);
}
try (RemoteIndexCache cache = new RemoteIndexCache(20L, null,
logDir.toString())) {
cache.internalCache().cleanUp();
report("INITIAL", cacheDir, cache);
Thread.sleep(11_000L);
report("AFTER 11s", cacheDir, cache);
}
}
private static void report(String phase, Path dir, RemoteIndexCache cache)
throws Exception {
try (Stream<Path> paths = Files.list(dir)) {
long deleted = paths.filter(path ->
path.toString().endsWith(".deleted")).count();
System.out.println(phase + ": deleted files=" + deleted
+ ", scheduled=" + cache.expiredIdxPendingForDeletion());
}
}
}
{code}
Observed output from one run:
{code}
INITIAL: deleted files=597, scheduled=41
AFTER 11s: deleted files=474, scheduled=0
{code}
The default deletion delay is 10 seconds. Exact counts depend on when Caffeine
performs maintenance; this does not happen on every startup. Entries evicted
before scheduler startup have no pending cleanup task.
*Expected:* All evicted index files are removed after the cleanup delay.
*Actual:* Some .deleted files remain even though the cleanup queue is empty.
They occupy disk outside the active cache accounting until the next
initialization, which removes leftover .deleted files. The verified impact is
retained local cache files; this reproduction does not establish message loss
or a permanent memory leak.
h3. Suggested fix and validation
Make cleanup scheduling available before restoring cache entries, and close the
scheduler and opened indexes if initialization fails. Add a regression test
restoring an on-disk cache larger than the configured capacity.
As a minimal comparison, moving cleanerScheduler.startup() before init()
produced:
{code}
INITIAL: deleted files=597, scheduled=199
AFTER 11s: deleted files=0, scheduled=0
{code}
I plan to contribute a fix and regression test.
h3. Related issues
This differs from KAFKA-18787, which addressed .deleted/.tmp files already
present when initialization begins. Here, valid indexes are evicted during the
subsequent restoration pass and their newly submitted cleanup tasks are lost.
Scheduler-based cleanup was introduced by [PR #19462 /
KAFKA-17184|https://github.com/apache/kafka/pull/19462].
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)