loserwang1024 commented on code in PR #3511:
URL: https://github.com/apache/fluss/pull/3511#discussion_r3458960225
##########
fluss-server/src/main/java/org/apache/fluss/server/RpcServiceBase.java:
##########
@@ -614,6 +623,15 @@ protected MetadataResponse processMetadataRequest(
long[] partitionIds = request.getPartitionsIds();
List<Long> partitionIdsNotExistsInCache = new ArrayList<>();
for (long partitionId : partitionIds) {
+ // Fast-path: throw immediately for partition IDs known to not
exist,
Review Comment:
I think negative cache check should NOT be placed before metadata cache
lookup.
Problem scenario — a race condition exists:
1. A new partition is being created; its ID has been assigned.
2. A client sends a metadata request, but the server's metadata cache hasn't
synced yet, and ZK may not have the entry written yet either.
3. Server queries ZK, finds nothing → calls markNonExistent(partitionId).
4. Partition creation completes; metadata cache is updated via ZK watch.
5. Subsequent requests are blocked by the negative cache and can never
retrieve the now-existing partition.
Suggested fix: Move the negative cache check to after the metadata cache
lookup but before the ZK query:
##########
fluss-server/src/main/java/org/apache/fluss/server/metadata/PartitionNegativeCache.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.server.metadata;
+
+import org.apache.fluss.annotation.VisibleForTesting;
+import org.apache.fluss.utils.clock.Clock;
+import org.apache.fluss.utils.clock.SystemClock;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.time.Duration;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * A thread-safe negative cache for partition IDs that are known to not exist
in ZooKeeper.
+ *
+ * <p>This cache helps reduce ZooKeeper pressure when clients repeatedly
request metadata for
+ * partitions that have been deleted (e.g., during hourly partition rotation).
Instead of querying
+ * ZK every time, we cache the "not exist" result and return it directly.
+ *
+ * <p>The cache uses access-time-based TTL: entries are evicted after a
configurable duration of no
+ * access. As long as clients keep asking for the same non-existent partition,
the cache entry stays
+ * alive and protects ZK.
+ */
+@ThreadSafe
+public class PartitionNegativeCache {
+
+ /** Default TTL for negative cache entries: 10 minutes of no access. */
+ private static final Duration DEFAULT_TTL = Duration.ofMinutes(10);
+
+ /**
+ * Cleanup interval: only run eviction check when at least this many
milliseconds have passed
+ * since the last cleanup.
+ */
+ private static final long CLEANUP_INTERVAL_MS = 60_000;
+
+ private final ConcurrentHashMap<Long, Long> cache;
Review Comment:
Why not useUse Guava Cache but implementation by yourself?
1. **Unreliable expiration cleanup** — maybeCleanup() is only called inside
markNonExistent(). If no new partitions are marked as non-existent for a long
time (in practice, markNonExistent is only triggered after a ZK query confirms
non-existence), expired entries will linger in memory indefinitely. While
isKnownNonExistent() lazily removes individual expired entries, it never
performs a full sweep.
2. **The project already has shaded Guava, so you can use it directly**:
```java
import org.apache.fluss.shaded.guava32.com.google.common.cache.Cache;
import org.apache.fluss.shaded.guava32.com.google.common.cache.CacheBuilder;
private final Cache<Long, Boolean> negativeCache = CacheBuilder.newBuilder()
.expireAfterAccess(10, TimeUnit.MINUTES)
.maximumSize(10000) // prevent OOM in extreme cases
.build();
```
3. Advantages of Guava Cache:
* Automatic eviction: access-time-based TTL without manual maintenance
* Bounded size: maximumSize prevents unbounded growth (the current
implementation has no size limit)
* Battle-tested thread safety: no need to roll your own CAS logic
@wuchong ,WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
