englefly commented on code in PR #70400: URL: https://github.com/apache/doris/pull/70400#discussion_r4625616363
########## fe/fe-core/src/main/java/org/apache/doris/statistics/cache/StatisticsCacheLoader.java: ########## @@ -0,0 +1,53 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.statistics.cache; + +import com.github.benmanes.caffeine.cache.AsyncCacheLoader; + +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.Executor; +import java.util.concurrent.RejectedExecutionException; + +/** Keeps statistics SQL off the cache maintenance executor. */ +final class StatisticsCacheLoader<K, V> implements AsyncCacheLoader<K, V> { + private final BasicAsyncCacheLoader<K, V> delegate; + private final Executor executor; + + StatisticsCacheLoader(BasicAsyncCacheLoader<K, V> delegate, Executor executor) { + this.delegate = delegate; + this.executor = executor; + } + + @Override + public CompletableFuture<V> asyncLoad(K key, Executor ignored) { + // Throw synchronously on rejection, so Caffeine does not publish a pending entry. + return delegate.asyncLoad(key, executor); + } + + @Override + public CompletableFuture<V> asyncReload(K key, V oldValue, Executor ignored) { + try { + return asyncLoad(key, ignored); + } catch (RejectedExecutionException e) { Review Comment: **[P2] Back off automatic refresh after queue rejection.** Returning an already cancelled future keeps the old value eligible for immediate refresh. In a Caffeine 3.2.4 check, 1,000 consecutive hits on one refresh-eligible key produced 1,000 reload attempts. With this implementation, each hit allocates an async task and throws a rejection exception on the planning thread while the queue is full, adding CPU and allocation pressure during overload. Please add a short retry backoff for rejected automatic refreshes and cover repeated hits and recovery with a fake ticker. ########## fe/fe-core/src/main/java/org/apache/doris/statistics/cache/StatisticsCache.java: ########## @@ -243,16 +276,35 @@ public void updateColStatsCache(long ctlId, long dbId, long tblId, long idxId, S } public void refreshColStatsSync(long ctlId, long dbId, long tblId, long idxId, String colName) { - columnStatisticsCache.synchronous().refresh(new StatisticsCacheKey(ctlId, dbId, tblId, idxId, colName)); + refresh(columnStatisticsCache, new StatisticsCacheKey(ctlId, dbId, tblId, idxId, colName)); } public void refreshHistogramSync(long ctlId, long dbId, long tblId, long idxId, String colName) { - histogramCache.synchronous().refresh(new StatisticsCacheKey(ctlId, dbId, tblId, idxId, colName)); + refresh(histogramCache, new StatisticsCacheKey(ctlId, dbId, tblId, idxId, colName)); + } + + private <K, V> void refresh(AsyncLoadingCache<K, V> cache, K key) { + CompletableFuture<V> previous = cache.asMap().get(key); Review Comment: **[P3] Avoid triggering an automatic refresh while reading the previous future.** In Caffeine 3.2.4, `asMap().get(key)` can trigger refresh for a refresh-eligible entry. When the queue is full, that attempt is rejected and the following `synchronous().refresh(key)` submits again. I verified that these two calls each initiate a reload for the same key under rejection. Please use a read without the refresh side effect while preserving the conditional removal of the previous future. ########## fe/fe-core/src/test/java/org/apache/doris/statistics/cache/StatisticsCacheOverloadTest.java: ########## @@ -0,0 +1,174 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.statistics.cache; + +import org.apache.doris.common.FeConstants; +import org.apache.doris.statistics.model.ColumnStatistic; +import org.apache.doris.statistics.model.ColumnStatisticBuilder; +import org.apache.doris.statistics.model.PartitionColumnStatistic; + +import com.github.benmanes.caffeine.cache.AsyncLoadingCache; +import com.github.benmanes.caffeine.cache.Caffeine; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.time.Duration; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; + +@Timeout(30) +class StatisticsCacheOverloadTest { + private final ThreadPoolExecutor loads = pool(); + private final ThreadPoolExecutor maintenance = new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS, + new LinkedBlockingQueue<>(16), new ThreadPoolExecutor.AbortPolicy()); + private final CountDownLatch release = new CountDownLatch(1); + private final CountDownLatch drained = new CountDownLatch(1); + + private static ThreadPoolExecutor pool() { + return new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS, + new LinkedBlockingQueue<>(1), new ThreadPoolExecutor.AbortPolicy()); + } + + @AfterEach + void tearDown() throws InterruptedException { + release.countDown(); + loads.shutdownNow(); + maintenance.shutdownNow(); + Assertions.assertTrue(loads.awaitTermination(10, TimeUnit.SECONDS)); + Assertions.assertTrue(maintenance.awaitTermination(10, TimeUnit.SECONDS)); + } + + private void saturate() throws InterruptedException { + CountDownLatch started = new CountDownLatch(1); + loads.execute(() -> { + started.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }); + Assertions.assertTrue(started.await(10, TimeUnit.SECONDS)); + loads.execute(drained::countDown); + } + + private StatisticsCacheLoader<Integer, Integer> loader() { + return new StatisticsCacheLoader<>(new BasicAsyncCacheLoader<Integer, Integer>() { + @Override + protected Integer doLoad(Integer key) { + return key; + } + }, loads); + } + + @Test + void rejectedColdLoadsLeaveNoPendingEntriesAndCanRecover() throws Exception { + saturate(); + AsyncLoadingCache<Integer, Integer> cache = Caffeine.newBuilder() + .maximumSize(1).executor(maintenance).buildAsync(loader()); + for (int i = 0; i < 1000; i++) { + final int key = i; + Assertions.assertThrows(RejectedExecutionException.class, () -> cache.get(key)); + } + Assertions.assertTrue(cache.asMap().isEmpty()); + Assertions.assertEquals(1, loads.getQueue().size()); + release.countDown(); + Assertions.assertTrue(drained.await(10, TimeUnit.SECONDS)); + Assertions.assertEquals(7, cache.get(7).get(10, TimeUnit.SECONDS)); + } + + @Test + void pendingKeysAreDeduplicatedAndShareTheExecutorLimit() throws Exception { + CountDownLatch started = new CountDownLatch(1); + loads.execute(() -> { + started.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }); + Assertions.assertTrue(started.await(10, TimeUnit.SECONDS)); + AsyncLoadingCache<Integer, Integer> first = Caffeine.newBuilder() + .maximumSize(1).executor(maintenance).buildAsync(loader()); + AsyncLoadingCache<Integer, Integer> second = Caffeine.newBuilder() + .maximumSize(1).executor(maintenance).buildAsync(loader()); + CompletableFuture<Integer> pending = first.get(1); + Assertions.assertSame(pending, first.get(1)); + Assertions.assertThrows(RejectedExecutionException.class, () -> second.get(2)); + Assertions.assertTrue(second.asMap().isEmpty()); + release.countDown(); + Assertions.assertEquals(1, pending.get(10, TimeUnit.SECONDS)); + } + + @Test + void automaticRefreshRetainsOldValueAndDoesNotBlockMaintenance() throws Exception { + saturate(); + AtomicLong clock = new AtomicLong(); + AsyncLoadingCache<Integer, Integer> cache = Caffeine.newBuilder() + .maximumSize(1).ticker(clock::get).refreshAfterWrite(Duration.ofSeconds(1)) + .executor(maintenance).buildAsync(loader()); + cache.put(7, CompletableFuture.completedFuture(70)); + clock.set(TimeUnit.SECONDS.toNanos(2)); + Assertions.assertEquals(70, cache.get(7).get(10, TimeUnit.SECONDS)); + Assertions.assertEquals(70, cache.getIfPresent(7).join()); + CountDownLatch maintained = new CountDownLatch(1); + // A maintenance task can run while SQL loading remains saturated. + maintenance.execute(maintained::countDown); + Assertions.assertTrue(maintained.await(10, TimeUnit.SECONDS)); + } + + @Test + void statisticsLookupsAndPreheatDegradeWithoutPropagatingRejection() throws Exception { + saturate(); + StatisticsCache cache = new StatisticsCache(loads, maintenance); Review Comment: **[P3] Cover queue saturation through the production constructor.** The overload tests inject their own bounded pools with `AbortPolicy`. They would still pass if the production constructor stopped honoring `stats_fetch_queue_size` or switched to the blocking fixed-pool overload. Please add a test that fills the configured queue through the public construction path and verifies that a lookup returns unknown stats immediately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
