[
https://issues.apache.org/jira/browse/POOL-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480759#comment-17480759
]
Patrick Barry commented on POOL-404:
------------------------------------
Here is the test I extracted. This definitely demonstrates the confusing
logging statement with incrementing pool #'s, but surprisingly does not show
the evictor thread still running. Trying to figure out what is different with
our mainline code.
Lettuce 6.1.6Release, and latest generic pool 2.11.1
Startup a local running redis server.
{code:java}
package com.moproblems.core.cache.redis.client;
import io.lettuce.core.ClientOptions;
import io.lettuce.core.RedisClient;
import io.lettuce.core.RedisURI;
import io.lettuce.core.api.StatefulRedisConnection;
import io.lettuce.core.resource.DefaultClientResources;
import io.lettuce.core.resource.Delay;
import io.lettuce.core.resource.DirContextDnsResolver;
import io.lettuce.core.support.ConnectionPoolSupport;
import io.netty.util.HashedWheelTimer;
import io.netty.util.Timer;
import io.netty.util.concurrent.DefaultThreadFactory;
import org.apache.commons.pool2.impl.GenericObjectPool;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import static java.time.Duration.*;
import static java.util.concurrent.TimeUnit.MILLISECONDS;
public class PoolProblem {
static Logger logger = LoggerFactory.getLogger(PoolProblem.class);
public static void main(String[] args) throws InterruptedException {
RedisURI elasticacheUri = RedisURI.Builder.redis("localhost", 6379)
.build();
Timer timer = new HashedWheelTimer(new
DefaultThreadFactory("my-lettuce-timer", true));
RedisClient redisClient =
RedisClient.create(DefaultClientResources.builder()
.ioThreadPoolSize(8)
.computationThreadPoolSize(8)
.dnsResolver(new DirContextDnsResolver())
.reconnectDelay(Delay.exponential(0, 1500, MILLISECONDS, 2))
.timer(timer)
.build(), elasticacheUri);
redisClient.setOptions(ClientOptions.builder()
.autoReconnect(true)
.cancelCommandsOnReconnectFailure(true)
.disconnectedBehavior(ClientOptions.DisconnectedBehavior.REJECT_COMMANDS)
.pingBeforeActivateConnection(true)
.requestQueueSize(10)
.build());
GenericObjectPoolConfig<StatefulRedisConnection<String, String>> config
= new GenericObjectPoolConfig<>();
config.setMinIdle(2);
config.setMaxTotal(10);
config.setMaxIdle(10);
config.setMaxWait(ofSeconds(1));
config.setMinEvictableIdleTime(ofMinutes(5));
config.setTestOnBorrow(true);
config.setTimeBetweenEvictionRuns(ofMinutes(5));
List<Callable<String>> tasks = new ArrayList<>();
GenericObjectPool<StatefulRedisConnection<String, String>> pool =
ConnectionPoolSupport.createGenericObjectPool(redisClient::connect, config);
for (int i = 0; i < 5; i++) {
ExecutorService execService = Executors.newFixedThreadPool(25);
for (int count = 0; count < 100; count++) {
Callable<String> c = () -> {
try (StatefulRedisConnection<String, String> connection =
pool.borrowObject()) {
logger.info("Running test on pool {}, connection {}",
System.identityHashCode(pool), System.identityHashCode(connection));
Thread.sleep(100);
return connection.sync().set("key", "100");
} catch (Exception e) {
e.printStackTrace();
}
return "{}";
};
tasks.add(c);
}
List<Future<String>> results = execService.invokeAll(tasks);
logger.info("Completed {} tests", results.size());
execService.shutdownNow();
}
logger.info("Done");
timer.stop();
pool.close();
redisClient.shutdown(ZERO, Duration.ofSeconds(20));
List<String> threadsStillAlive = new ArrayList<>();
for (Thread t : Thread.getAllStackTraces().keySet()) {
if (t.getName().contains("lettuce-") ||
t.getName().contains("pool")) {
threadsStillAlive.add(t.getName());
}
}
logger.info("Threads still alive after attempted shutdown: {}",
threadsStillAlive);
}
}
{code}
> No way to close evictor thread
> ------------------------------
>
> Key: POOL-404
> URL: https://issues.apache.org/jira/browse/POOL-404
> Project: Commons Pool
> Issue Type: Bug
> Affects Versions: 2.11.1
> Reporter: Patrick Barry
> Priority: Major
>
> Using GenericObjectPool<StatefulRedisConnection<String, String>> to help with
> lettuce client/redis connection management. I have everything shutting down
> cleanly except the commons-pool-evictor. I see this problem has been
> reported many times in the past, but all the changes are still not allowing
> this thread to shut down cleanly on close. I am using version 2.11.1. I
> have tried to code around this issue, but because EvictionTimer.java is so
> locked down, there is very little that can done to change the behavior of how
> this class interacts with GenericObjectPool.
> For this thread to shutdown, the taskMap has to be empty, which it never is
> in my case. So even though we call close() on the pool, this class fails to
> shutdown the embedded executor because it thinks it has more tasks.
> Looking at this code, it did remove 1 entry from taskMap, but we had many
> more in that map. Is there a way to clear this map, so it will allow this
> thread/executor to shutdown?
> {code:java}
> static synchronized void cancel(final BaseGenericObjectPool<?>.Evictor
> evictor, final Duration timeout,
> final boolean restarting) {
> if (evictor != null) {
> evictor.cancel(); //why does this not interrupt!?
> remove(evictor);
> }
> if (!restarting && executor != null && taskMap.isEmpty()) { //<-- How do
> you force taskMap to be empty!?
> executor.shutdown();
> try {
> executor.awaitTermination(timeout.toMillis(),
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException e) {
> // Swallow
> // Significant API changes would be required to propagate this
> }
> executor.setCorePoolSize(0);
> executor = null;
> }
> }{code}
> }
> I had all these entries in the taskMap when trying to shut down:
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@73d4066e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@3c69362a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2412a42b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@45404d5
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@29138d3a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5cbe2654
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6dbcf214
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@496a31da
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@7c251f90
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@51841ac6
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@5ba26eb0
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@435e60ff
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@17d32e9b
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@66f0548d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@2e6f610d
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@1e86a5a7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@10afe71a
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@741f8dbe
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@212dfd39
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@a2ddf26
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@49ede9c7
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@65d57e4e
> org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor@6daf7d37
> *Method calls:*
> pool.close() ->
> stopEvictor(); ->
> startEvictor(Duration.ofMillis(-1L)); ->
> EvictionTimer.cancel(evictor, evictorShutdownTimeoutDuration, false); ->
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
