[
https://issues.apache.org/jira/browse/CASSANDRA-15170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883755#comment-16883755
]
Alex Petrov commented on CASSANDRA-15170:
-----------------------------------------
[~jmeredithco] thank you for the patch. I have several minor nits:
* {{numClusterNodes}} seems to be unused in {{ResourceLeakTest}}
* I'm not 100% sure why we need changes to logging to remove instance IDs
from some log messages and adding {{INSTANCE}} prefix to logger names.
* We have a [shutdown
hook|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L641],
which should be using the instance class loader, but because we're running it
after the instance class loader is already shut down, we get the exception [1].
The error message it throws is unclear, and I would probably override
{InstanceClassLoader#close} to make it more obvious what's going on: if class
loader is already closed, we should thrown with a message that it's been
already shut down. In addition to this, I'd probably avoid adding a JVM
shutdown hook, and close this explicitly. I think this was existing prior to
this patch.
* On multiple runs, I've also seen the exceptions [2] and [3]. I'm not
claiming that this patch has caused them.
* We're seemingly logging each log message twice right now. I think this is
also pre-existing, and this can be resolved by using only one of the two
console appenders.
[1]
{code}
java.lang.NoClassDefFoundError:
org/apache/cassandra/utils/logging/LoggingSupportFactory
at
org.apache.cassandra.service.StorageService$1.runMayThrow(StorageService.java:638)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.cassandra.utils.logging.LoggingSupportFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at
org.apache.cassandra.distributed.impl.InstanceClassLoader.loadClassInternal(InstanceClassLoader.java:95)
at
org.apache.cassandra.distributed.impl.InstanceClassLoader.loadClass(InstanceClassLoader.java:84)
... 4 more
{code}
[2]
{code}
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:58)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:162)
at
org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:907)
at
org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:873)
at
org.apache.cassandra.schema.SchemaKeyspace.lambda$flush$19(SchemaKeyspace.java:348)
at
com.google.common.collect.ImmutableList.forEach(ImmutableList.java:407)
at
org.apache.cassandra.schema.SchemaKeyspace.flush(SchemaKeyspace.java:348)
at
org.apache.cassandra.schema.SchemaKeyspace.applyChanges(SchemaKeyspace.java:1282)
at org.apache.cassandra.schema.Schema.merge(Schema.java:653)
at
org.apache.cassandra.schema.Schema.mergeAndAnnounceVersion(Schema.java:586)
at
org.apache.cassandra.schema.MigrationTask.lambda$runMayThrow$0(MigrationTask.java:91)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:58)
at
org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:77)
at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:93)
at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:44)
at
org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:885)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
[3]
{code}
SEVERE: RuntimeException while executing runnable
org.apache.cassandra.db.ColumnFamilyStore$Flush$1@46975039 with executor
org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@7616fb13[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 21]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:58)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:162)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:141)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:130)
at
com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:88)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
at java.util.concurrent.FutureTask.set(FutureTask.java:233)
at java.util.concurrent.FutureTask.run(FutureTask.java:274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
> Reduce the time needed to release in-JVM dtest cluster resources after close
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-15170
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15170
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest
> Reporter: Jon Meredith
> Assignee: Jon Meredith
> Priority: Normal
>
> There are a few issues that slow the in-JVM dtests from reclaiming metaspace
> once the cluster is closed.
> IsolatedExecutor issues the shutdown on a SingleExecutorThreadPool, sometimes
> this thread was still running 10s after the dtest cluster was closed.
> Instead, switch to a ThreadPoolExecutor with a core pool size of 0 so that
> the thread executing the class loader close executes sooner.
> If an OutboundTcpConnection is waiting to connect() and the endpoint is not
> answering, it has to wait for a timeout before it exits. Instead it should
> check the isShutdown flag and terminate early if shutdown has been requested.
> In 3.0 and above, HintsCatalog.load uses java.nio.Files.list outside of a
> try-with-resources construct and leaks a file handle for the directory. This
> doesn't matter for normal usage, it leaks a file handle for each dtest
> Instance created.
> On trunk, Netty global event executor threads are still running and delay GC
> for the instance class loader.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
