[
https://issues.apache.org/jira/browse/CASSANDRA-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Klambauer updated CASSANDRA-19579:
-----------------------------------------
Description:
We are checking remaining/lingering threads during shutdown.
we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2"
Id=146 TIMED_WAITING"
this one seems to be created during shutdown / session close and not
awaited/shut down:
{noformat}
addTask:156, GlobalEventExecutor (io.netty.util.concurrent)
execute0:225, GlobalEventExecutor (io.netty.util.concurrent)
execute:221, GlobalEventExecutor (io.netty.util.concurrent)
onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context)
onChildrenClosed:589, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
lambda$close$9:552, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
run:-1, 860270832
(com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508)
tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent)
tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
- Async stack trace
addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
claim:568, CompletableFuture$UniCompletion (java.util.concurrent)
tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent)
tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
- Async stack trace
<init>:767, CompletableFuture$UniRun (java.util.concurrent)
uniRunStage:801, CompletableFuture (java.util.concurrent)
thenRunAsync:2136, CompletableFuture (java.util.concurrent)
thenRunAsync:143, CompletableFuture (java.util.concurrent)
whenAllDone:75, CompletableFutures
(com.datastax.oss.driver.internal.core.util.concurrent)
close:551, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
access$1000:300, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
lambda$closeAsync$1:272, DefaultSession
(com.datastax.oss.driver.internal.core.session)
runTask:98, PromiseTask (io.netty.util.concurrent)
run:106, PromiseTask (io.netty.util.concurrent)
runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent)
runTask:-1, AbstractEventExecutor (io.netty.util.concurrent)
- Async stack trace
addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
submit:118, AbstractExecutorService (java.util.concurrent)
submit:118, AbstractEventExecutor (io.netty.util.concurrent)
on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent)
closeSafely:286, DefaultSession (com.datastax.oss.driver.internal.core.session)
closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session)
close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core)
shutdown:172, DataStaxBackend
(com.dynatrace.apm.server.core.persistence.dcc.impl.datastax.backend)
shutdown:389, DccServiceImpl
(com.dynatrace.apm.server.core.persistence.dcc.impl.common)
shutdown:78, CassandraManagement (com.compuware.apm.server.core.lifecycle)
shutdownWithoutExit:3811, ServerLifecycle (com.compuware.apm.server.core.api)
shutdown:3648, ServerLifecycle (com.compuware.apm.server.core.api)
run:121, ServerShutdownRunner (com.compuware.apm.server.core)
run:829, Thread (java.lang)
{noformat}
the initial close here is called on com.datastax.oss.driver.api.core.CqlSession.
netty framework suggests to call
io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity
during shutdown to await event thread stopping
(slightly related issue in netty: [https://github.com/netty/netty/issues/2084] )
suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with some
timeout during close around here:
[https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199]
noting that this might slow down closing for up to 2 seconds if the netty issue
comment is correct.
this is on latest datastax java driver version: 4.17,
was:
We are checking remaining/lingering threads during shutdown.
we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2"
Id=146 TIMED_WAITING"
this one seems to be created during shutdown / session close and not
awaited/shut down:
{noformat}
addTask:156, GlobalEventExecutor (io.netty.util.concurrent)
execute0:225, GlobalEventExecutor (io.netty.util.concurrent)
execute:221, GlobalEventExecutor (io.netty.util.concurrent)
onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context)
onChildrenClosed:589, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
lambda$close$9:552, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
run:-1, 860270832
(com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508)
tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent)
tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
- Async stack trace
addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
claim:568, CompletableFuture$UniCompletion (java.util.concurrent)
tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent)
tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
- Async stack trace
<init>:767, CompletableFuture$UniRun (java.util.concurrent)
uniRunStage:801, CompletableFuture (java.util.concurrent)
thenRunAsync:2136, CompletableFuture (java.util.concurrent)
thenRunAsync:143, CompletableFuture (java.util.concurrent)
whenAllDone:75, CompletableFutures
(com.datastax.oss.driver.internal.core.util.concurrent)
close:551, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
access$1000:300, DefaultSession$SingleThreaded
(com.datastax.oss.driver.internal.core.session)
lambda$closeAsync$1:272, DefaultSession
(com.datastax.oss.driver.internal.core.session)
runTask:98, PromiseTask (io.netty.util.concurrent)
run:106, PromiseTask (io.netty.util.concurrent)
runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent)
runTask:-1, AbstractEventExecutor (io.netty.util.concurrent)
- Async stack trace
addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
submit:118, AbstractExecutorService (java.util.concurrent)
submit:118, AbstractEventExecutor (io.netty.util.concurrent)
on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent)
closeSafely:286, DefaultSession (com.datastax.oss.driver.internal.core.session)
closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session)
close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core)
shutdown:172, DataStaxBackend
(com.dynatrace.apm.server.core.persistence.dcc.impl.datastax.backend)
shutdown:389, DccServiceImpl
(com.dynatrace.apm.server.core.persistence.dcc.impl.common)
shutdown:78, CassandraManagement (com.compuware.apm.server.core.lifecycle)
shutdownWithoutExit:3811, ServerLifecycle (com.compuware.apm.server.core.api)
shutdown:3648, ServerLifecycle (com.compuware.apm.server.core.api)
run:121, ServerShutdownRunner (com.compuware.apm.server.core)
run:829, Thread (java.lang)
{noformat}
the initial close here is called on com.datastax.oss.driver.api.core.CqlSession.
netty framework suggests to call
io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity
during shutdown to await event thread stopping
(slightly related issue in netty: https://github.com/netty/netty/issues/2084 )
suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with some
timeout during close around here:
https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199
noting that this might slow down closing for up to 2 seconds if the netty issue
comment is correct.
> threads lingering after driver shutdown: session close starts thread and
> doesn't await its stop
> -----------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-19579
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19579
> Project: Cassandra
> Issue Type: Bug
> Components: Client/java-driver
> Reporter: Thomas Klambauer
> Assignee: Henry Hughes
> Priority: Normal
>
> We are checking remaining/lingering threads during shutdown.
> we noticed some with naming pattern/thread factory:
> ""globalEventExecutor-1-2" Id=146 TIMED_WAITING"
> this one seems to be created during shutdown / session close and not
> awaited/shut down:
> {noformat}
> addTask:156, GlobalEventExecutor (io.netty.util.concurrent)
> execute0:225, GlobalEventExecutor (io.netty.util.concurrent)
> execute:221, GlobalEventExecutor (io.netty.util.concurrent)
> onClose:188, DefaultNettyOptions
> (com.datastax.oss.driver.internal.core.context)
> onChildrenClosed:589, DefaultSession$SingleThreaded
> (com.datastax.oss.driver.internal.core.session)
> lambda$close$9:552, DefaultSession$SingleThreaded
> (com.datastax.oss.driver.internal.core.session)
> run:-1, 860270832
> (com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508)
> tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent)
> tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
> - Async stack trace
> addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
> claim:568, CompletableFuture$UniCompletion (java.util.concurrent)
> tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent)
> tryFire:-1, CompletableFuture$UniRun (java.util.concurrent)
> - Async stack trace
> <init>:767, CompletableFuture$UniRun (java.util.concurrent)
> uniRunStage:801, CompletableFuture (java.util.concurrent)
> thenRunAsync:2136, CompletableFuture (java.util.concurrent)
> thenRunAsync:143, CompletableFuture (java.util.concurrent)
> whenAllDone:75, CompletableFutures
> (com.datastax.oss.driver.internal.core.util.concurrent)
> close:551, DefaultSession$SingleThreaded
> (com.datastax.oss.driver.internal.core.session)
> access$1000:300, DefaultSession$SingleThreaded
> (com.datastax.oss.driver.internal.core.session)
> lambda$closeAsync$1:272, DefaultSession
> (com.datastax.oss.driver.internal.core.session)
> runTask:98, PromiseTask (io.netty.util.concurrent)
> run:106, PromiseTask (io.netty.util.concurrent)
> runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent)
> runTask:-1, AbstractEventExecutor (io.netty.util.concurrent)
> - Async stack trace
> addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:836, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent)
> execute:817, SingleThreadEventExecutor (io.netty.util.concurrent)
> submit:118, AbstractExecutorService (java.util.concurrent)
> submit:118, AbstractEventExecutor (io.netty.util.concurrent)
> on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent)
> closeSafely:286, DefaultSession
> (com.datastax.oss.driver.internal.core.session)
> closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session)
> close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core)
> shutdown:172, DataStaxBackend
> (com.dynatrace.apm.server.core.persistence.dcc.impl.datastax.backend)
> shutdown:389, DccServiceImpl
> (com.dynatrace.apm.server.core.persistence.dcc.impl.common)
> shutdown:78, CassandraManagement (com.compuware.apm.server.core.lifecycle)
> shutdownWithoutExit:3811, ServerLifecycle (com.compuware.apm.server.core.api)
> shutdown:3648, ServerLifecycle (com.compuware.apm.server.core.api)
> run:121, ServerShutdownRunner (com.compuware.apm.server.core)
> run:829, Thread (java.lang)
> {noformat}
> the initial close here is called on
> com.datastax.oss.driver.api.core.CqlSession.
> netty framework suggests to call
> io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity
> during shutdown to await event thread stopping
> (slightly related issue in netty:
> [https://github.com/netty/netty/issues/2084] )
> suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with
> some timeout during close around here:
> [https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199]
> noting that this might slow down closing for up to 2 seconds if the netty
> issue comment is correct.
> this is on latest datastax java driver version: 4.17,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
